FA-74236 / Feature flag rollout bucketing / Open access
Flag override resolution: An invalid layer short-circuits to the default · case 01
A malformed environment value hides a valid remote value.
ROOT CAUSE
After the first present-but-invalid layer the function returns the default.
VERIFIED REPAIR
Skip invalid layers and continue with the next one.
Unsuccessful approach: Returning the raw invalid value serves a wrongly typed flag.
Case contract
Layers in precedence order: env var FLAG_<KEY uppercased with "-" -> "_">, local overrides[key], remote[key], then the default. A layer is used only if its value is valid for the default's type, else the next layer is tried. Non-env values must have exactly the default's type (bool is not int). Env strings: bool defaults accept "true"/"false" case-insensitively; int defaults accept digits with an optional leading "-"; str defaults accept any string. Return [value, source].
Why this case matters
Local and environment overrides are how operators pin flags during incidents; precedence must be exact.
1 / The failure
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(key, default, remote, overrides, env):
def coerce(v, from_env):
if from_env:
if isinstance(default, bool):
low = v.lower()
return (True, low == 'true') if low in ('true', 'false') else (False, None)
if isinstance(default, int):
body = v[1:] if v.startswith('-') else v
return (True, int(v)) if body.isdigit() else (False, None)
return (True, v)
return (True, v) if type(v) is type(default) else (False, None)
name = 'FLAG_' + key.upper().replace('-', '_')
layers = [('env', env, name, True), ('override', overrides, key, False), ('remote', remote, key, False)]
for source, table, k, from_env in layers:
if k in table:
ok, value = coerce(table[k], from_env)
if ok:
return [value, source]
return [default, 'default']
return [default, 'default']
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('boolean remote value rejected for integer flag', ['beta', 3, {'beta': True}, {}, {}], [3, 'default']),
('environment beats local override',
['new-ui', False, {}, {'new-ui': False}, {'FLAG_NEW_UI': 'true'}],
[True, 'env']),
('dashed key maps to underscored env name',
['dark-mode-v2', 'blue', {}, {}, {'FLAG_DARK_MODE_V2': 'green'}],
['green', 'env']),
('invalid env value falls through to remote',
['beta', 0, {'beta': 5}, {}, {'FLAG_BETA': 'lots'}],
[5, 'remote']),
('negative integer from env', ['beta', 0, {}, {}, {'FLAG_BETA': '-3'}], [-3, 'env']),
('non-boolean env text is invalid for boolean flag',
['beta', True, {'beta': True}, {}, {'FLAG_BETA': 'yes'}],
[True, 'remote']),
('override layer sample 1', ['beta', 7, {'beta': 1}, {}, {}], [1, 'remote']),
('override layer sample 2', ['new-ui', 'blue', {'new-ui': False}, {}, {}], ['blue', 'default'])],
[('environment beats local override',
['new-ui', False, {}, {'new-ui': False}, {'FLAG_NEW_UI': 'true'}],
[True, 'env']),
('dashed key maps to underscored env name',
['dark-mode-v2', 'blue', {}, {}, {'FLAG_DARK_MODE_V2': 'green'}],
['green', 'env']),
('invalid env value falls through to remote',
['beta', 0, {'beta': 5}, {}, {'FLAG_BETA': 'lots'}],
[5, 'remote']),
('negative integer from env', ['beta', 0, {}, {}, {'FLAG_BETA': '-3'}], [-3, 'env']),
('plus sign is not a valid integer', ['beta', 0, {'beta': 9}, {}, {'FLAG_BETA': '+4'}], [9, 'remote']),
('non-boolean env text is invalid for boolean flag',
['beta', True, {'beta': True}, {}, {'FLAG_BETA': 'yes'}],
[True, 'remote']),
('override layer sample 6', ['beta', True, {}, {'beta': 0}, {'FLAG_beta': '-3'}], [True, 'default']),
('override layer sample 55', ['beta', False, {'beta': True}, {}, {'FLAG_BETA': 'yes'}], [True, 'remote'])],
[('dashed key maps to underscored env name',
['dark-mode-v2', 'blue', {}, {}, {'FLAG_DARK_MODE_V2': 'green'}],
['green', 'env']),
('invalid env value falls through to remote',
['beta', 0, {'beta': 5}, {}, {'FLAG_BETA': 'lots'}],
[5, 'remote']),
('negative integer from env', ['beta', 0, {}, {}, {'FLAG_BETA': '-3'}], [-3, 'env']),
('plus sign is not a valid integer', ['beta', 0, {'beta': 9}, {}, {'FLAG_BETA': '+4'}], [9, 'remote']),
('env boolean is case-insensitive', ['beta', False, {}, {}, {'FLAG_BETA': 'TRUE'}], [True, 'env']),
('override layer sample 11', ['beta', True, {'beta': -2}, {'beta': 0}, {}], [True, 'default']),
('override layer sample 15',
['dark-mode-v2', True, {'dark-mode-v2': 0}, {'dark-mode-v2': 3}, {'FLAG_dark_mode_v2': 'yes'}],
[True, 'default']),
('override layer sample 38',
['new-ui', 7, {'new-ui': 'x'}, {'new-ui': 0}, {'FLAG_NEW_UI': 'true'}],
[0, 'override'])],
[('invalid env value falls through to remote',
['beta', 0, {'beta': 5}, {}, {'FLAG_BETA': 'lots'}],
[5, 'remote']),
('negative integer from env', ['beta', 0, {}, {}, {'FLAG_BETA': '-3'}], [-3, 'env']),
('plus sign is not a valid integer', ['beta', 0, {'beta': 9}, {}, {'FLAG_BETA': '+4'}], [9, 'remote']),
('env boolean is case-insensitive', ['beta', False, {}, {}, {'FLAG_BETA': 'TRUE'}], [True, 'env']),
('non-boolean env text is invalid for boolean flag',
['beta', True, {'beta': True}, {}, {'FLAG_BETA': 'yes'}],
[True, 'remote']),
('override layer sample 16', ['dark-mode-v2', 'blue', {'dark-mode-v2': 1}, {}, {}], ['blue', 'default']),
('override layer sample 25',
['dark-mode-v2', 'blue', {'dark-mode-v2': True}, {'dark-mode-v2': -2}, {}],
['blue', 'default']),
('override layer sample 28', ['beta', 7, {'beta': 0}, {'beta': 'x'}, {}], [0, 'remote'])],
[('negative integer from env', ['beta', 0, {}, {}, {'FLAG_BETA': '-3'}], [-3, 'env']),
('plus sign is not a valid integer', ['beta', 0, {'beta': 9}, {}, {'FLAG_BETA': '+4'}], [9, 'remote']),
('env boolean is case-insensitive', ['beta', False, {}, {}, {'FLAG_BETA': 'TRUE'}], [True, 'env']),
('non-boolean env text is invalid for boolean flag',
['beta', True, {'beta': True}, {}, {'FLAG_BETA': 'yes'}],
[True, 'remote']),
('local override beats remote', ['beta', 'a', {'beta': 'b'}, {'beta': 'c'}, {}], ['c', 'override']),
('override layer sample 5',
['dark-mode-v2', 7, {'dark-mode-v2': 1}, {}, {'FLAG_DARK_MODE_V2': 'yes'}],
[1, 'remote']),
('override layer sample 21',
['dark-mode-v2', False, {}, {}, {'FLAG_DARK-MODE-V2': 'true'}],
[False, 'default']),
('override layer sample 32', ['new-ui', False, {'new-ui': -2}, {'new-ui': -2}, {}], [False, 'default'])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| boolean remote value rejected for integer flag | [3, 'default'] | [3, 'default'] | Passed |
| environment beats local override | [True, 'env'] | [True, 'env'] | Passed |
| dashed key maps to underscored env name | ['green', 'env'] | ['green', 'env'] | Passed |
| invalid env value falls through to remote | [0, 'default'] | [5, 'remote'] | Failed |
| negative integer from env | [-3, 'env'] | [-3, 'env'] | Passed |
| non-boolean env text is invalid for boolean flag | [True, 'default'] | [True, 'remote'] | Failed |
| override layer sample 1 | [1, 'remote'] | [1, 'remote'] | Passed |
| override layer sample 2 | ['blue', 'default'] | ['blue', 'default'] | Passed |
SHA-256 / eb5867b7a37da6a0ec8dcd7c5c2680bfc8439d18de962232863f4f1e2a077b4c
2 / The unsuccessful fix
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(key, default, remote, overrides, env):
def coerce(v, from_env):
if from_env:
if isinstance(default, bool):
low = v.lower()
return (True, low == 'true') if low in ('true', 'false') else (False, None)
if isinstance(default, int):
body = v[1:] if v.startswith('-') else v
return (True, int(v)) if body.isdigit() else (False, None)
return (True, v)
return (True, v) if type(v) is type(default) else (False, None)
name = 'FLAG_' + key.upper().replace('-', '_')
layers = [('env', env, name, True), ('override', overrides, key, False), ('remote', remote, key, False)]
for source, table, k, from_env in layers:
if k in table:
ok, value = coerce(table[k], from_env)
if ok:
return [value, source]
return [table[k], source]
return [default, 'default']
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('boolean remote value rejected for integer flag', ['beta', 3, {'beta': True}, {}, {}], [3, 'default']),
('environment beats local override',
['new-ui', False, {}, {'new-ui': False}, {'FLAG_NEW_UI': 'true'}],
[True, 'env']),
('dashed key maps to underscored env name',
['dark-mode-v2', 'blue', {}, {}, {'FLAG_DARK_MODE_V2': 'green'}],
['green', 'env']),
('invalid env value falls through to remote',
['beta', 0, {'beta': 5}, {}, {'FLAG_BETA': 'lots'}],
[5, 'remote']),
('negative integer from env', ['beta', 0, {}, {}, {'FLAG_BETA': '-3'}], [-3, 'env']),
('non-boolean env text is invalid for boolean flag',
['beta', True, {'beta': True}, {}, {'FLAG_BETA': 'yes'}],
[True, 'remote']),
('override layer sample 1', ['beta', 7, {'beta': 1}, {}, {}], [1, 'remote']),
('override layer sample 2', ['new-ui', 'blue', {'new-ui': False}, {}, {}], ['blue', 'default'])],
[('environment beats local override',
['new-ui', False, {}, {'new-ui': False}, {'FLAG_NEW_UI': 'true'}],
[True, 'env']),
('dashed key maps to underscored env name',
['dark-mode-v2', 'blue', {}, {}, {'FLAG_DARK_MODE_V2': 'green'}],
['green', 'env']),
('invalid env value falls through to remote',
['beta', 0, {'beta': 5}, {}, {'FLAG_BETA': 'lots'}],
[5, 'remote']),
('negative integer from env', ['beta', 0, {}, {}, {'FLAG_BETA': '-3'}], [-3, 'env']),
('plus sign is not a valid integer', ['beta', 0, {'beta': 9}, {}, {'FLAG_BETA': '+4'}], [9, 'remote']),
('non-boolean env text is invalid for boolean flag',
['beta', True, {'beta': True}, {}, {'FLAG_BETA': 'yes'}],
[True, 'remote']),
('override layer sample 6', ['beta', True, {}, {'beta': 0}, {'FLAG_beta': '-3'}], [True, 'default']),
('override layer sample 55', ['beta', False, {'beta': True}, {}, {'FLAG_BETA': 'yes'}], [True, 'remote'])],
[('dashed key maps to underscored env name',
['dark-mode-v2', 'blue', {}, {}, {'FLAG_DARK_MODE_V2': 'green'}],
['green', 'env']),
('invalid env value falls through to remote',
['beta', 0, {'beta': 5}, {}, {'FLAG_BETA': 'lots'}],
[5, 'remote']),
('negative integer from env', ['beta', 0, {}, {}, {'FLAG_BETA': '-3'}], [-3, 'env']),
('plus sign is not a valid integer', ['beta', 0, {'beta': 9}, {}, {'FLAG_BETA': '+4'}], [9, 'remote']),
('env boolean is case-insensitive', ['beta', False, {}, {}, {'FLAG_BETA': 'TRUE'}], [True, 'env']),
('override layer sample 11', ['beta', True, {'beta': -2}, {'beta': 0}, {}], [True, 'default']),
('override layer sample 15',
['dark-mode-v2', True, {'dark-mode-v2': 0}, {'dark-mode-v2': 3}, {'FLAG_dark_mode_v2': 'yes'}],
[True, 'default']),
('override layer sample 38',
['new-ui', 7, {'new-ui': 'x'}, {'new-ui': 0}, {'FLAG_NEW_UI': 'true'}],
[0, 'override'])],
[('invalid env value falls through to remote',
['beta', 0, {'beta': 5}, {}, {'FLAG_BETA': 'lots'}],
[5, 'remote']),
('negative integer from env', ['beta', 0, {}, {}, {'FLAG_BETA': '-3'}], [-3, 'env']),
('plus sign is not a valid integer', ['beta', 0, {'beta': 9}, {}, {'FLAG_BETA': '+4'}], [9, 'remote']),
('env boolean is case-insensitive', ['beta', False, {}, {}, {'FLAG_BETA': 'TRUE'}], [True, 'env']),
('non-boolean env text is invalid for boolean flag',
['beta', True, {'beta': True}, {}, {'FLAG_BETA': 'yes'}],
[True, 'remote']),
('override layer sample 16', ['dark-mode-v2', 'blue', {'dark-mode-v2': 1}, {}, {}], ['blue', 'default']),
('override layer sample 25',
['dark-mode-v2', 'blue', {'dark-mode-v2': True}, {'dark-mode-v2': -2}, {}],
['blue', 'default']),
('override layer sample 28', ['beta', 7, {'beta': 0}, {'beta': 'x'}, {}], [0, 'remote'])],
[('negative integer from env', ['beta', 0, {}, {}, {'FLAG_BETA': '-3'}], [-3, 'env']),
('plus sign is not a valid integer', ['beta', 0, {'beta': 9}, {}, {'FLAG_BETA': '+4'}], [9, 'remote']),
('env boolean is case-insensitive', ['beta', False, {}, {}, {'FLAG_BETA': 'TRUE'}], [True, 'env']),
('non-boolean env text is invalid for boolean flag',
['beta', True, {'beta': True}, {}, {'FLAG_BETA': 'yes'}],
[True, 'remote']),
('local override beats remote', ['beta', 'a', {'beta': 'b'}, {'beta': 'c'}, {}], ['c', 'override']),
('override layer sample 5',
['dark-mode-v2', 7, {'dark-mode-v2': 1}, {}, {'FLAG_DARK_MODE_V2': 'yes'}],
[1, 'remote']),
('override layer sample 21',
['dark-mode-v2', False, {}, {}, {'FLAG_DARK-MODE-V2': 'true'}],
[False, 'default']),
('override layer sample 32', ['new-ui', False, {'new-ui': -2}, {'new-ui': -2}, {}], [False, 'default'])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| boolean remote value rejected for integer flag | [True, 'remote'] | [3, 'default'] | Failed |
| environment beats local override | [True, 'env'] | [True, 'env'] | Passed |
| dashed key maps to underscored env name | ['green', 'env'] | ['green', 'env'] | Passed |
| invalid env value falls through to remote | ['lots', 'env'] | [5, 'remote'] | Failed |
| negative integer from env | [-3, 'env'] | [-3, 'env'] | Passed |
| non-boolean env text is invalid for boolean flag | ['yes', 'env'] | [True, 'remote'] | Failed |
| override layer sample 1 | [1, 'remote'] | [1, 'remote'] | Passed |
| override layer sample 2 | [False, 'remote'] | ['blue', 'default'] | Failed |
SHA-256 / 10c4fcd97d2b75a7d631abe09a5eb4f5dfbab204a031b5ded3d735ac203feee6
3 / The verified repair
Exit 0"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(key, default, remote, overrides, env):
def coerce(v, from_env):
if from_env:
if isinstance(default, bool):
low = v.lower()
return (True, low == 'true') if low in ('true', 'false') else (False, None)
if isinstance(default, int):
body = v[1:] if v.startswith('-') else v
return (True, int(v)) if body.isdigit() else (False, None)
return (True, v)
return (True, v) if type(v) is type(default) else (False, None)
name = 'FLAG_' + key.upper().replace('-', '_')
layers = [('env', env, name, True), ('override', overrides, key, False), ('remote', remote, key, False)]
for source, table, k, from_env in layers:
if k in table:
ok, value = coerce(table[k], from_env)
if ok:
return [value, source]
return [default, 'default']
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('boolean remote value rejected for integer flag', ['beta', 3, {'beta': True}, {}, {}], [3, 'default']),
('environment beats local override',
['new-ui', False, {}, {'new-ui': False}, {'FLAG_NEW_UI': 'true'}],
[True, 'env']),
('dashed key maps to underscored env name',
['dark-mode-v2', 'blue', {}, {}, {'FLAG_DARK_MODE_V2': 'green'}],
['green', 'env']),
('invalid env value falls through to remote',
['beta', 0, {'beta': 5}, {}, {'FLAG_BETA': 'lots'}],
[5, 'remote']),
('negative integer from env', ['beta', 0, {}, {}, {'FLAG_BETA': '-3'}], [-3, 'env']),
('non-boolean env text is invalid for boolean flag',
['beta', True, {'beta': True}, {}, {'FLAG_BETA': 'yes'}],
[True, 'remote']),
('override layer sample 1', ['beta', 7, {'beta': 1}, {}, {}], [1, 'remote']),
('override layer sample 2', ['new-ui', 'blue', {'new-ui': False}, {}, {}], ['blue', 'default'])],
[('environment beats local override',
['new-ui', False, {}, {'new-ui': False}, {'FLAG_NEW_UI': 'true'}],
[True, 'env']),
('dashed key maps to underscored env name',
['dark-mode-v2', 'blue', {}, {}, {'FLAG_DARK_MODE_V2': 'green'}],
['green', 'env']),
('invalid env value falls through to remote',
['beta', 0, {'beta': 5}, {}, {'FLAG_BETA': 'lots'}],
[5, 'remote']),
('negative integer from env', ['beta', 0, {}, {}, {'FLAG_BETA': '-3'}], [-3, 'env']),
('plus sign is not a valid integer', ['beta', 0, {'beta': 9}, {}, {'FLAG_BETA': '+4'}], [9, 'remote']),
('non-boolean env text is invalid for boolean flag',
['beta', True, {'beta': True}, {}, {'FLAG_BETA': 'yes'}],
[True, 'remote']),
('override layer sample 6', ['beta', True, {}, {'beta': 0}, {'FLAG_beta': '-3'}], [True, 'default']),
('override layer sample 55', ['beta', False, {'beta': True}, {}, {'FLAG_BETA': 'yes'}], [True, 'remote'])],
[('dashed key maps to underscored env name',
['dark-mode-v2', 'blue', {}, {}, {'FLAG_DARK_MODE_V2': 'green'}],
['green', 'env']),
('invalid env value falls through to remote',
['beta', 0, {'beta': 5}, {}, {'FLAG_BETA': 'lots'}],
[5, 'remote']),
('negative integer from env', ['beta', 0, {}, {}, {'FLAG_BETA': '-3'}], [-3, 'env']),
('plus sign is not a valid integer', ['beta', 0, {'beta': 9}, {}, {'FLAG_BETA': '+4'}], [9, 'remote']),
('env boolean is case-insensitive', ['beta', False, {}, {}, {'FLAG_BETA': 'TRUE'}], [True, 'env']),
('override layer sample 11', ['beta', True, {'beta': -2}, {'beta': 0}, {}], [True, 'default']),
('override layer sample 15',
['dark-mode-v2', True, {'dark-mode-v2': 0}, {'dark-mode-v2': 3}, {'FLAG_dark_mode_v2': 'yes'}],
[True, 'default']),
('override layer sample 38',
['new-ui', 7, {'new-ui': 'x'}, {'new-ui': 0}, {'FLAG_NEW_UI': 'true'}],
[0, 'override'])],
[('invalid env value falls through to remote',
['beta', 0, {'beta': 5}, {}, {'FLAG_BETA': 'lots'}],
[5, 'remote']),
('negative integer from env', ['beta', 0, {}, {}, {'FLAG_BETA': '-3'}], [-3, 'env']),
('plus sign is not a valid integer', ['beta', 0, {'beta': 9}, {}, {'FLAG_BETA': '+4'}], [9, 'remote']),
('env boolean is case-insensitive', ['beta', False, {}, {}, {'FLAG_BETA': 'TRUE'}], [True, 'env']),
('non-boolean env text is invalid for boolean flag',
['beta', True, {'beta': True}, {}, {'FLAG_BETA': 'yes'}],
[True, 'remote']),
('override layer sample 16', ['dark-mode-v2', 'blue', {'dark-mode-v2': 1}, {}, {}], ['blue', 'default']),
('override layer sample 25',
['dark-mode-v2', 'blue', {'dark-mode-v2': True}, {'dark-mode-v2': -2}, {}],
['blue', 'default']),
('override layer sample 28', ['beta', 7, {'beta': 0}, {'beta': 'x'}, {}], [0, 'remote'])],
[('negative integer from env', ['beta', 0, {}, {}, {'FLAG_BETA': '-3'}], [-3, 'env']),
('plus sign is not a valid integer', ['beta', 0, {'beta': 9}, {}, {'FLAG_BETA': '+4'}], [9, 'remote']),
('env boolean is case-insensitive', ['beta', False, {}, {}, {'FLAG_BETA': 'TRUE'}], [True, 'env']),
('non-boolean env text is invalid for boolean flag',
['beta', True, {'beta': True}, {}, {'FLAG_BETA': 'yes'}],
[True, 'remote']),
('local override beats remote', ['beta', 'a', {'beta': 'b'}, {'beta': 'c'}, {}], ['c', 'override']),
('override layer sample 5',
['dark-mode-v2', 7, {'dark-mode-v2': 1}, {}, {'FLAG_DARK_MODE_V2': 'yes'}],
[1, 'remote']),
('override layer sample 21',
['dark-mode-v2', False, {}, {}, {'FLAG_DARK-MODE-V2': 'true'}],
[False, 'default']),
('override layer sample 32', ['new-ui', False, {'new-ui': -2}, {'new-ui': -2}, {}], [False, 'default'])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| boolean remote value rejected for integer flag | [3, 'default'] | [3, 'default'] | Passed |
| environment beats local override | [True, 'env'] | [True, 'env'] | Passed |
| dashed key maps to underscored env name | ['green', 'env'] | ['green', 'env'] | Passed |
| invalid env value falls through to remote | [5, 'remote'] | [5, 'remote'] | Passed |
| negative integer from env | [-3, 'env'] | [-3, 'env'] | Passed |
| non-boolean env text is invalid for boolean flag | [True, 'remote'] | [True, 'remote'] | Passed |
| override layer sample 1 | [1, 'remote'] | [1, 'remote'] | Passed |
| override layer sample 2 | ['blue', 'default'] | ['blue', 'default'] | Passed |
SHA-256 / cc94280d2ba5764354f837134dd4d7d79b15134bf973600626286358b0192e80
Verification & scope
A deterministic toy flag-evaluation model with a stipulated contract; it does not reproduce any vendor SDK byte for byte. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.
Observations recorded using Python 3.12.14 at 2026-09-29T14:48:54.803054+00:00.
Case digest / e6b76c8ab7922dc860e832d0a6c1c621c16ef758504f4373f1040ab0d75990bd