FA-74096 / Feature flag rollout bucketing / Open access
Global holdout exclusion: The holdout reuses the flag bucket · case 01
Holdout membership differs per flag, so every user is held out of some launches and exposed to others.
ROOT CAUSE
h is computed from flag + ":" + user, the same input as the rollout bucket.
VERIFIED REPAIR
Hash "holdout:" + user so holdout membership is global and independent of the flag.
Unsuccessful approach: Dropping the colon from the holdout prefix changes the hash input and moves users between groups.
Case contract
h = crc32("holdout:" + user) % 100 and b = crc32(flag + ":" + user) % 100. Users listed exactly in force get treatment (forced). Otherwise h < holdout_pct puts the user in the global holdout (control, holdout). Otherwise b < rollout gives treatment, else control (reason rollout). Return [variant, reason, h, b].
Why this case matters
A global holdout measures the cumulative effect of all launches and must stay uncontaminated.
1 / The failure
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
import zlib
N = 1
observations = []
def solve(user, flag, holdout_pct, rollout, force):
h = zlib.crc32((flag + ':' + user).encode()) % 100
b = zlib.crc32((flag + ':' + user).encode()) % 100
if user in force:
return ['treatment', 'forced', h, b]
if h < holdout_pct:
return ['control', 'holdout', h, b]
if b < rollout:
return ['treatment', 'rollout', h, b]
return ['control', 'rollout', h, b]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),
('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),
('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),
('holdout user with full rollout stays control',
['u14', 'checkout', 10, 100, []],
['control', 'holdout', 9, 58]),
('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),
('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),
('holdout sample 1', ['u1665', 'search', 93, 0, ['qa']], ['control', 'holdout', 92, 6]),
('holdout sample 2', ['u2377', 'search', 10, 20, ['u2']], ['control', 'rollout', 24, 94])],
[('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),
('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),
('holdout user with full rollout stays control',
['u14', 'checkout', 10, 100, []],
['control', 'holdout', 9, 58]),
('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),
('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),
('holdout sample 2', ['u2377', 'search', 10, 20, ['u2']], ['control', 'rollout', 24, 94]),
('holdout sample 6', ['u1739', 'checkout', 74, 100, ['u1739']], ['treatment', 'forced', 73, 48]),
('holdout sample 7', ['u2898', 'search', 11, 50, []], ['control', 'holdout', 10, 20])],
[('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),
('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),
('holdout user with full rollout stays control',
['u14', 'checkout', 10, 100, []],
['control', 'holdout', 9, 58]),
('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),
('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),
('holdout sample 9', ['u1996', 'search', 16, 20, ['u1']], ['control', 'rollout', 16, 78]),
('holdout sample 11', ['u2412', 'checkout', 50, 50, []], ['control', 'rollout', 76, 65]),
('holdout sample 12', ['u764', 'search', 69, 20, ['qa']], ['control', 'rollout', 69, 79])],
[('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),
('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),
('holdout user with full rollout stays control',
['u14', 'checkout', 10, 100, []],
['control', 'holdout', 9, 58]),
('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),
('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),
('holdout sample 16', ['u1149', 'search', 48, 50, ['qa']], ['control', 'rollout', 48, 62]),
('holdout sample 17', ['u2253', 'search', 8, 100, ['qa']], ['treatment', 'rollout', 8, 26]),
('holdout sample 18', ['u1517', 'checkout', 0, 20, ['qa']], ['control', 'rollout', 86, 51])],
[('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),
('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),
('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),
('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),
('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),
('holdout sample 21', ['u1207', 'search', 0, 20, []], ['control', 'rollout', 66, 64]),
('holdout sample 22', ['u2600', 'checkout', 88, 50, []], ['control', 'holdout', 87, 42]),
('holdout sample 23', ['u1007', 'checkout', 52, 100, ['qa']], ['treatment', 'rollout', 52, 1])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| forced QA user inside the holdout | ['treatment', 'forced', 94, 94] | ['treatment', 'forced', 6, 94] | Failed |
| forcing matches whole user ids only | ['treatment', 'rollout', 44, 44] | ['control', 'holdout', 9, 44] | Failed |
| holdout boundary is exclusive | ['treatment', 'rollout', 94, 94] | ['treatment', 'rollout', 6, 94] | Failed |
| holdout user with full rollout stays control | ['treatment', 'rollout', 58, 58] | ['control', 'holdout', 9, 58] | Failed |
| user outside holdout follows rollout | ['treatment', 'rollout', 52, 52] | ['treatment', 'rollout', 30, 52] | Failed |
| zero rollout control | ['control', 'rollout', 5, 5] | ['control', 'rollout', 35, 5] | Failed |
| holdout sample 1 | ['control', 'holdout', 6, 6] | ['control', 'holdout', 92, 6] | Failed |
| holdout sample 2 | ['control', 'rollout', 94, 94] | ['control', 'rollout', 24, 94] | Failed |
SHA-256 / 66a4cfc4312aa66020f9f81d34b549b933d98c77e727a6d67d500e09317fe56a
2 / The unsuccessful fix
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
import zlib
N = 1
observations = []
def solve(user, flag, holdout_pct, rollout, force):
h = zlib.crc32(('holdout' + user).encode()) % 100
b = zlib.crc32((flag + ':' + user).encode()) % 100
if user in force:
return ['treatment', 'forced', h, b]
if h < holdout_pct:
return ['control', 'holdout', h, b]
if b < rollout:
return ['treatment', 'rollout', h, b]
return ['control', 'rollout', h, b]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),
('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),
('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),
('holdout user with full rollout stays control',
['u14', 'checkout', 10, 100, []],
['control', 'holdout', 9, 58]),
('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),
('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),
('holdout sample 1', ['u1665', 'search', 93, 0, ['qa']], ['control', 'holdout', 92, 6]),
('holdout sample 2', ['u2377', 'search', 10, 20, ['u2']], ['control', 'rollout', 24, 94])],
[('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),
('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),
('holdout user with full rollout stays control',
['u14', 'checkout', 10, 100, []],
['control', 'holdout', 9, 58]),
('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),
('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),
('holdout sample 2', ['u2377', 'search', 10, 20, ['u2']], ['control', 'rollout', 24, 94]),
('holdout sample 6', ['u1739', 'checkout', 74, 100, ['u1739']], ['treatment', 'forced', 73, 48]),
('holdout sample 7', ['u2898', 'search', 11, 50, []], ['control', 'holdout', 10, 20])],
[('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),
('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),
('holdout user with full rollout stays control',
['u14', 'checkout', 10, 100, []],
['control', 'holdout', 9, 58]),
('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),
('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),
('holdout sample 9', ['u1996', 'search', 16, 20, ['u1']], ['control', 'rollout', 16, 78]),
('holdout sample 11', ['u2412', 'checkout', 50, 50, []], ['control', 'rollout', 76, 65]),
('holdout sample 12', ['u764', 'search', 69, 20, ['qa']], ['control', 'rollout', 69, 79])],
[('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),
('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),
('holdout user with full rollout stays control',
['u14', 'checkout', 10, 100, []],
['control', 'holdout', 9, 58]),
('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),
('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),
('holdout sample 16', ['u1149', 'search', 48, 50, ['qa']], ['control', 'rollout', 48, 62]),
('holdout sample 17', ['u2253', 'search', 8, 100, ['qa']], ['treatment', 'rollout', 8, 26]),
('holdout sample 18', ['u1517', 'checkout', 0, 20, ['qa']], ['control', 'rollout', 86, 51])],
[('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),
('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),
('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),
('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),
('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),
('holdout sample 21', ['u1207', 'search', 0, 20, []], ['control', 'rollout', 66, 64]),
('holdout sample 22', ['u2600', 'checkout', 88, 50, []], ['control', 'holdout', 87, 42]),
('holdout sample 23', ['u1007', 'checkout', 52, 100, ['qa']], ['treatment', 'rollout', 52, 1])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| forced QA user inside the holdout | ['treatment', 'forced', 0, 94] | ['treatment', 'forced', 6, 94] | Failed |
| forcing matches whole user ids only | ['control', 'holdout', 2, 44] | ['control', 'holdout', 9, 44] | Failed |
| holdout boundary is exclusive | ['control', 'holdout', 0, 94] | ['treatment', 'rollout', 6, 94] | Failed |
| holdout user with full rollout stays control | ['control', 'holdout', 2, 58] | ['control', 'holdout', 9, 58] | Failed |
| user outside holdout follows rollout | ['treatment', 'rollout', 8, 52] | ['treatment', 'rollout', 30, 52] | Failed |
| zero rollout control | ['control', 'rollout', 81, 5] | ['control', 'rollout', 35, 5] | Failed |
| holdout sample 1 | ['control', 'holdout', 22, 6] | ['control', 'holdout', 92, 6] | Failed |
| holdout sample 2 | ['control', 'rollout', 90, 94] | ['control', 'rollout', 24, 94] | Failed |
SHA-256 / 54b96009f6f356fe8a12613772625d12d1662c97207c6d69d5b0a3f9e25588f1
3 / The verified repair
Exit 0"""Failure Map reference implementation. Python standard library only."""
import json
import zlib
N = 1
observations = []
def solve(user, flag, holdout_pct, rollout, force):
h = zlib.crc32(('holdout:' + user).encode()) % 100
b = zlib.crc32((flag + ':' + user).encode()) % 100
if user in force:
return ['treatment', 'forced', h, b]
if h < holdout_pct:
return ['control', 'holdout', h, b]
if b < rollout:
return ['treatment', 'rollout', h, b]
return ['control', 'rollout', h, b]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),
('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),
('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),
('holdout user with full rollout stays control',
['u14', 'checkout', 10, 100, []],
['control', 'holdout', 9, 58]),
('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),
('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),
('holdout sample 1', ['u1665', 'search', 93, 0, ['qa']], ['control', 'holdout', 92, 6]),
('holdout sample 2', ['u2377', 'search', 10, 20, ['u2']], ['control', 'rollout', 24, 94])],
[('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),
('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),
('holdout user with full rollout stays control',
['u14', 'checkout', 10, 100, []],
['control', 'holdout', 9, 58]),
('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),
('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),
('holdout sample 2', ['u2377', 'search', 10, 20, ['u2']], ['control', 'rollout', 24, 94]),
('holdout sample 6', ['u1739', 'checkout', 74, 100, ['u1739']], ['treatment', 'forced', 73, 48]),
('holdout sample 7', ['u2898', 'search', 11, 50, []], ['control', 'holdout', 10, 20])],
[('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),
('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),
('holdout user with full rollout stays control',
['u14', 'checkout', 10, 100, []],
['control', 'holdout', 9, 58]),
('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),
('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),
('holdout sample 9', ['u1996', 'search', 16, 20, ['u1']], ['control', 'rollout', 16, 78]),
('holdout sample 11', ['u2412', 'checkout', 50, 50, []], ['control', 'rollout', 76, 65]),
('holdout sample 12', ['u764', 'search', 69, 20, ['qa']], ['control', 'rollout', 69, 79])],
[('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),
('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),
('holdout user with full rollout stays control',
['u14', 'checkout', 10, 100, []],
['control', 'holdout', 9, 58]),
('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),
('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),
('holdout sample 16', ['u1149', 'search', 48, 50, ['qa']], ['control', 'rollout', 48, 62]),
('holdout sample 17', ['u2253', 'search', 8, 100, ['qa']], ['treatment', 'rollout', 8, 26]),
('holdout sample 18', ['u1517', 'checkout', 0, 20, ['qa']], ['control', 'rollout', 86, 51])],
[('forced QA user inside the holdout', ['u6', 'checkout', 10, 0, ['u6']], ['treatment', 'forced', 6, 94]),
('forcing matches whole user ids only', ['u14', 'search', 10, 100, ['u1']], ['control', 'holdout', 9, 44]),
('holdout boundary is exclusive', ['u6', 'checkout', 6, 100, []], ['treatment', 'rollout', 6, 94]),
('user outside holdout follows rollout', ['u777', 'search', 0, 100, []], ['treatment', 'rollout', 30, 52]),
('zero rollout control', ['u778', 'search', 0, 0, []], ['control', 'rollout', 35, 5]),
('holdout sample 21', ['u1207', 'search', 0, 20, []], ['control', 'rollout', 66, 64]),
('holdout sample 22', ['u2600', 'checkout', 88, 50, []], ['control', 'holdout', 87, 42]),
('holdout sample 23', ['u1007', 'checkout', 52, 100, ['qa']], ['treatment', 'rollout', 52, 1])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| forced QA user inside the holdout | ['treatment', 'forced', 6, 94] | ['treatment', 'forced', 6, 94] | Passed |
| forcing matches whole user ids only | ['control', 'holdout', 9, 44] | ['control', 'holdout', 9, 44] | Passed |
| holdout boundary is exclusive | ['treatment', 'rollout', 6, 94] | ['treatment', 'rollout', 6, 94] | Passed |
| holdout user with full rollout stays control | ['control', 'holdout', 9, 58] | ['control', 'holdout', 9, 58] | Passed |
| user outside holdout follows rollout | ['treatment', 'rollout', 30, 52] | ['treatment', 'rollout', 30, 52] | Passed |
| zero rollout control | ['control', 'rollout', 35, 5] | ['control', 'rollout', 35, 5] | Passed |
| holdout sample 1 | ['control', 'holdout', 92, 6] | ['control', 'holdout', 92, 6] | Passed |
| holdout sample 2 | ['control', 'rollout', 24, 94] | ['control', 'rollout', 24, 94] | Passed |
SHA-256 / 50851e743dc37c8890662b124d9a27dc7283d4c47c407b20a7b7f422ec2bb41b
Verification & scope
A deterministic toy flag-evaluation model with a stipulated contract; it does not reproduce any vendor SDK byte for byte. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.
Observations recorded using Python 3.12.14 at 2026-09-29T14:48:53.541389+00:00.
Case digest / 81bc0d0fcb73a61022b15efc13313dbf3ba5ff0bd828a93ab9a176f0e7a407cd