FA-74271 / Feature flag rollout bucketing / Open access
Traffic then variant allocation: The variant split reuses the traffic bucket · case 01
At 20 percent traffic every allocated user lands in the first arm.
ROOT CAUSE
The variant bucket is the same number t that already passed t < traffic.
VERIFIED REPAIR
Draw the variant bucket from an independent hash of salt + ":variant:" + user.
Unsuccessful approach: Rescaling t into [0, 100) keeps the split coupled to traffic and still disagrees with the contract hash.
Case contract
Stage 1: t = crc32(salt + ":traffic:" + user) % 100; users with t >= traffic (1..100) are not in the experiment. Stage 2 uses an independent v = crc32(salt + ":variant:" + user) % 100 and walks weights ([arm, percent of allocated users], summing to 100) cumulatively, half-open. Return [in_experiment, arm or None].
Why this case matters
Separating traffic allocation from the variant split keeps arms balanced when traffic changes.
1 / The failure
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
import zlib
N = 1
observations = []
def solve(salt, user, traffic, weights):
t = zlib.crc32((salt + ':traffic:' + user).encode()) % 100
if t >= traffic:
return [False, None]
v = t
acc = 0
for arm, w in weights:
acc += w
if v < acc:
return [True, arm]
return [True, weights[-1][0]]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('low traffic still fills every arm',
['exp1', 'm0', 20, [['control', 50], ['treatment', 50]]],
[True, 'treatment']),
('low traffic second user', ['exp1', 'm5', 20, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('arm weights are relative to allocated users',
['exp1', 'm8', 20, [['control', 20], ['treatment', 80]]],
[True, 'control']),
('traffic boundary excludes t equal to traffic', ['exp1', 'm0', 14, [['a', 50], ['b', 50]]], [False, None]),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('allocation sample 1', ['exp2', 'u3390', 9, [['control', 20], ['treatment', 80]]], [False, None]),
('allocation sample 2', ['exp2', 'u2292', 50, [['control', 20], ['treatment', 80]]], [False, None]),
('allocation sample 3', ['exp2', 'u2623', 67, [['control', 20], ['treatment', 80]]], [True, 'control'])],
[('low traffic second user', ['exp1', 'm5', 20, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('arm weights are relative to allocated users',
['exp1', 'm8', 20, [['control', 20], ['treatment', 80]]],
[True, 'control']),
('traffic boundary excludes t equal to traffic', ['exp1', 'm0', 14, [['a', 50], ['b', 50]]], [False, None]),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('full traffic control', ['exp2', 'n1', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 6', ['exp1', 'u2423', 50, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 13', ['exp2', 'u3641', 50, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 20', ['exp1', 'u1715', 57, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b'])],
[('low traffic still fills every arm',
['exp1', 'm0', 20, [['control', 50], ['treatment', 50]]],
[True, 'treatment']),
('arm weights are relative to allocated users',
['exp1', 'm8', 20, [['control', 20], ['treatment', 80]]],
[True, 'control']),
('traffic boundary excludes t equal to traffic', ['exp1', 'm0', 14, [['a', 50], ['b', 50]]], [False, None]),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('full traffic control', ['exp2', 'n1', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 11', ['exp1', 'u2908', 30, [['a', 34], ['b', 33], ['c', 33]]], [False, None]),
('allocation sample 12', ['exp1', 'u1733', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 50', ['exp1', 'u2120', 100, [['control', 50], ['treatment', 50]]], [True, 'control'])],
[('low traffic still fills every arm',
['exp1', 'm0', 20, [['control', 50], ['treatment', 50]]],
[True, 'treatment']),
('low traffic second user', ['exp1', 'm5', 20, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('traffic boundary excludes t equal to traffic', ['exp1', 'm0', 14, [['a', 50], ['b', 50]]], [False, None]),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('full traffic control', ['exp2', 'n1', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 3', ['exp2', 'u2623', 67, [['control', 20], ['treatment', 80]]], [True, 'control']),
('allocation sample 9', ['exp1', 'u3147', 77, [['a', 34], ['b', 33], ['c', 33]]], [True, 'a']),
('allocation sample 16', ['exp2', 'u2262', 53, [['control', 20], ['treatment', 80]]], [True, 'treatment'])],
[('low traffic still fills every arm',
['exp1', 'm0', 20, [['control', 50], ['treatment', 50]]],
[True, 'treatment']),
('low traffic second user', ['exp1', 'm5', 20, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('arm weights are relative to allocated users',
['exp1', 'm8', 20, [['control', 20], ['treatment', 80]]],
[True, 'control']),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('full traffic control', ['exp2', 'n1', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 21', ['exp1', 'u843', 100, [['control', 20], ['treatment', 80]]], [True, 'treatment']),
('allocation sample 33', ['exp1', 'u1247', 50, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('allocation sample 41', ['exp2', 'u3401', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b'])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| low traffic still fills every arm | [True, 'control'] | [True, 'treatment'] | Failed |
| low traffic second user | [True, 'control'] | [True, 'treatment'] | Failed |
| arm weights are relative to allocated users | [True, 'control'] | [True, 'control'] | Passed |
| traffic boundary excludes t equal to traffic | [False, None] | [False, None] | Passed |
| traffic boundary includes t just below | [True, 'a'] | [True, 'b'] | Failed |
| allocation sample 1 | [False, None] | [False, None] | Passed |
| allocation sample 2 | [False, None] | [False, None] | Passed |
| allocation sample 3 | [True, 'treatment'] | [True, 'control'] | Failed |
SHA-256 / 54d7fad87398cb1483d3c9749e8f12e1f314e49dabad9e7a95e675573f055aca
2 / The unsuccessful fix
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
import zlib
N = 1
observations = []
def solve(salt, user, traffic, weights):
t = zlib.crc32((salt + ':traffic:' + user).encode()) % 100
if t >= traffic:
return [False, None]
v = t * 100 // traffic
acc = 0
for arm, w in weights:
acc += w
if v < acc:
return [True, arm]
return [True, weights[-1][0]]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('low traffic still fills every arm',
['exp1', 'm0', 20, [['control', 50], ['treatment', 50]]],
[True, 'treatment']),
('low traffic second user', ['exp1', 'm5', 20, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('arm weights are relative to allocated users',
['exp1', 'm8', 20, [['control', 20], ['treatment', 80]]],
[True, 'control']),
('traffic boundary excludes t equal to traffic', ['exp1', 'm0', 14, [['a', 50], ['b', 50]]], [False, None]),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('allocation sample 1', ['exp2', 'u3390', 9, [['control', 20], ['treatment', 80]]], [False, None]),
('allocation sample 2', ['exp2', 'u2292', 50, [['control', 20], ['treatment', 80]]], [False, None]),
('allocation sample 3', ['exp2', 'u2623', 67, [['control', 20], ['treatment', 80]]], [True, 'control'])],
[('low traffic second user', ['exp1', 'm5', 20, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('arm weights are relative to allocated users',
['exp1', 'm8', 20, [['control', 20], ['treatment', 80]]],
[True, 'control']),
('traffic boundary excludes t equal to traffic', ['exp1', 'm0', 14, [['a', 50], ['b', 50]]], [False, None]),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('full traffic control', ['exp2', 'n1', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 6', ['exp1', 'u2423', 50, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 13', ['exp2', 'u3641', 50, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 20', ['exp1', 'u1715', 57, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b'])],
[('low traffic still fills every arm',
['exp1', 'm0', 20, [['control', 50], ['treatment', 50]]],
[True, 'treatment']),
('arm weights are relative to allocated users',
['exp1', 'm8', 20, [['control', 20], ['treatment', 80]]],
[True, 'control']),
('traffic boundary excludes t equal to traffic', ['exp1', 'm0', 14, [['a', 50], ['b', 50]]], [False, None]),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('full traffic control', ['exp2', 'n1', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 11', ['exp1', 'u2908', 30, [['a', 34], ['b', 33], ['c', 33]]], [False, None]),
('allocation sample 12', ['exp1', 'u1733', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 50', ['exp1', 'u2120', 100, [['control', 50], ['treatment', 50]]], [True, 'control'])],
[('low traffic still fills every arm',
['exp1', 'm0', 20, [['control', 50], ['treatment', 50]]],
[True, 'treatment']),
('low traffic second user', ['exp1', 'm5', 20, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('traffic boundary excludes t equal to traffic', ['exp1', 'm0', 14, [['a', 50], ['b', 50]]], [False, None]),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('full traffic control', ['exp2', 'n1', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 3', ['exp2', 'u2623', 67, [['control', 20], ['treatment', 80]]], [True, 'control']),
('allocation sample 9', ['exp1', 'u3147', 77, [['a', 34], ['b', 33], ['c', 33]]], [True, 'a']),
('allocation sample 16', ['exp2', 'u2262', 53, [['control', 20], ['treatment', 80]]], [True, 'treatment'])],
[('low traffic still fills every arm',
['exp1', 'm0', 20, [['control', 50], ['treatment', 50]]],
[True, 'treatment']),
('low traffic second user', ['exp1', 'm5', 20, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('arm weights are relative to allocated users',
['exp1', 'm8', 20, [['control', 20], ['treatment', 80]]],
[True, 'control']),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('full traffic control', ['exp2', 'n1', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 21', ['exp1', 'u843', 100, [['control', 20], ['treatment', 80]]], [True, 'treatment']),
('allocation sample 33', ['exp1', 'u1247', 50, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('allocation sample 41', ['exp2', 'u3401', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b'])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| low traffic still fills every arm | [True, 'treatment'] | [True, 'treatment'] | Passed |
| low traffic second user | [True, 'control'] | [True, 'treatment'] | Failed |
| arm weights are relative to allocated users | [True, 'treatment'] | [True, 'control'] | Failed |
| traffic boundary excludes t equal to traffic | [False, None] | [False, None] | Passed |
| traffic boundary includes t just below | [True, 'b'] | [True, 'b'] | Passed |
| allocation sample 1 | [False, None] | [False, None] | Passed |
| allocation sample 2 | [False, None] | [False, None] | Passed |
| allocation sample 3 | [True, 'treatment'] | [True, 'control'] | Failed |
SHA-256 / 3195063570dd44f13e223eabeedef7c9b300e63d01fe629be2b036c674fe013f
3 / The verified repair
Exit 0"""Failure Map reference implementation. Python standard library only."""
import json
import zlib
N = 1
observations = []
def solve(salt, user, traffic, weights):
t = zlib.crc32((salt + ':traffic:' + user).encode()) % 100
if t >= traffic:
return [False, None]
v = zlib.crc32((salt + ':variant:' + user).encode()) % 100
acc = 0
for arm, w in weights:
acc += w
if v < acc:
return [True, arm]
return [True, weights[-1][0]]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('low traffic still fills every arm',
['exp1', 'm0', 20, [['control', 50], ['treatment', 50]]],
[True, 'treatment']),
('low traffic second user', ['exp1', 'm5', 20, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('arm weights are relative to allocated users',
['exp1', 'm8', 20, [['control', 20], ['treatment', 80]]],
[True, 'control']),
('traffic boundary excludes t equal to traffic', ['exp1', 'm0', 14, [['a', 50], ['b', 50]]], [False, None]),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('allocation sample 1', ['exp2', 'u3390', 9, [['control', 20], ['treatment', 80]]], [False, None]),
('allocation sample 2', ['exp2', 'u2292', 50, [['control', 20], ['treatment', 80]]], [False, None]),
('allocation sample 3', ['exp2', 'u2623', 67, [['control', 20], ['treatment', 80]]], [True, 'control'])],
[('low traffic second user', ['exp1', 'm5', 20, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('arm weights are relative to allocated users',
['exp1', 'm8', 20, [['control', 20], ['treatment', 80]]],
[True, 'control']),
('traffic boundary excludes t equal to traffic', ['exp1', 'm0', 14, [['a', 50], ['b', 50]]], [False, None]),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('full traffic control', ['exp2', 'n1', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 6', ['exp1', 'u2423', 50, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 13', ['exp2', 'u3641', 50, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 20', ['exp1', 'u1715', 57, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b'])],
[('low traffic still fills every arm',
['exp1', 'm0', 20, [['control', 50], ['treatment', 50]]],
[True, 'treatment']),
('arm weights are relative to allocated users',
['exp1', 'm8', 20, [['control', 20], ['treatment', 80]]],
[True, 'control']),
('traffic boundary excludes t equal to traffic', ['exp1', 'm0', 14, [['a', 50], ['b', 50]]], [False, None]),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('full traffic control', ['exp2', 'n1', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 11', ['exp1', 'u2908', 30, [['a', 34], ['b', 33], ['c', 33]]], [False, None]),
('allocation sample 12', ['exp1', 'u1733', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 50', ['exp1', 'u2120', 100, [['control', 50], ['treatment', 50]]], [True, 'control'])],
[('low traffic still fills every arm',
['exp1', 'm0', 20, [['control', 50], ['treatment', 50]]],
[True, 'treatment']),
('low traffic second user', ['exp1', 'm5', 20, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('traffic boundary excludes t equal to traffic', ['exp1', 'm0', 14, [['a', 50], ['b', 50]]], [False, None]),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('full traffic control', ['exp2', 'n1', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 3', ['exp2', 'u2623', 67, [['control', 20], ['treatment', 80]]], [True, 'control']),
('allocation sample 9', ['exp1', 'u3147', 77, [['a', 34], ['b', 33], ['c', 33]]], [True, 'a']),
('allocation sample 16', ['exp2', 'u2262', 53, [['control', 20], ['treatment', 80]]], [True, 'treatment'])],
[('low traffic still fills every arm',
['exp1', 'm0', 20, [['control', 50], ['treatment', 50]]],
[True, 'treatment']),
('low traffic second user', ['exp1', 'm5', 20, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('arm weights are relative to allocated users',
['exp1', 'm8', 20, [['control', 20], ['treatment', 80]]],
[True, 'control']),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('full traffic control', ['exp2', 'n1', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 21', ['exp1', 'u843', 100, [['control', 20], ['treatment', 80]]], [True, 'treatment']),
('allocation sample 33', ['exp1', 'u1247', 50, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('allocation sample 41', ['exp2', 'u3401', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b'])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| low traffic still fills every arm | [True, 'treatment'] | [True, 'treatment'] | Passed |
| low traffic second user | [True, 'treatment'] | [True, 'treatment'] | Passed |
| arm weights are relative to allocated users | [True, 'control'] | [True, 'control'] | Passed |
| traffic boundary excludes t equal to traffic | [False, None] | [False, None] | Passed |
| traffic boundary includes t just below | [True, 'b'] | [True, 'b'] | Passed |
| allocation sample 1 | [False, None] | [False, None] | Passed |
| allocation sample 2 | [False, None] | [False, None] | Passed |
| allocation sample 3 | [True, 'control'] | [True, 'control'] | Passed |
SHA-256 / c75acb93634100b1778e2a191dc38b177a53717f1a9b348769a9b64b9785f959
Verification & scope
A deterministic toy flag-evaluation model with a stipulated contract; it does not reproduce any vendor SDK byte for byte. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.
Observations recorded using Python 3.12.14 at 2026-09-29T14:48:55.145352+00:00.
Case digest / e89933b899b9ae1ba768b82b78b93a81d67f8f4b25b64df293b5195e14a658bf