FA-74276 / Feature flag rollout bucketing / Open access
Traffic then variant allocation: Arm weights are scaled by traffic · case 01
At partial traffic most allocated users fall through to the last arm.
ROOT CAUSE
Cumulative weights are multiplied by traffic / 100 although they are shares of allocated users.
VERIFIED REPAIR
Accumulate the raw arm percentages.
Unsuccessful approach: Dividing by traffic inflates early arms so later arms starve.
Case contract
Stage 1: t = crc32(salt + ":traffic:" + user) % 100; users with t >= traffic (1..100) are not in the experiment. Stage 2 uses an independent v = crc32(salt + ":variant:" + user) % 100 and walks weights ([arm, percent of allocated users], summing to 100) cumulatively, half-open. Return [in_experiment, arm or None].
Why this case matters
Separating traffic allocation from the variant split keeps arms balanced when traffic changes.
1 / The failure
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
import zlib
N = 1
observations = []
def solve(salt, user, traffic, weights):
t = zlib.crc32((salt + ':traffic:' + user).encode()) % 100
if t >= traffic:
return [False, None]
v = zlib.crc32((salt + ':variant:' + user).encode()) % 100
acc = 0
for arm, w in weights:
acc += w * traffic / 100
if v < acc:
return [True, arm]
return [True, weights[-1][0]]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('low traffic still fills every arm',
['exp1', 'm0', 20, [['control', 50], ['treatment', 50]]],
[True, 'treatment']),
('low traffic second user', ['exp1', 'm5', 20, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('arm weights are relative to allocated users',
['exp1', 'm8', 20, [['control', 20], ['treatment', 80]]],
[True, 'control']),
('traffic boundary excludes t equal to traffic', ['exp1', 'm0', 14, [['a', 50], ['b', 50]]], [False, None]),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('allocation sample 1', ['exp2', 'u3390', 9, [['control', 20], ['treatment', 80]]], [False, None]),
('allocation sample 2', ['exp2', 'u2292', 50, [['control', 20], ['treatment', 80]]], [False, None]),
('allocation sample 6', ['exp1', 'u2423', 50, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b'])],
[('low traffic second user', ['exp1', 'm5', 20, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('arm weights are relative to allocated users',
['exp1', 'm8', 20, [['control', 20], ['treatment', 80]]],
[True, 'control']),
('traffic boundary excludes t equal to traffic', ['exp1', 'm0', 14, [['a', 50], ['b', 50]]], [False, None]),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('full traffic control', ['exp2', 'n1', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 6', ['exp1', 'u2423', 50, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 7', ['exp1', 'u3237', 10, [['control', 50], ['treatment', 50]]], [False, None]),
('allocation sample 40', ['exp2', 'u3484', 50, [['control', 50], ['treatment', 50]]], [True, 'treatment'])],
[('low traffic still fills every arm',
['exp1', 'm0', 20, [['control', 50], ['treatment', 50]]],
[True, 'treatment']),
('arm weights are relative to allocated users',
['exp1', 'm8', 20, [['control', 20], ['treatment', 80]]],
[True, 'control']),
('traffic boundary excludes t equal to traffic', ['exp1', 'm0', 14, [['a', 50], ['b', 50]]], [False, None]),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('full traffic control', ['exp2', 'n1', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 6', ['exp1', 'u2423', 50, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 11', ['exp1', 'u2908', 30, [['a', 34], ['b', 33], ['c', 33]]], [False, None]),
('allocation sample 13', ['exp2', 'u3641', 50, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b'])],
[('low traffic still fills every arm',
['exp1', 'm0', 20, [['control', 50], ['treatment', 50]]],
[True, 'treatment']),
('low traffic second user', ['exp1', 'm5', 20, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('traffic boundary excludes t equal to traffic', ['exp1', 'm0', 14, [['a', 50], ['b', 50]]], [False, None]),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('full traffic control', ['exp2', 'n1', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 6', ['exp1', 'u2423', 50, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 16', ['exp2', 'u2262', 53, [['control', 20], ['treatment', 80]]], [True, 'treatment']),
('allocation sample 17', ['exp2', 'u276', 10, [['a', 34], ['b', 33], ['c', 33]]], [False, None])],
[('low traffic still fills every arm',
['exp1', 'm0', 20, [['control', 50], ['treatment', 50]]],
[True, 'treatment']),
('low traffic second user', ['exp1', 'm5', 20, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('arm weights are relative to allocated users',
['exp1', 'm8', 20, [['control', 20], ['treatment', 80]]],
[True, 'control']),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('full traffic control', ['exp2', 'n1', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 6', ['exp1', 'u2423', 50, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 21', ['exp1', 'u843', 100, [['control', 20], ['treatment', 80]]], [True, 'treatment']),
('allocation sample 44', ['exp2', 'u1337', 52, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b'])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| low traffic still fills every arm | [True, 'treatment'] | [True, 'treatment'] | Passed |
| low traffic second user | [True, 'treatment'] | [True, 'treatment'] | Passed |
| arm weights are relative to allocated users | [True, 'control'] | [True, 'control'] | Passed |
| traffic boundary excludes t equal to traffic | [False, None] | [False, None] | Passed |
| traffic boundary includes t just below | [True, 'b'] | [True, 'b'] | Passed |
| allocation sample 1 | [False, None] | [False, None] | Passed |
| allocation sample 2 | [False, None] | [False, None] | Passed |
| allocation sample 6 | [True, 'c'] | [True, 'b'] | Failed |
SHA-256 / 1ee2f17d6bb5c9b5a9aa94635c010b05b4941a40e2514b7aa921acdda76fe1ce
2 / The unsuccessful fix
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
import zlib
N = 1
observations = []
def solve(salt, user, traffic, weights):
t = zlib.crc32((salt + ':traffic:' + user).encode()) % 100
if t >= traffic:
return [False, None]
v = zlib.crc32((salt + ':variant:' + user).encode()) % 100
acc = 0
for arm, w in weights:
acc += w * 100 / traffic
if v < acc:
return [True, arm]
return [True, weights[-1][0]]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('low traffic still fills every arm',
['exp1', 'm0', 20, [['control', 50], ['treatment', 50]]],
[True, 'treatment']),
('low traffic second user', ['exp1', 'm5', 20, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('arm weights are relative to allocated users',
['exp1', 'm8', 20, [['control', 20], ['treatment', 80]]],
[True, 'control']),
('traffic boundary excludes t equal to traffic', ['exp1', 'm0', 14, [['a', 50], ['b', 50]]], [False, None]),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('allocation sample 1', ['exp2', 'u3390', 9, [['control', 20], ['treatment', 80]]], [False, None]),
('allocation sample 2', ['exp2', 'u2292', 50, [['control', 20], ['treatment', 80]]], [False, None]),
('allocation sample 6', ['exp1', 'u2423', 50, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b'])],
[('low traffic second user', ['exp1', 'm5', 20, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('arm weights are relative to allocated users',
['exp1', 'm8', 20, [['control', 20], ['treatment', 80]]],
[True, 'control']),
('traffic boundary excludes t equal to traffic', ['exp1', 'm0', 14, [['a', 50], ['b', 50]]], [False, None]),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('full traffic control', ['exp2', 'n1', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 6', ['exp1', 'u2423', 50, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 7', ['exp1', 'u3237', 10, [['control', 50], ['treatment', 50]]], [False, None]),
('allocation sample 40', ['exp2', 'u3484', 50, [['control', 50], ['treatment', 50]]], [True, 'treatment'])],
[('low traffic still fills every arm',
['exp1', 'm0', 20, [['control', 50], ['treatment', 50]]],
[True, 'treatment']),
('arm weights are relative to allocated users',
['exp1', 'm8', 20, [['control', 20], ['treatment', 80]]],
[True, 'control']),
('traffic boundary excludes t equal to traffic', ['exp1', 'm0', 14, [['a', 50], ['b', 50]]], [False, None]),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('full traffic control', ['exp2', 'n1', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 6', ['exp1', 'u2423', 50, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 11', ['exp1', 'u2908', 30, [['a', 34], ['b', 33], ['c', 33]]], [False, None]),
('allocation sample 13', ['exp2', 'u3641', 50, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b'])],
[('low traffic still fills every arm',
['exp1', 'm0', 20, [['control', 50], ['treatment', 50]]],
[True, 'treatment']),
('low traffic second user', ['exp1', 'm5', 20, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('traffic boundary excludes t equal to traffic', ['exp1', 'm0', 14, [['a', 50], ['b', 50]]], [False, None]),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('full traffic control', ['exp2', 'n1', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 6', ['exp1', 'u2423', 50, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 16', ['exp2', 'u2262', 53, [['control', 20], ['treatment', 80]]], [True, 'treatment']),
('allocation sample 17', ['exp2', 'u276', 10, [['a', 34], ['b', 33], ['c', 33]]], [False, None])],
[('low traffic still fills every arm',
['exp1', 'm0', 20, [['control', 50], ['treatment', 50]]],
[True, 'treatment']),
('low traffic second user', ['exp1', 'm5', 20, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('arm weights are relative to allocated users',
['exp1', 'm8', 20, [['control', 20], ['treatment', 80]]],
[True, 'control']),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('full traffic control', ['exp2', 'n1', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 6', ['exp1', 'u2423', 50, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 21', ['exp1', 'u843', 100, [['control', 20], ['treatment', 80]]], [True, 'treatment']),
('allocation sample 44', ['exp2', 'u1337', 52, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b'])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| low traffic still fills every arm | [True, 'control'] | [True, 'treatment'] | Failed |
| low traffic second user | [True, 'control'] | [True, 'treatment'] | Failed |
| arm weights are relative to allocated users | [True, 'control'] | [True, 'control'] | Passed |
| traffic boundary excludes t equal to traffic | [False, None] | [False, None] | Passed |
| traffic boundary includes t just below | [True, 'a'] | [True, 'b'] | Failed |
| allocation sample 1 | [False, None] | [False, None] | Passed |
| allocation sample 2 | [False, None] | [False, None] | Passed |
| allocation sample 6 | [True, 'a'] | [True, 'b'] | Failed |
SHA-256 / bafd68f54857d23e4d59ff00433c52fef9a9aea7de0e8a95c6bca25952eb8bfc
3 / The verified repair
Exit 0"""Failure Map reference implementation. Python standard library only."""
import json
import zlib
N = 1
observations = []
def solve(salt, user, traffic, weights):
t = zlib.crc32((salt + ':traffic:' + user).encode()) % 100
if t >= traffic:
return [False, None]
v = zlib.crc32((salt + ':variant:' + user).encode()) % 100
acc = 0
for arm, w in weights:
acc += w
if v < acc:
return [True, arm]
return [True, weights[-1][0]]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('low traffic still fills every arm',
['exp1', 'm0', 20, [['control', 50], ['treatment', 50]]],
[True, 'treatment']),
('low traffic second user', ['exp1', 'm5', 20, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('arm weights are relative to allocated users',
['exp1', 'm8', 20, [['control', 20], ['treatment', 80]]],
[True, 'control']),
('traffic boundary excludes t equal to traffic', ['exp1', 'm0', 14, [['a', 50], ['b', 50]]], [False, None]),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('allocation sample 1', ['exp2', 'u3390', 9, [['control', 20], ['treatment', 80]]], [False, None]),
('allocation sample 2', ['exp2', 'u2292', 50, [['control', 20], ['treatment', 80]]], [False, None]),
('allocation sample 6', ['exp1', 'u2423', 50, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b'])],
[('low traffic second user', ['exp1', 'm5', 20, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('arm weights are relative to allocated users',
['exp1', 'm8', 20, [['control', 20], ['treatment', 80]]],
[True, 'control']),
('traffic boundary excludes t equal to traffic', ['exp1', 'm0', 14, [['a', 50], ['b', 50]]], [False, None]),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('full traffic control', ['exp2', 'n1', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 6', ['exp1', 'u2423', 50, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 7', ['exp1', 'u3237', 10, [['control', 50], ['treatment', 50]]], [False, None]),
('allocation sample 40', ['exp2', 'u3484', 50, [['control', 50], ['treatment', 50]]], [True, 'treatment'])],
[('low traffic still fills every arm',
['exp1', 'm0', 20, [['control', 50], ['treatment', 50]]],
[True, 'treatment']),
('arm weights are relative to allocated users',
['exp1', 'm8', 20, [['control', 20], ['treatment', 80]]],
[True, 'control']),
('traffic boundary excludes t equal to traffic', ['exp1', 'm0', 14, [['a', 50], ['b', 50]]], [False, None]),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('full traffic control', ['exp2', 'n1', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 6', ['exp1', 'u2423', 50, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 11', ['exp1', 'u2908', 30, [['a', 34], ['b', 33], ['c', 33]]], [False, None]),
('allocation sample 13', ['exp2', 'u3641', 50, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b'])],
[('low traffic still fills every arm',
['exp1', 'm0', 20, [['control', 50], ['treatment', 50]]],
[True, 'treatment']),
('low traffic second user', ['exp1', 'm5', 20, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('traffic boundary excludes t equal to traffic', ['exp1', 'm0', 14, [['a', 50], ['b', 50]]], [False, None]),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('full traffic control', ['exp2', 'n1', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 6', ['exp1', 'u2423', 50, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 16', ['exp2', 'u2262', 53, [['control', 20], ['treatment', 80]]], [True, 'treatment']),
('allocation sample 17', ['exp2', 'u276', 10, [['a', 34], ['b', 33], ['c', 33]]], [False, None])],
[('low traffic still fills every arm',
['exp1', 'm0', 20, [['control', 50], ['treatment', 50]]],
[True, 'treatment']),
('low traffic second user', ['exp1', 'm5', 20, [['control', 50], ['treatment', 50]]], [True, 'treatment']),
('arm weights are relative to allocated users',
['exp1', 'm8', 20, [['control', 20], ['treatment', 80]]],
[True, 'control']),
('traffic boundary includes t just below', ['exp1', 'm5', 6, [['a', 50], ['b', 50]]], [True, 'b']),
('full traffic control', ['exp2', 'n1', 100, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 6', ['exp1', 'u2423', 50, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b']),
('allocation sample 21', ['exp1', 'u843', 100, [['control', 20], ['treatment', 80]]], [True, 'treatment']),
('allocation sample 44', ['exp2', 'u1337', 52, [['a', 34], ['b', 33], ['c', 33]]], [True, 'b'])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| low traffic still fills every arm | [True, 'treatment'] | [True, 'treatment'] | Passed |
| low traffic second user | [True, 'treatment'] | [True, 'treatment'] | Passed |
| arm weights are relative to allocated users | [True, 'control'] | [True, 'control'] | Passed |
| traffic boundary excludes t equal to traffic | [False, None] | [False, None] | Passed |
| traffic boundary includes t just below | [True, 'b'] | [True, 'b'] | Passed |
| allocation sample 1 | [False, None] | [False, None] | Passed |
| allocation sample 2 | [False, None] | [False, None] | Passed |
| allocation sample 6 | [True, 'b'] | [True, 'b'] | Passed |
SHA-256 / 161d33e25b571ef3943b263ca9f4b11f1442d2e1f9664d9071bb3766d930b396
Verification & scope
A deterministic toy flag-evaluation model with a stipulated contract; it does not reproduce any vendor SDK byte for byte. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.
Observations recorded using Python 3.12.14 at 2026-09-29T14:48:55.188408+00:00.
Case digest / dce3bb1838b99b740e629916fbf752364d9b80e67e94c58c8f1c422acf42af51