FA-74326 / Feature flag rollout bucketing / Open access
Per-context evaluation memo: Memo ignores the flag version · case 01
After a flag is edited, clients keep serving results computed from the old version.
ROOT CAUSE
The memo key omits the flag version.
VERIFIED REPAIR
Include the flag version in the memo key.
Unsuccessful approach: Dropping the flag name while adding the version lets different flags share results.
Case contract
calls are [flag, context dict of string attributes, flag version]. The toy evaluation is (sum of code points of flag + key + country) + version) % 3. Results are memoised by (flag, context contents independent of attribute order, version). Return [value, hit|miss] per call.
Why this case matters
Evaluation caches in SDKs must key on every input that can change the result.
1 / The failure
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(calls):
memo = {}
out = []
for flag, ctx, version in calls:
k = (flag, tuple(sorted(ctx.items())))
if k in memo:
out.append([memo[k], 'hit'])
continue
value = (sum(map(ord, flag + ctx.get('key', '') + ctx.get('country', ''))) + version) % 3
memo[k] = value
out.append([value, 'miss'])
return out
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('call sequence sample 1',
[[['f', {'key': 'ab', 'country': 'US'}, 2],
['g', {'country': 'US', 'key': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'DE'}, 1],
['g', {'country': 'ab', 'key': 'c'}, 1],
['g', {'country': 'US', 'key': 'ba'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 2]]],
[[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']]),
('call sequence sample 2',
[[['f', {'country': 'DE', 'key': 'ab'}, 1],
['f', {'key': 'ab', 'country': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'ab'}, 2],
['f', {'key': 'ba', 'country': 'ab'}, 2],
['f', {'key': 'ab', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'c'}, 1]]],
[[0, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 3',
[[['f', {'country': 'DE', 'key': 'c'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 2]]],
[[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 17',
[[['f', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'DE'}, 2],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['g', {'key': 'ab', 'country': 'DE'}, 1]]],
[[0, 'miss'], [1, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 35',
[[['g', {'country': 'DE', 'key': 'c'}, 1],
['f', {'key': 'c', 'country': 'ab'}, 1],
['f', {'country': 'US', 'key': 'ab'}, 2],
['f', {'key': 'c', 'country': 'US'}, 2],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'US'}, 1]]],
[[1, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 1',
[[['f', {'key': 'ab', 'country': 'US'}, 2],
['g', {'country': 'US', 'key': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'DE'}, 1],
['g', {'country': 'ab', 'key': 'c'}, 1],
['g', {'country': 'US', 'key': 'ba'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 2]]],
[[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']]),
('call sequence sample 11',
[[['f', {'key': 'ba', 'country': 'US'}, 2],
['g', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'key': 'ab', 'country': 'DE'}, 2],
['g', {'key': 'ba', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ab'}, 2]]],
[[2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'hit'], [0, 'miss']]),
('call sequence sample 44',
[[['g', {'key': 'ab', 'country': 'ab'}, 1],
['f', {'key': 'c', 'country': 'DE'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'country': 'ab', 'key': 'c'}, 1],
['g', {'key': 'c', 'country': 'US'}, 2],
['f', {'country': 'US', 'key': 'c'}, 2]]],
[[2, 'miss'], [0, 'miss'], [1, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 4',
[[['g', {'key': 'ba', 'country': 'ab'}, 1],
['f', {'key': 'ba', 'country': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ba'}, 2],
['g', {'key': 'ba', 'country': 'US'}, 1]]],
[[2, 'miss'], [2, 'miss'], [0, 'miss'], [2, 'miss']]),
('call sequence sample 16',
[[['g', {'country': 'ab', 'key': 'ba'}, 1],
['f', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 1]]],
[[2, 'miss'], [0, 'miss'], [2, 'miss']]),
('call sequence sample 39',
[[['f', {'key': 'ba', 'country': 'US'}, 2],
['f', {'key': 'ba', 'country': 'ab'}, 1],
['f', {'key': 'ba', 'country': 'US'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 1],
['f', {'country': 'DE', 'key': 'c'}, 1]]],
[[2, 'miss'], [1, 'miss'], [1, 'miss'], [2, 'miss'], [0, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 3',
[[['f', {'country': 'DE', 'key': 'c'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 2]]],
[[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']]),
('call sequence sample 21',
[[['g', {'key': 'c', 'country': 'US'}, 2],
['f', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'DE'}, 1],
['g', {'key': 'c', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1]]],
[[0, 'miss'], [0, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss']]),
('call sequence sample 27',
[[['f', {'country': 'ab', 'key': 'c'}, 1],
['f', {'key': 'ab', 'country': 'US'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 1],
['g', {'country': 'US', 'key': 'c'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 1],
['g', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'hit'], [0, 'miss']])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| version bump misses the memo | [[1, 'miss'], [1, 'hit']] | [[1, 'miss'], [2, 'miss']] | Failed |
| attribute change misses the memo | [[1, 'miss'], [0, 'miss']] | [[1, 'miss'], [0, 'miss']] | Passed |
| attribute order does not matter | [[1, 'miss'], [1, 'hit']] | [[1, 'miss'], [1, 'hit']] | Passed |
| swapped attribute values are different contexts | [[2, 'miss'], [2, 'miss']] | [[2, 'miss'], [2, 'miss']] | Passed |
| different flags do not share entries | [[1, 'miss'], [2, 'miss']] | [[1, 'miss'], [2, 'miss']] | Passed |
| call sequence sample 1 | [[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'hit']] | [[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | Failed |
| call sequence sample 2 | [[0, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']] | [[0, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']] | Passed |
| call sequence sample 3 | [[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'hit']] | [[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | Failed |
SHA-256 / 5e09eaf866b1421618dcf784058adc8331ccdcb34b08e2379ccfc75e3639d27f
2 / The unsuccessful fix
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(calls):
memo = {}
out = []
for flag, ctx, version in calls:
k = (tuple(sorted(ctx.items())), version)
if k in memo:
out.append([memo[k], 'hit'])
continue
value = (sum(map(ord, flag + ctx.get('key', '') + ctx.get('country', ''))) + version) % 3
memo[k] = value
out.append([value, 'miss'])
return out
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('call sequence sample 1',
[[['f', {'key': 'ab', 'country': 'US'}, 2],
['g', {'country': 'US', 'key': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'DE'}, 1],
['g', {'country': 'ab', 'key': 'c'}, 1],
['g', {'country': 'US', 'key': 'ba'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 2]]],
[[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']]),
('call sequence sample 2',
[[['f', {'country': 'DE', 'key': 'ab'}, 1],
['f', {'key': 'ab', 'country': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'ab'}, 2],
['f', {'key': 'ba', 'country': 'ab'}, 2],
['f', {'key': 'ab', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'c'}, 1]]],
[[0, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 3',
[[['f', {'country': 'DE', 'key': 'c'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 2]]],
[[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 17',
[[['f', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'DE'}, 2],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['g', {'key': 'ab', 'country': 'DE'}, 1]]],
[[0, 'miss'], [1, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 35',
[[['g', {'country': 'DE', 'key': 'c'}, 1],
['f', {'key': 'c', 'country': 'ab'}, 1],
['f', {'country': 'US', 'key': 'ab'}, 2],
['f', {'key': 'c', 'country': 'US'}, 2],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'US'}, 1]]],
[[1, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 1',
[[['f', {'key': 'ab', 'country': 'US'}, 2],
['g', {'country': 'US', 'key': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'DE'}, 1],
['g', {'country': 'ab', 'key': 'c'}, 1],
['g', {'country': 'US', 'key': 'ba'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 2]]],
[[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']]),
('call sequence sample 11',
[[['f', {'key': 'ba', 'country': 'US'}, 2],
['g', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'key': 'ab', 'country': 'DE'}, 2],
['g', {'key': 'ba', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ab'}, 2]]],
[[2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'hit'], [0, 'miss']]),
('call sequence sample 44',
[[['g', {'key': 'ab', 'country': 'ab'}, 1],
['f', {'key': 'c', 'country': 'DE'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'country': 'ab', 'key': 'c'}, 1],
['g', {'key': 'c', 'country': 'US'}, 2],
['f', {'country': 'US', 'key': 'c'}, 2]]],
[[2, 'miss'], [0, 'miss'], [1, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 4',
[[['g', {'key': 'ba', 'country': 'ab'}, 1],
['f', {'key': 'ba', 'country': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ba'}, 2],
['g', {'key': 'ba', 'country': 'US'}, 1]]],
[[2, 'miss'], [2, 'miss'], [0, 'miss'], [2, 'miss']]),
('call sequence sample 16',
[[['g', {'country': 'ab', 'key': 'ba'}, 1],
['f', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 1]]],
[[2, 'miss'], [0, 'miss'], [2, 'miss']]),
('call sequence sample 39',
[[['f', {'key': 'ba', 'country': 'US'}, 2],
['f', {'key': 'ba', 'country': 'ab'}, 1],
['f', {'key': 'ba', 'country': 'US'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 1],
['f', {'country': 'DE', 'key': 'c'}, 1]]],
[[2, 'miss'], [1, 'miss'], [1, 'miss'], [2, 'miss'], [0, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 3',
[[['f', {'country': 'DE', 'key': 'c'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 2]]],
[[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']]),
('call sequence sample 21',
[[['g', {'key': 'c', 'country': 'US'}, 2],
['f', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'DE'}, 1],
['g', {'key': 'c', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1]]],
[[0, 'miss'], [0, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss']]),
('call sequence sample 27',
[[['f', {'country': 'ab', 'key': 'c'}, 1],
['f', {'key': 'ab', 'country': 'US'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 1],
['g', {'country': 'US', 'key': 'c'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 1],
['g', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'hit'], [0, 'miss']])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| version bump misses the memo | [[1, 'miss'], [2, 'miss']] | [[1, 'miss'], [2, 'miss']] | Passed |
| attribute change misses the memo | [[1, 'miss'], [0, 'miss']] | [[1, 'miss'], [0, 'miss']] | Passed |
| attribute order does not matter | [[1, 'miss'], [1, 'hit']] | [[1, 'miss'], [1, 'hit']] | Passed |
| swapped attribute values are different contexts | [[2, 'miss'], [2, 'miss']] | [[2, 'miss'], [2, 'miss']] | Passed |
| different flags do not share entries | [[1, 'miss'], [1, 'hit']] | [[1, 'miss'], [2, 'miss']] | Failed |
| call sequence sample 1 | [[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'hit']] | [[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | Failed |
| call sequence sample 2 | [[0, 'miss'], [1, 'miss'], [0, 'miss'], [0, 'hit'], [1, 'miss'], [1, 'miss']] | [[0, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']] | Failed |
| call sequence sample 3 | [[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'hit']] | [[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | Failed |
SHA-256 / 6420e90b478d15da19360fc404bae3c31bfed6a305e1c2d20c9efd182c51f413
3 / The verified repair
Exit 0"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(calls):
memo = {}
out = []
for flag, ctx, version in calls:
k = (flag, tuple(sorted(ctx.items())), version)
if k in memo:
out.append([memo[k], 'hit'])
continue
value = (sum(map(ord, flag + ctx.get('key', '') + ctx.get('country', ''))) + version) % 3
memo[k] = value
out.append([value, 'miss'])
return out
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('call sequence sample 1',
[[['f', {'key': 'ab', 'country': 'US'}, 2],
['g', {'country': 'US', 'key': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'DE'}, 1],
['g', {'country': 'ab', 'key': 'c'}, 1],
['g', {'country': 'US', 'key': 'ba'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 2]]],
[[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']]),
('call sequence sample 2',
[[['f', {'country': 'DE', 'key': 'ab'}, 1],
['f', {'key': 'ab', 'country': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'ab'}, 2],
['f', {'key': 'ba', 'country': 'ab'}, 2],
['f', {'key': 'ab', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'c'}, 1]]],
[[0, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 3',
[[['f', {'country': 'DE', 'key': 'c'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 2]]],
[[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 17',
[[['f', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'DE'}, 2],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['g', {'key': 'ab', 'country': 'DE'}, 1]]],
[[0, 'miss'], [1, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 35',
[[['g', {'country': 'DE', 'key': 'c'}, 1],
['f', {'key': 'c', 'country': 'ab'}, 1],
['f', {'country': 'US', 'key': 'ab'}, 2],
['f', {'key': 'c', 'country': 'US'}, 2],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'US'}, 1]]],
[[1, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 1',
[[['f', {'key': 'ab', 'country': 'US'}, 2],
['g', {'country': 'US', 'key': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'DE'}, 1],
['g', {'country': 'ab', 'key': 'c'}, 1],
['g', {'country': 'US', 'key': 'ba'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 2]]],
[[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']]),
('call sequence sample 11',
[[['f', {'key': 'ba', 'country': 'US'}, 2],
['g', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'key': 'ab', 'country': 'DE'}, 2],
['g', {'key': 'ba', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ab'}, 2]]],
[[2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'hit'], [0, 'miss']]),
('call sequence sample 44',
[[['g', {'key': 'ab', 'country': 'ab'}, 1],
['f', {'key': 'c', 'country': 'DE'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'country': 'ab', 'key': 'c'}, 1],
['g', {'key': 'c', 'country': 'US'}, 2],
['f', {'country': 'US', 'key': 'c'}, 2]]],
[[2, 'miss'], [0, 'miss'], [1, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 4',
[[['g', {'key': 'ba', 'country': 'ab'}, 1],
['f', {'key': 'ba', 'country': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ba'}, 2],
['g', {'key': 'ba', 'country': 'US'}, 1]]],
[[2, 'miss'], [2, 'miss'], [0, 'miss'], [2, 'miss']]),
('call sequence sample 16',
[[['g', {'country': 'ab', 'key': 'ba'}, 1],
['f', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 1]]],
[[2, 'miss'], [0, 'miss'], [2, 'miss']]),
('call sequence sample 39',
[[['f', {'key': 'ba', 'country': 'US'}, 2],
['f', {'key': 'ba', 'country': 'ab'}, 1],
['f', {'key': 'ba', 'country': 'US'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 1],
['f', {'country': 'DE', 'key': 'c'}, 1]]],
[[2, 'miss'], [1, 'miss'], [1, 'miss'], [2, 'miss'], [0, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 3',
[[['f', {'country': 'DE', 'key': 'c'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 2]]],
[[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']]),
('call sequence sample 21',
[[['g', {'key': 'c', 'country': 'US'}, 2],
['f', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'DE'}, 1],
['g', {'key': 'c', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1]]],
[[0, 'miss'], [0, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss']]),
('call sequence sample 27',
[[['f', {'country': 'ab', 'key': 'c'}, 1],
['f', {'key': 'ab', 'country': 'US'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 1],
['g', {'country': 'US', 'key': 'c'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 1],
['g', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'hit'], [0, 'miss']])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| version bump misses the memo | [[1, 'miss'], [2, 'miss']] | [[1, 'miss'], [2, 'miss']] | Passed |
| attribute change misses the memo | [[1, 'miss'], [0, 'miss']] | [[1, 'miss'], [0, 'miss']] | Passed |
| attribute order does not matter | [[1, 'miss'], [1, 'hit']] | [[1, 'miss'], [1, 'hit']] | Passed |
| swapped attribute values are different contexts | [[2, 'miss'], [2, 'miss']] | [[2, 'miss'], [2, 'miss']] | Passed |
| different flags do not share entries | [[1, 'miss'], [2, 'miss']] | [[1, 'miss'], [2, 'miss']] | Passed |
| call sequence sample 1 | [[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | [[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | Passed |
| call sequence sample 2 | [[0, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']] | [[0, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']] | Passed |
| call sequence sample 3 | [[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | [[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | Passed |
SHA-256 / 0e2437494fc8b62a761bd1d7ec4f43140cf8af06e656344630a5e5d1b969123b
Verification & scope
A deterministic toy flag-evaluation model with a stipulated contract; it does not reproduce any vendor SDK byte for byte. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.
Observations recorded using Python 3.12.14 at 2026-09-29T14:48:55.697257+00:00.
Case digest / d5e756e6f7a82c9ec5bafb448f49b9ce017ec9b39798c7535d41a635989bae7d