FA-74336 / Feature flag rollout bucketing / Open access
Per-context evaluation memo: Memo depends on attribute insertion order · case 01
Identical contexts built in different orders miss the cache and double evaluation counts.
ROOT CAUSE
The memo key is tuple(ctx.items()) without sorting.
VERIFIED REPAIR
Sort the attribute items before building the key.
Unsuccessful approach: Sorting only the values loses which attribute each value belongs to.
Case contract
calls are [flag, context dict of string attributes, flag version]. The toy evaluation is (sum of code points of flag + key + country) + version) % 3. Results are memoised by (flag, context contents independent of attribute order, version). Return [value, hit|miss] per call.
Why this case matters
Evaluation caches in SDKs must key on every input that can change the result.
1 / The failure
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(calls):
memo = {}
out = []
for flag, ctx, version in calls:
k = (flag, tuple(ctx.items()), version)
if k in memo:
out.append([memo[k], 'hit'])
continue
value = (sum(map(ord, flag + ctx.get('key', '') + ctx.get('country', ''))) + version) % 3
memo[k] = value
out.append([value, 'miss'])
return out
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('call sequence sample 1',
[[['f', {'key': 'ab', 'country': 'US'}, 2],
['g', {'country': 'US', 'key': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'DE'}, 1],
['g', {'country': 'ab', 'key': 'c'}, 1],
['g', {'country': 'US', 'key': 'ba'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 2]]],
[[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']]),
('call sequence sample 2',
[[['f', {'country': 'DE', 'key': 'ab'}, 1],
['f', {'key': 'ab', 'country': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'ab'}, 2],
['f', {'key': 'ba', 'country': 'ab'}, 2],
['f', {'key': 'ab', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'c'}, 1]]],
[[0, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 3',
[[['f', {'country': 'DE', 'key': 'c'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 2]]],
[[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']])],
[('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 6',
[[['f', {'country': 'ab', 'key': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 7',
[[['f', {'key': 'ab', 'country': 'ab'}, 2],
['g', {'key': 'ba', 'country': 'DE'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['g', {'key': 'ab', 'country': 'ab'}, 2],
['f', {'key': 'ba', 'country': 'ab'}, 1],
['f', {'country': 'ab', 'key': 'c'}, 1]]],
[[2, 'miss'], [1, 'miss'], [1, 'hit'], [0, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 59',
[[['g', {'key': 'ba', 'country': 'US'}, 1],
['f', {'key': 'c', 'country': 'ab'}, 1],
['f', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'country': 'ab', 'key': 'c'}, 1]]],
[[2, 'miss'], [1, 'miss'], [0, 'miss'], [1, 'hit']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 11',
[[['f', {'key': 'ba', 'country': 'US'}, 2],
['g', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'key': 'ab', 'country': 'DE'}, 2],
['g', {'key': 'ba', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ab'}, 2]]],
[[2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'hit'], [0, 'miss']]),
('call sequence sample 12',
[[['g', {'country': 'US', 'key': 'ba'}, 2], ['f', {'country': 'DE', 'key': 'ba'}, 1]]],
[[0, 'miss'], [0, 'miss']]),
('call sequence sample 13',
[[['g', {'country': 'DE', 'key': 'ba'}, 2],
['g', {'key': 'ab', 'country': 'ab'}, 1],
['g', {'key': 'ab', 'country': 'ab'}, 2],
['f', {'country': 'US', 'key': 'c'}, 1],
['f', {'country': 'ab', 'key': 'c'}, 2]]],
[[2, 'miss'], [2, 'miss'], [0, 'miss'], [1, 'miss'], [2, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 7',
[[['f', {'key': 'ab', 'country': 'ab'}, 2],
['g', {'key': 'ba', 'country': 'DE'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['g', {'key': 'ab', 'country': 'ab'}, 2],
['f', {'key': 'ba', 'country': 'ab'}, 1],
['f', {'country': 'ab', 'key': 'c'}, 1]]],
[[2, 'miss'], [1, 'miss'], [1, 'hit'], [0, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 16',
[[['g', {'country': 'ab', 'key': 'ba'}, 1],
['f', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 1]]],
[[2, 'miss'], [0, 'miss'], [2, 'miss']]),
('call sequence sample 17',
[[['f', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'DE'}, 2],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['g', {'key': 'ab', 'country': 'DE'}, 1]]],
[[0, 'miss'], [1, 'miss'], [1, 'miss'], [1, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 21',
[[['g', {'key': 'c', 'country': 'US'}, 2],
['f', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'DE'}, 1],
['g', {'key': 'c', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1]]],
[[0, 'miss'], [0, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss']]),
('call sequence sample 22',
[[['g', {'key': 'ab', 'country': 'DE'}, 1],
['f', {'country': 'ab', 'key': 'ba'}, 1],
['g', {'country': 'ab', 'key': 'ab'}, 2],
['f', {'country': 'US', 'key': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'DE'}, 1],
['f', {'country': 'ab', 'key': 'c'}, 2]]],
[[1, 'miss'], [1, 'miss'], [0, 'miss'], [1, 'miss'], [1, 'miss'], [2, 'miss']])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| version bump misses the memo | [[1, 'miss'], [2, 'miss']] | [[1, 'miss'], [2, 'miss']] | Passed |
| attribute change misses the memo | [[1, 'miss'], [0, 'miss']] | [[1, 'miss'], [0, 'miss']] | Passed |
| attribute order does not matter | [[1, 'miss'], [1, 'miss']] | [[1, 'miss'], [1, 'hit']] | Failed |
| swapped attribute values are different contexts | [[2, 'miss'], [2, 'miss']] | [[2, 'miss'], [2, 'miss']] | Passed |
| different flags do not share entries | [[1, 'miss'], [2, 'miss']] | [[1, 'miss'], [2, 'miss']] | Passed |
| call sequence sample 1 | [[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | [[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | Passed |
| call sequence sample 2 | [[0, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']] | [[0, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']] | Passed |
| call sequence sample 3 | [[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | [[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | Passed |
SHA-256 / 2bbdf8b73b0b317aba102b7aa8d5dbb79dd27a02fd5cddf6c95048f32bcc91fa
2 / The unsuccessful fix
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(calls):
memo = {}
out = []
for flag, ctx, version in calls:
k = (flag, tuple(sorted(ctx.values())), version)
if k in memo:
out.append([memo[k], 'hit'])
continue
value = (sum(map(ord, flag + ctx.get('key', '') + ctx.get('country', ''))) + version) % 3
memo[k] = value
out.append([value, 'miss'])
return out
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('call sequence sample 1',
[[['f', {'key': 'ab', 'country': 'US'}, 2],
['g', {'country': 'US', 'key': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'DE'}, 1],
['g', {'country': 'ab', 'key': 'c'}, 1],
['g', {'country': 'US', 'key': 'ba'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 2]]],
[[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']]),
('call sequence sample 2',
[[['f', {'country': 'DE', 'key': 'ab'}, 1],
['f', {'key': 'ab', 'country': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'ab'}, 2],
['f', {'key': 'ba', 'country': 'ab'}, 2],
['f', {'key': 'ab', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'c'}, 1]]],
[[0, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 3',
[[['f', {'country': 'DE', 'key': 'c'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 2]]],
[[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']])],
[('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 6',
[[['f', {'country': 'ab', 'key': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 7',
[[['f', {'key': 'ab', 'country': 'ab'}, 2],
['g', {'key': 'ba', 'country': 'DE'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['g', {'key': 'ab', 'country': 'ab'}, 2],
['f', {'key': 'ba', 'country': 'ab'}, 1],
['f', {'country': 'ab', 'key': 'c'}, 1]]],
[[2, 'miss'], [1, 'miss'], [1, 'hit'], [0, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 59',
[[['g', {'key': 'ba', 'country': 'US'}, 1],
['f', {'key': 'c', 'country': 'ab'}, 1],
['f', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'country': 'ab', 'key': 'c'}, 1]]],
[[2, 'miss'], [1, 'miss'], [0, 'miss'], [1, 'hit']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 11',
[[['f', {'key': 'ba', 'country': 'US'}, 2],
['g', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'key': 'ab', 'country': 'DE'}, 2],
['g', {'key': 'ba', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ab'}, 2]]],
[[2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'hit'], [0, 'miss']]),
('call sequence sample 12',
[[['g', {'country': 'US', 'key': 'ba'}, 2], ['f', {'country': 'DE', 'key': 'ba'}, 1]]],
[[0, 'miss'], [0, 'miss']]),
('call sequence sample 13',
[[['g', {'country': 'DE', 'key': 'ba'}, 2],
['g', {'key': 'ab', 'country': 'ab'}, 1],
['g', {'key': 'ab', 'country': 'ab'}, 2],
['f', {'country': 'US', 'key': 'c'}, 1],
['f', {'country': 'ab', 'key': 'c'}, 2]]],
[[2, 'miss'], [2, 'miss'], [0, 'miss'], [1, 'miss'], [2, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 7',
[[['f', {'key': 'ab', 'country': 'ab'}, 2],
['g', {'key': 'ba', 'country': 'DE'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['g', {'key': 'ab', 'country': 'ab'}, 2],
['f', {'key': 'ba', 'country': 'ab'}, 1],
['f', {'country': 'ab', 'key': 'c'}, 1]]],
[[2, 'miss'], [1, 'miss'], [1, 'hit'], [0, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 16',
[[['g', {'country': 'ab', 'key': 'ba'}, 1],
['f', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 1]]],
[[2, 'miss'], [0, 'miss'], [2, 'miss']]),
('call sequence sample 17',
[[['f', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'DE'}, 2],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['g', {'key': 'ab', 'country': 'DE'}, 1]]],
[[0, 'miss'], [1, 'miss'], [1, 'miss'], [1, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 21',
[[['g', {'key': 'c', 'country': 'US'}, 2],
['f', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'DE'}, 1],
['g', {'key': 'c', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1]]],
[[0, 'miss'], [0, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss']]),
('call sequence sample 22',
[[['g', {'key': 'ab', 'country': 'DE'}, 1],
['f', {'country': 'ab', 'key': 'ba'}, 1],
['g', {'country': 'ab', 'key': 'ab'}, 2],
['f', {'country': 'US', 'key': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'DE'}, 1],
['f', {'country': 'ab', 'key': 'c'}, 2]]],
[[1, 'miss'], [1, 'miss'], [0, 'miss'], [1, 'miss'], [1, 'miss'], [2, 'miss']])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| version bump misses the memo | [[1, 'miss'], [2, 'miss']] | [[1, 'miss'], [2, 'miss']] | Passed |
| attribute change misses the memo | [[1, 'miss'], [0, 'miss']] | [[1, 'miss'], [0, 'miss']] | Passed |
| attribute order does not matter | [[1, 'miss'], [1, 'hit']] | [[1, 'miss'], [1, 'hit']] | Passed |
| swapped attribute values are different contexts | [[2, 'miss'], [2, 'hit']] | [[2, 'miss'], [2, 'miss']] | Failed |
| different flags do not share entries | [[1, 'miss'], [2, 'miss']] | [[1, 'miss'], [2, 'miss']] | Passed |
| call sequence sample 1 | [[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | [[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | Passed |
| call sequence sample 2 | [[0, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']] | [[0, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']] | Passed |
| call sequence sample 3 | [[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | [[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | Passed |
SHA-256 / e46b23e81969bd6806c7770a60f71891b5795c014504cb3a73c10b2bf24dede2
3 / The verified repair
Exit 0"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(calls):
memo = {}
out = []
for flag, ctx, version in calls:
k = (flag, tuple(sorted(ctx.items())), version)
if k in memo:
out.append([memo[k], 'hit'])
continue
value = (sum(map(ord, flag + ctx.get('key', '') + ctx.get('country', ''))) + version) % 3
memo[k] = value
out.append([value, 'miss'])
return out
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('call sequence sample 1',
[[['f', {'key': 'ab', 'country': 'US'}, 2],
['g', {'country': 'US', 'key': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'DE'}, 1],
['g', {'country': 'ab', 'key': 'c'}, 1],
['g', {'country': 'US', 'key': 'ba'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 2]]],
[[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']]),
('call sequence sample 2',
[[['f', {'country': 'DE', 'key': 'ab'}, 1],
['f', {'key': 'ab', 'country': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'ab'}, 2],
['f', {'key': 'ba', 'country': 'ab'}, 2],
['f', {'key': 'ab', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'c'}, 1]]],
[[0, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 3',
[[['f', {'country': 'DE', 'key': 'c'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 2]]],
[[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']])],
[('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 6',
[[['f', {'country': 'ab', 'key': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 7',
[[['f', {'key': 'ab', 'country': 'ab'}, 2],
['g', {'key': 'ba', 'country': 'DE'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['g', {'key': 'ab', 'country': 'ab'}, 2],
['f', {'key': 'ba', 'country': 'ab'}, 1],
['f', {'country': 'ab', 'key': 'c'}, 1]]],
[[2, 'miss'], [1, 'miss'], [1, 'hit'], [0, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 59',
[[['g', {'key': 'ba', 'country': 'US'}, 1],
['f', {'key': 'c', 'country': 'ab'}, 1],
['f', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'country': 'ab', 'key': 'c'}, 1]]],
[[2, 'miss'], [1, 'miss'], [0, 'miss'], [1, 'hit']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 11',
[[['f', {'key': 'ba', 'country': 'US'}, 2],
['g', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'key': 'ab', 'country': 'DE'}, 2],
['g', {'key': 'ba', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ab'}, 2]]],
[[2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'hit'], [0, 'miss']]),
('call sequence sample 12',
[[['g', {'country': 'US', 'key': 'ba'}, 2], ['f', {'country': 'DE', 'key': 'ba'}, 1]]],
[[0, 'miss'], [0, 'miss']]),
('call sequence sample 13',
[[['g', {'country': 'DE', 'key': 'ba'}, 2],
['g', {'key': 'ab', 'country': 'ab'}, 1],
['g', {'key': 'ab', 'country': 'ab'}, 2],
['f', {'country': 'US', 'key': 'c'}, 1],
['f', {'country': 'ab', 'key': 'c'}, 2]]],
[[2, 'miss'], [2, 'miss'], [0, 'miss'], [1, 'miss'], [2, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 7',
[[['f', {'key': 'ab', 'country': 'ab'}, 2],
['g', {'key': 'ba', 'country': 'DE'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['g', {'key': 'ab', 'country': 'ab'}, 2],
['f', {'key': 'ba', 'country': 'ab'}, 1],
['f', {'country': 'ab', 'key': 'c'}, 1]]],
[[2, 'miss'], [1, 'miss'], [1, 'hit'], [0, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 16',
[[['g', {'country': 'ab', 'key': 'ba'}, 1],
['f', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 1]]],
[[2, 'miss'], [0, 'miss'], [2, 'miss']]),
('call sequence sample 17',
[[['f', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'DE'}, 2],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['g', {'key': 'ab', 'country': 'DE'}, 1]]],
[[0, 'miss'], [1, 'miss'], [1, 'miss'], [1, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 21',
[[['g', {'key': 'c', 'country': 'US'}, 2],
['f', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'DE'}, 1],
['g', {'key': 'c', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1]]],
[[0, 'miss'], [0, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss']]),
('call sequence sample 22',
[[['g', {'key': 'ab', 'country': 'DE'}, 1],
['f', {'country': 'ab', 'key': 'ba'}, 1],
['g', {'country': 'ab', 'key': 'ab'}, 2],
['f', {'country': 'US', 'key': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'DE'}, 1],
['f', {'country': 'ab', 'key': 'c'}, 2]]],
[[1, 'miss'], [1, 'miss'], [0, 'miss'], [1, 'miss'], [1, 'miss'], [2, 'miss']])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| version bump misses the memo | [[1, 'miss'], [2, 'miss']] | [[1, 'miss'], [2, 'miss']] | Passed |
| attribute change misses the memo | [[1, 'miss'], [0, 'miss']] | [[1, 'miss'], [0, 'miss']] | Passed |
| attribute order does not matter | [[1, 'miss'], [1, 'hit']] | [[1, 'miss'], [1, 'hit']] | Passed |
| swapped attribute values are different contexts | [[2, 'miss'], [2, 'miss']] | [[2, 'miss'], [2, 'miss']] | Passed |
| different flags do not share entries | [[1, 'miss'], [2, 'miss']] | [[1, 'miss'], [2, 'miss']] | Passed |
| call sequence sample 1 | [[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | [[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | Passed |
| call sequence sample 2 | [[0, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']] | [[0, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']] | Passed |
| call sequence sample 3 | [[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | [[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | Passed |
SHA-256 / 006c0644b5c16b3a69ef91d753369eb4bad7dce76ea1593b67e202670d87a636
Verification & scope
A deterministic toy flag-evaluation model with a stipulated contract; it does not reproduce any vendor SDK byte for byte. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.
Observations recorded using Python 3.12.14 at 2026-09-29T14:48:55.854663+00:00.
Case digest / 8becd98fa38324a5241ceff96d8d87ea97d29e3e4cab8f072639ddfadd473dbf