FA-74331 / Feature flag rollout bucketing / Open access
Per-context evaluation memo: Memo is keyed by context key only · case 01
A user who changes country keeps the result computed for the old country.
ROOT CAUSE
The memo key uses ctx["key"] instead of every context attribute.
VERIFIED REPAIR
Key the memo on all attribute names and values.
Unsuccessful approach: Keying on attribute names without values still conflates different values.
Case contract
calls are [flag, context dict of string attributes, flag version]. The toy evaluation is (sum of code points of flag + key + country) + version) % 3. Results are memoised by (flag, context contents independent of attribute order, version). Return [value, hit|miss] per call.
Why this case matters
Evaluation caches in SDKs must key on every input that can change the result.
1 / The failure
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(calls):
memo = {}
out = []
for flag, ctx, version in calls:
k = (flag, ctx.get('key'), version)
if k in memo:
out.append([memo[k], 'hit'])
continue
value = (sum(map(ord, flag + ctx.get('key', '') + ctx.get('country', ''))) + version) % 3
memo[k] = value
out.append([value, 'miss'])
return out
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('call sequence sample 1',
[[['f', {'key': 'ab', 'country': 'US'}, 2],
['g', {'country': 'US', 'key': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'DE'}, 1],
['g', {'country': 'ab', 'key': 'c'}, 1],
['g', {'country': 'US', 'key': 'ba'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 2]]],
[[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']]),
('call sequence sample 2',
[[['f', {'country': 'DE', 'key': 'ab'}, 1],
['f', {'key': 'ab', 'country': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'ab'}, 2],
['f', {'key': 'ba', 'country': 'ab'}, 2],
['f', {'key': 'ab', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'c'}, 1]]],
[[0, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 3',
[[['f', {'country': 'DE', 'key': 'c'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 2]]],
[[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']])],
[('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 6',
[[['f', {'country': 'ab', 'key': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 7',
[[['f', {'key': 'ab', 'country': 'ab'}, 2],
['g', {'key': 'ba', 'country': 'DE'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['g', {'key': 'ab', 'country': 'ab'}, 2],
['f', {'key': 'ba', 'country': 'ab'}, 1],
['f', {'country': 'ab', 'key': 'c'}, 1]]],
[[2, 'miss'], [1, 'miss'], [1, 'hit'], [0, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 11',
[[['f', {'key': 'ba', 'country': 'US'}, 2],
['g', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'key': 'ab', 'country': 'DE'}, 2],
['g', {'key': 'ba', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ab'}, 2]]],
[[2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'hit'], [0, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 11',
[[['f', {'key': 'ba', 'country': 'US'}, 2],
['g', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'key': 'ab', 'country': 'DE'}, 2],
['g', {'key': 'ba', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ab'}, 2]]],
[[2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'hit'], [0, 'miss']]),
('call sequence sample 16',
[[['g', {'country': 'ab', 'key': 'ba'}, 1],
['f', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 1]]],
[[2, 'miss'], [0, 'miss'], [2, 'miss']]),
('call sequence sample 29',
[[['f', {'key': 'ba', 'country': 'DE'}, 1],
['f', {'key': 'c', 'country': 'DE'}, 2],
['f', {'country': 'DE', 'key': 'ab'}, 1],
['f', {'country': 'US', 'key': 'ab'}, 1],
['g', {'country': 'DE', 'key': 'ab'}, 1]]],
[[0, 'miss'], [1, 'miss'], [0, 'miss'], [1, 'miss'], [1, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 16',
[[['g', {'country': 'ab', 'key': 'ba'}, 1],
['f', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 1]]],
[[2, 'miss'], [0, 'miss'], [2, 'miss']]),
('call sequence sample 23',
[[['f', {'country': 'US', 'key': 'ab'}, 1],
['f', {'country': 'ab', 'key': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 1]]],
[[1, 'miss'], [1, 'miss'], [2, 'miss']]),
('call sequence sample 55',
[[['f', {'country': 'ab', 'key': 'ba'}, 1],
['g', {'country': 'US', 'key': 'c'}, 1],
['g', {'key': 'ab', 'country': 'ab'}, 1],
['f', {'key': 'c', 'country': 'ab'}, 1],
['f', {'country': 'DE', 'key': 'c'}, 2],
['g', {'key': 'c', 'country': 'ab'}, 1]]],
[[1, 'miss'], [2, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss'], [2, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 10',
[[['f', {'key': 'c', 'country': 'DE'}, 1],
['f', {'country': 'US', 'key': 'ba'}, 2],
['g', {'country': 'ab', 'key': 'ba'}, 1],
['g', {'key': 'ba', 'country': 'US'}, 1],
['g', {'country': 'ab', 'key': 'ba'}, 1]]],
[[0, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'hit']]),
('call sequence sample 21',
[[['g', {'key': 'c', 'country': 'US'}, 2],
['f', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'DE'}, 1],
['g', {'key': 'c', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1]]],
[[0, 'miss'], [0, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss']]),
('call sequence sample 30',
[[['f', {'key': 'c', 'country': 'US'}, 2],
['g', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'country': 'US', 'key': 'c'}, 1]]],
[[2, 'miss'], [1, 'miss'], [2, 'miss']])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| version bump misses the memo | [[1, 'miss'], [2, 'miss']] | [[1, 'miss'], [2, 'miss']] | Passed |
| attribute change misses the memo | [[1, 'miss'], [1, 'hit']] | [[1, 'miss'], [0, 'miss']] | Failed |
| attribute order does not matter | [[1, 'miss'], [1, 'hit']] | [[1, 'miss'], [1, 'hit']] | Passed |
| swapped attribute values are different contexts | [[2, 'miss'], [2, 'miss']] | [[2, 'miss'], [2, 'miss']] | Passed |
| different flags do not share entries | [[1, 'miss'], [2, 'miss']] | [[1, 'miss'], [2, 'miss']] | Passed |
| call sequence sample 1 | [[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [1, 'hit'], [0, 'miss']] | [[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | Failed |
| call sequence sample 2 | [[0, 'miss'], [0, 'hit'], [0, 'miss'], [2, 'miss'], [0, 'hit'], [1, 'miss']] | [[0, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']] | Failed |
| call sequence sample 3 | [[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | [[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | Passed |
SHA-256 / cef6f503a9e19271d74a120e361b9ebbd03908d0d6f4a44d547292a401aecc09
2 / The unsuccessful fix
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(calls):
memo = {}
out = []
for flag, ctx, version in calls:
k = (flag, tuple(sorted(ctx)), version)
if k in memo:
out.append([memo[k], 'hit'])
continue
value = (sum(map(ord, flag + ctx.get('key', '') + ctx.get('country', ''))) + version) % 3
memo[k] = value
out.append([value, 'miss'])
return out
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('call sequence sample 1',
[[['f', {'key': 'ab', 'country': 'US'}, 2],
['g', {'country': 'US', 'key': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'DE'}, 1],
['g', {'country': 'ab', 'key': 'c'}, 1],
['g', {'country': 'US', 'key': 'ba'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 2]]],
[[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']]),
('call sequence sample 2',
[[['f', {'country': 'DE', 'key': 'ab'}, 1],
['f', {'key': 'ab', 'country': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'ab'}, 2],
['f', {'key': 'ba', 'country': 'ab'}, 2],
['f', {'key': 'ab', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'c'}, 1]]],
[[0, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 3',
[[['f', {'country': 'DE', 'key': 'c'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 2]]],
[[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']])],
[('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 6',
[[['f', {'country': 'ab', 'key': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 7',
[[['f', {'key': 'ab', 'country': 'ab'}, 2],
['g', {'key': 'ba', 'country': 'DE'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['g', {'key': 'ab', 'country': 'ab'}, 2],
['f', {'key': 'ba', 'country': 'ab'}, 1],
['f', {'country': 'ab', 'key': 'c'}, 1]]],
[[2, 'miss'], [1, 'miss'], [1, 'hit'], [0, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 11',
[[['f', {'key': 'ba', 'country': 'US'}, 2],
['g', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'key': 'ab', 'country': 'DE'}, 2],
['g', {'key': 'ba', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ab'}, 2]]],
[[2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'hit'], [0, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 11',
[[['f', {'key': 'ba', 'country': 'US'}, 2],
['g', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'key': 'ab', 'country': 'DE'}, 2],
['g', {'key': 'ba', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ab'}, 2]]],
[[2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'hit'], [0, 'miss']]),
('call sequence sample 16',
[[['g', {'country': 'ab', 'key': 'ba'}, 1],
['f', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 1]]],
[[2, 'miss'], [0, 'miss'], [2, 'miss']]),
('call sequence sample 29',
[[['f', {'key': 'ba', 'country': 'DE'}, 1],
['f', {'key': 'c', 'country': 'DE'}, 2],
['f', {'country': 'DE', 'key': 'ab'}, 1],
['f', {'country': 'US', 'key': 'ab'}, 1],
['g', {'country': 'DE', 'key': 'ab'}, 1]]],
[[0, 'miss'], [1, 'miss'], [0, 'miss'], [1, 'miss'], [1, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 16',
[[['g', {'country': 'ab', 'key': 'ba'}, 1],
['f', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 1]]],
[[2, 'miss'], [0, 'miss'], [2, 'miss']]),
('call sequence sample 23',
[[['f', {'country': 'US', 'key': 'ab'}, 1],
['f', {'country': 'ab', 'key': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 1]]],
[[1, 'miss'], [1, 'miss'], [2, 'miss']]),
('call sequence sample 55',
[[['f', {'country': 'ab', 'key': 'ba'}, 1],
['g', {'country': 'US', 'key': 'c'}, 1],
['g', {'key': 'ab', 'country': 'ab'}, 1],
['f', {'key': 'c', 'country': 'ab'}, 1],
['f', {'country': 'DE', 'key': 'c'}, 2],
['g', {'key': 'c', 'country': 'ab'}, 1]]],
[[1, 'miss'], [2, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss'], [2, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 10',
[[['f', {'key': 'c', 'country': 'DE'}, 1],
['f', {'country': 'US', 'key': 'ba'}, 2],
['g', {'country': 'ab', 'key': 'ba'}, 1],
['g', {'key': 'ba', 'country': 'US'}, 1],
['g', {'country': 'ab', 'key': 'ba'}, 1]]],
[[0, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'hit']]),
('call sequence sample 21',
[[['g', {'key': 'c', 'country': 'US'}, 2],
['f', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'DE'}, 1],
['g', {'key': 'c', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1]]],
[[0, 'miss'], [0, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss']]),
('call sequence sample 30',
[[['f', {'key': 'c', 'country': 'US'}, 2],
['g', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'country': 'US', 'key': 'c'}, 1]]],
[[2, 'miss'], [1, 'miss'], [2, 'miss']])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| version bump misses the memo | [[1, 'miss'], [2, 'miss']] | [[1, 'miss'], [2, 'miss']] | Passed |
| attribute change misses the memo | [[1, 'miss'], [1, 'hit']] | [[1, 'miss'], [0, 'miss']] | Failed |
| attribute order does not matter | [[1, 'miss'], [1, 'hit']] | [[1, 'miss'], [1, 'hit']] | Passed |
| swapped attribute values are different contexts | [[2, 'miss'], [2, 'hit']] | [[2, 'miss'], [2, 'miss']] | Failed |
| different flags do not share entries | [[1, 'miss'], [2, 'miss']] | [[1, 'miss'], [2, 'miss']] | Passed |
| call sequence sample 1 | [[2, 'miss'], [2, 'miss'], [2, 'hit'], [2, 'hit'], [2, 'hit'], [0, 'miss']] | [[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | Failed |
| call sequence sample 2 | [[0, 'miss'], [0, 'hit'], [0, 'miss'], [2, 'miss'], [0, 'hit'], [1, 'miss']] | [[0, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']] | Failed |
| call sequence sample 3 | [[0, 'miss'], [1, 'miss'], [2, 'miss'], [1, 'hit'], [1, 'hit'], [0, 'miss']] | [[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | Failed |
SHA-256 / 0126186ea5cfd0857bc77e4864f30e2da792c06b25af5fc077c337c86897d120
3 / The verified repair
Exit 0"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(calls):
memo = {}
out = []
for flag, ctx, version in calls:
k = (flag, tuple(sorted(ctx.items())), version)
if k in memo:
out.append([memo[k], 'hit'])
continue
value = (sum(map(ord, flag + ctx.get('key', '') + ctx.get('country', ''))) + version) % 3
memo[k] = value
out.append([value, 'miss'])
return out
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('call sequence sample 1',
[[['f', {'key': 'ab', 'country': 'US'}, 2],
['g', {'country': 'US', 'key': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'DE'}, 1],
['g', {'country': 'ab', 'key': 'c'}, 1],
['g', {'country': 'US', 'key': 'ba'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 2]]],
[[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']]),
('call sequence sample 2',
[[['f', {'country': 'DE', 'key': 'ab'}, 1],
['f', {'key': 'ab', 'country': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'ab'}, 2],
['f', {'key': 'ba', 'country': 'ab'}, 2],
['f', {'key': 'ab', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'c'}, 1]]],
[[0, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 3',
[[['f', {'country': 'DE', 'key': 'c'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 2]]],
[[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']])],
[('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 6',
[[['f', {'country': 'ab', 'key': 'ab'}, 1],
['g', {'key': 'ba', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 7',
[[['f', {'key': 'ab', 'country': 'ab'}, 2],
['g', {'key': 'ba', 'country': 'DE'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1],
['g', {'key': 'ab', 'country': 'ab'}, 2],
['f', {'key': 'ba', 'country': 'ab'}, 1],
['f', {'country': 'ab', 'key': 'c'}, 1]]],
[[2, 'miss'], [1, 'miss'], [1, 'hit'], [0, 'miss'], [1, 'miss'], [1, 'miss']]),
('call sequence sample 11',
[[['f', {'key': 'ba', 'country': 'US'}, 2],
['g', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'key': 'ab', 'country': 'DE'}, 2],
['g', {'key': 'ba', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ab'}, 2]]],
[[2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'hit'], [0, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 11',
[[['f', {'key': 'ba', 'country': 'US'}, 2],
['g', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'key': 'ab', 'country': 'DE'}, 2],
['g', {'key': 'ba', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ab'}, 2],
['g', {'country': 'ab', 'key': 'ab'}, 2]]],
[[2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'hit'], [0, 'miss']]),
('call sequence sample 16',
[[['g', {'country': 'ab', 'key': 'ba'}, 1],
['f', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 1]]],
[[2, 'miss'], [0, 'miss'], [2, 'miss']]),
('call sequence sample 29',
[[['f', {'key': 'ba', 'country': 'DE'}, 1],
['f', {'key': 'c', 'country': 'DE'}, 2],
['f', {'country': 'DE', 'key': 'ab'}, 1],
['f', {'country': 'US', 'key': 'ab'}, 1],
['g', {'country': 'DE', 'key': 'ab'}, 1]]],
[[0, 'miss'], [1, 'miss'], [0, 'miss'], [1, 'miss'], [1, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('swapped attribute values are different contexts',
[[['g', {'key': 'ab', 'country': 'US'}, 1], ['g', {'key': 'US', 'country': 'ab'}, 1]]],
[[2, 'miss'], [2, 'miss']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 16',
[[['g', {'country': 'ab', 'key': 'ba'}, 1],
['f', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'country': 'US', 'key': 'ab'}, 1]]],
[[2, 'miss'], [0, 'miss'], [2, 'miss']]),
('call sequence sample 23',
[[['f', {'country': 'US', 'key': 'ab'}, 1],
['f', {'country': 'ab', 'key': 'ab'}, 1],
['g', {'key': 'c', 'country': 'ab'}, 1]]],
[[1, 'miss'], [1, 'miss'], [2, 'miss']]),
('call sequence sample 55',
[[['f', {'country': 'ab', 'key': 'ba'}, 1],
['g', {'country': 'US', 'key': 'c'}, 1],
['g', {'key': 'ab', 'country': 'ab'}, 1],
['f', {'key': 'c', 'country': 'ab'}, 1],
['f', {'country': 'DE', 'key': 'c'}, 2],
['g', {'key': 'c', 'country': 'ab'}, 1]]],
[[1, 'miss'], [2, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss'], [2, 'miss']])],
[('version bump misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'US'}, 2]]],
[[1, 'miss'], [2, 'miss']]),
('attribute change misses the memo',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'key': 'ab', 'country': 'DE'}, 1]]],
[[1, 'miss'], [0, 'miss']]),
('attribute order does not matter',
[[['f', {'key': 'ab', 'country': 'US'}, 1], ['f', {'country': 'US', 'key': 'ab'}, 1]]],
[[1, 'miss'], [1, 'hit']]),
('different flags do not share entries',
[[['f', {'key': 'c'}, 1], ['g', {'key': 'c'}, 1]]],
[[1, 'miss'], [2, 'miss']]),
('repeat call hits', [[['f', {'key': 'c'}, 1], ['f', {'key': 'c'}, 1]]], [[1, 'miss'], [1, 'hit']]),
('call sequence sample 10',
[[['f', {'key': 'c', 'country': 'DE'}, 1],
['f', {'country': 'US', 'key': 'ba'}, 2],
['g', {'country': 'ab', 'key': 'ba'}, 1],
['g', {'key': 'ba', 'country': 'US'}, 1],
['g', {'country': 'ab', 'key': 'ba'}, 1]]],
[[0, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'hit']]),
('call sequence sample 21',
[[['g', {'key': 'c', 'country': 'US'}, 2],
['f', {'country': 'DE', 'key': 'ba'}, 1],
['f', {'key': 'c', 'country': 'DE'}, 1],
['g', {'key': 'c', 'country': 'US'}, 1],
['g', {'country': 'DE', 'key': 'ba'}, 1]]],
[[0, 'miss'], [0, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss']]),
('call sequence sample 30',
[[['f', {'key': 'c', 'country': 'US'}, 2],
['g', {'key': 'ab', 'country': 'DE'}, 1],
['g', {'country': 'US', 'key': 'c'}, 1]]],
[[2, 'miss'], [1, 'miss'], [2, 'miss']])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| version bump misses the memo | [[1, 'miss'], [2, 'miss']] | [[1, 'miss'], [2, 'miss']] | Passed |
| attribute change misses the memo | [[1, 'miss'], [0, 'miss']] | [[1, 'miss'], [0, 'miss']] | Passed |
| attribute order does not matter | [[1, 'miss'], [1, 'hit']] | [[1, 'miss'], [1, 'hit']] | Passed |
| swapped attribute values are different contexts | [[2, 'miss'], [2, 'miss']] | [[2, 'miss'], [2, 'miss']] | Passed |
| different flags do not share entries | [[1, 'miss'], [2, 'miss']] | [[1, 'miss'], [2, 'miss']] | Passed |
| call sequence sample 1 | [[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | [[2, 'miss'], [2, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | Passed |
| call sequence sample 2 | [[0, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']] | [[0, 'miss'], [1, 'miss'], [0, 'miss'], [2, 'miss'], [1, 'miss'], [1, 'miss']] | Passed |
| call sequence sample 3 | [[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | [[0, 'miss'], [1, 'miss'], [2, 'miss'], [2, 'miss'], [2, 'miss'], [0, 'miss']] | Passed |
SHA-256 / 07577390992b78f639148bfab8430ca0680c0525761f4143a70d66e88f785e05
Verification & scope
A deterministic toy flag-evaluation model with a stipulated contract; it does not reproduce any vendor SDK byte for byte. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.
Observations recorded using Python 3.12.14 at 2026-09-29T14:48:55.811890+00:00.
Case digest / e04ae7699f84e38f00daad4057587484961855945bd9bf58f132847d7c850e06