FA-74781 / Experiment statistics / Open access
Team-draft interleaving credit: Clicks credit every ranker containing the document · case 01
Clicks on documents both rankers returned count for both, masking real preferences.
ROOT CAUSE
Credit checks whether the clicked document appears in each ranking rather than who placed it.
VERIFIED REPAIR
Credit only the team recorded for that position.
Unsuccessful approach: Crediting only documents exclusive to one ranking ignores the team that actually placed shared ones.
Case contract
Build an interleaved list of up to length documents. Each round both teams skip documents already placed; when both teams have the same number of picks, the next coin (cycled, consumed only on ties) chooses A on 0 and B on 1; otherwise the team with fewer picks goes. An exhausted team yields to the other. Clicks are positions; each credits the team that placed that document. Return [documents, teams, winner A/B/tie].
Why this case matters
Interleaving compares ranking algorithms with far fewer users than an A/B test, if credit is fair.
1 / The failure
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(rank_a, rank_b, coins, clicks, length):
out, team = [], []
ia = ib = ci = 0
while len(out) < length:
while ia < len(rank_a) and rank_a[ia] in out:
ia += 1
while ib < len(rank_b) and rank_b[ib] in out:
ib += 1
has_a, has_b = ia < len(rank_a), ib < len(rank_b)
if not has_a and not has_b:
break
na, nb = team.count('A'), team.count('B')
if na == nb:
pick_a = coins[ci % len(coins)] == 0
ci += 1
else:
pick_a = na < nb
if pick_a and not has_a:
pick_a = False
elif not pick_a and not has_b:
pick_a = True
if pick_a:
out.append(rank_a[ia])
team.append('A')
else:
out.append(rank_b[ib])
team.append('B')
credit_a = sum(1 for i in clicks if i < len(out) and out[i] in rank_a)
credit_b = sum(1 for i in clicks if i < len(out) and out[i] in rank_b)
winner = 'A' if credit_a > credit_b else 'B' if credit_b > credit_a else 'tie'
return [out, team, winner]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('shared top document is placed once',
[['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
[['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
('coins are consumed only on ties',
[['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
[['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
('team behind picks next',
[['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
[['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
('shared document credits the team that placed it',
[['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
[['d2', 'd1'], ['B', 'A'], 'B']),
('exhausted ranking yields',
[['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
[['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
('query sample 1',
[['d4', 'd3', 'd2'], ['d5', 'd6'], [0], [], 6],
[['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']),
('query sample 2',
[['d6', 'd5', 'd1', 'd2'], ['d6', 'd5'], [1, 1, 1], [], 4],
[['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie']),
('query sample 3',
[['d5', 'd4'], ['d6', 'd3', 'd5', 'd1'], [0], [], 5],
[['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'])],
[('coins are consumed only on ties',
[['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
[['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
('team behind picks next',
[['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
[['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
('shared document credits the team that placed it',
[['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
[['d2', 'd1'], ['B', 'A'], 'B']),
('exhausted ranking yields',
[['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
[['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
('query sample 6',
[['d1', 'd4', 'd5'], ['d5', 'd3', 'd2'], [0], [1], 6],
[['d1', 'd5', 'd4', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'B']),
('query sample 7',
[['d4', 'd1', 'd6'], ['d1', 'd6', 'd2', 'd4', 'd3'], [0, 0, 1], [4, 5], 5],
[['d4', 'd1', 'd6', 'd2', 'd3'], ['A', 'B', 'A', 'B', 'B'], 'B']),
('query sample 31',
[['d2', 'd4', 'd5', 'd6', 'd3'], ['d1', 'd3'], [1], [2, 5], 5],
[['d1', 'd2', 'd3', 'd4', 'd5'], ['B', 'A', 'B', 'A', 'A'], 'B'])],
[('shared top document is placed once',
[['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
[['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
('team behind picks next',
[['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
[['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
('shared document credits the team that placed it',
[['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
[['d2', 'd1'], ['B', 'A'], 'B']),
('exhausted ranking yields',
[['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
[['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
('query sample 6',
[['d1', 'd4', 'd5'], ['d5', 'd3', 'd2'], [0], [1], 6],
[['d1', 'd5', 'd4', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'B']),
('query sample 11',
[['d4', 'd2', 'd5'], ['d5', 'd2', 'd1', 'd6'], [1, 0], [], 6],
[['d5', 'd4', 'd2', 'd1', 'd6'], ['B', 'A', 'A', 'B', 'B'], 'tie']),
('query sample 12',
[['d1', 'd4', 'd5', 'd6', 'd2'], ['d6', 'd2', 'd5'], [0, 1, 0], [], 3],
[['d1', 'd6', 'd2'], ['A', 'B', 'B'], 'tie'])],
[('shared top document is placed once',
[['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
[['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
('coins are consumed only on ties',
[['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
[['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
('shared document credits the team that placed it',
[['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
[['d2', 'd1'], ['B', 'A'], 'B']),
('exhausted ranking yields',
[['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
[['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
('query sample 16',
[['d1', 'd3', 'd4', 'd2'], ['d3', 'd2', 'd1', 'd5'], [0, 1, 1], [], 4],
[['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'B', 'A'], 'tie']),
('query sample 17',
[['d4', 'd2'], ['d3', 'd5', 'd6', 'd4', 'd2'], [1, 1], [], 5],
[['d3', 'd4', 'd5', 'd2', 'd6'], ['B', 'A', 'B', 'A', 'B'], 'tie']),
('query sample 45',
[['d4', 'd1'], ['d5', 'd2', 'd3', 'd4'], [0, 1], [0, 5], 4],
[['d4', 'd5', 'd2', 'd1'], ['A', 'B', 'B', 'A'], 'A'])],
[('shared top document is placed once',
[['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
[['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
('coins are consumed only on ties',
[['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
[['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
('team behind picks next',
[['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
[['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
('exhausted ranking yields',
[['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
[['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
('query sample 21',
[['d5', 'd3'], ['d2', 'd3', 'd6', 'd4', 'd5'], [1, 1], [], 5],
[['d2', 'd5', 'd3', 'd6', 'd4'], ['B', 'A', 'B', 'B', 'B'], 'tie']),
('query sample 22',
[['d1', 'd4'], ['d1', 'd6', 'd3', 'd5', 'd2'], [0], [4], 4],
[['d1', 'd6', 'd4', 'd3'], ['A', 'B', 'A', 'B'], 'tie']),
('query sample 25',
[['d5', 'd3', 'd2'], ['d2', 'd3', 'd4'], [1], [0, 2], 5],
[['d2', 'd5', 'd3', 'd4'], ['B', 'A', 'B', 'B'], 'B'])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| shared top document is placed once | [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'tie'] | [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B'] | Failed |
| coins are consumed only on ties | [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B'] | [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B'] | Passed |
| team behind picks next | [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A'] | [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A'] | Passed |
| shared document credits the team that placed it | [['d2', 'd1'], ['B', 'A'], 'tie'] | [['d2', 'd1'], ['B', 'A'], 'B'] | Failed |
| exhausted ranking yields | [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B'] | [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B'] | Passed |
| query sample 1 | [['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie'] | [['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie'] | Passed |
| query sample 2 | [['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie'] | [['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie'] | Passed |
| query sample 3 | [['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'] | [['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'] | Passed |
SHA-256 / 7f70b9a14b1771f850fddd07b1955f46fc9ca0cbf1db16ec5ca73d14073875a1
2 / The unsuccessful fix
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(rank_a, rank_b, coins, clicks, length):
out, team = [], []
ia = ib = ci = 0
while len(out) < length:
while ia < len(rank_a) and rank_a[ia] in out:
ia += 1
while ib < len(rank_b) and rank_b[ib] in out:
ib += 1
has_a, has_b = ia < len(rank_a), ib < len(rank_b)
if not has_a and not has_b:
break
na, nb = team.count('A'), team.count('B')
if na == nb:
pick_a = coins[ci % len(coins)] == 0
ci += 1
else:
pick_a = na < nb
if pick_a and not has_a:
pick_a = False
elif not pick_a and not has_b:
pick_a = True
if pick_a:
out.append(rank_a[ia])
team.append('A')
else:
out.append(rank_b[ib])
team.append('B')
credit_a = sum(1 for i in clicks if i < len(out) and out[i] in rank_a and out[i] not in rank_b)
credit_b = sum(1 for i in clicks if i < len(out) and out[i] in rank_b and out[i] not in rank_a)
winner = 'A' if credit_a > credit_b else 'B' if credit_b > credit_a else 'tie'
return [out, team, winner]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('shared top document is placed once',
[['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
[['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
('coins are consumed only on ties',
[['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
[['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
('team behind picks next',
[['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
[['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
('shared document credits the team that placed it',
[['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
[['d2', 'd1'], ['B', 'A'], 'B']),
('exhausted ranking yields',
[['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
[['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
('query sample 1',
[['d4', 'd3', 'd2'], ['d5', 'd6'], [0], [], 6],
[['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']),
('query sample 2',
[['d6', 'd5', 'd1', 'd2'], ['d6', 'd5'], [1, 1, 1], [], 4],
[['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie']),
('query sample 3',
[['d5', 'd4'], ['d6', 'd3', 'd5', 'd1'], [0], [], 5],
[['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'])],
[('coins are consumed only on ties',
[['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
[['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
('team behind picks next',
[['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
[['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
('shared document credits the team that placed it',
[['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
[['d2', 'd1'], ['B', 'A'], 'B']),
('exhausted ranking yields',
[['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
[['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
('query sample 6',
[['d1', 'd4', 'd5'], ['d5', 'd3', 'd2'], [0], [1], 6],
[['d1', 'd5', 'd4', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'B']),
('query sample 7',
[['d4', 'd1', 'd6'], ['d1', 'd6', 'd2', 'd4', 'd3'], [0, 0, 1], [4, 5], 5],
[['d4', 'd1', 'd6', 'd2', 'd3'], ['A', 'B', 'A', 'B', 'B'], 'B']),
('query sample 31',
[['d2', 'd4', 'd5', 'd6', 'd3'], ['d1', 'd3'], [1], [2, 5], 5],
[['d1', 'd2', 'd3', 'd4', 'd5'], ['B', 'A', 'B', 'A', 'A'], 'B'])],
[('shared top document is placed once',
[['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
[['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
('team behind picks next',
[['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
[['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
('shared document credits the team that placed it',
[['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
[['d2', 'd1'], ['B', 'A'], 'B']),
('exhausted ranking yields',
[['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
[['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
('query sample 6',
[['d1', 'd4', 'd5'], ['d5', 'd3', 'd2'], [0], [1], 6],
[['d1', 'd5', 'd4', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'B']),
('query sample 11',
[['d4', 'd2', 'd5'], ['d5', 'd2', 'd1', 'd6'], [1, 0], [], 6],
[['d5', 'd4', 'd2', 'd1', 'd6'], ['B', 'A', 'A', 'B', 'B'], 'tie']),
('query sample 12',
[['d1', 'd4', 'd5', 'd6', 'd2'], ['d6', 'd2', 'd5'], [0, 1, 0], [], 3],
[['d1', 'd6', 'd2'], ['A', 'B', 'B'], 'tie'])],
[('shared top document is placed once',
[['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
[['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
('coins are consumed only on ties',
[['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
[['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
('shared document credits the team that placed it',
[['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
[['d2', 'd1'], ['B', 'A'], 'B']),
('exhausted ranking yields',
[['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
[['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
('query sample 16',
[['d1', 'd3', 'd4', 'd2'], ['d3', 'd2', 'd1', 'd5'], [0, 1, 1], [], 4],
[['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'B', 'A'], 'tie']),
('query sample 17',
[['d4', 'd2'], ['d3', 'd5', 'd6', 'd4', 'd2'], [1, 1], [], 5],
[['d3', 'd4', 'd5', 'd2', 'd6'], ['B', 'A', 'B', 'A', 'B'], 'tie']),
('query sample 45',
[['d4', 'd1'], ['d5', 'd2', 'd3', 'd4'], [0, 1], [0, 5], 4],
[['d4', 'd5', 'd2', 'd1'], ['A', 'B', 'B', 'A'], 'A'])],
[('shared top document is placed once',
[['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
[['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
('coins are consumed only on ties',
[['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
[['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
('team behind picks next',
[['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
[['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
('exhausted ranking yields',
[['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
[['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
('query sample 21',
[['d5', 'd3'], ['d2', 'd3', 'd6', 'd4', 'd5'], [1, 1], [], 5],
[['d2', 'd5', 'd3', 'd6', 'd4'], ['B', 'A', 'B', 'B', 'B'], 'tie']),
('query sample 22',
[['d1', 'd4'], ['d1', 'd6', 'd3', 'd5', 'd2'], [0], [4], 4],
[['d1', 'd6', 'd4', 'd3'], ['A', 'B', 'A', 'B'], 'tie']),
('query sample 25',
[['d5', 'd3', 'd2'], ['d2', 'd3', 'd4'], [1], [0, 2], 5],
[['d2', 'd5', 'd3', 'd4'], ['B', 'A', 'B', 'B'], 'B'])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| shared top document is placed once | [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'tie'] | [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B'] | Failed |
| coins are consumed only on ties | [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B'] | [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B'] | Passed |
| team behind picks next | [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A'] | [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A'] | Passed |
| shared document credits the team that placed it | [['d2', 'd1'], ['B', 'A'], 'tie'] | [['d2', 'd1'], ['B', 'A'], 'B'] | Failed |
| exhausted ranking yields | [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B'] | [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B'] | Passed |
| query sample 1 | [['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie'] | [['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie'] | Passed |
| query sample 2 | [['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie'] | [['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie'] | Passed |
| query sample 3 | [['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'] | [['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'] | Passed |
SHA-256 / 9abcd321f8454ec839361a4f43663fb9956e9703a3de0744d1bae3f8bb821651
3 / The verified repair
Exit 0"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(rank_a, rank_b, coins, clicks, length):
out, team = [], []
ia = ib = ci = 0
while len(out) < length:
while ia < len(rank_a) and rank_a[ia] in out:
ia += 1
while ib < len(rank_b) and rank_b[ib] in out:
ib += 1
has_a, has_b = ia < len(rank_a), ib < len(rank_b)
if not has_a and not has_b:
break
na, nb = team.count('A'), team.count('B')
if na == nb:
pick_a = coins[ci % len(coins)] == 0
ci += 1
else:
pick_a = na < nb
if pick_a and not has_a:
pick_a = False
elif not pick_a and not has_b:
pick_a = True
if pick_a:
out.append(rank_a[ia])
team.append('A')
else:
out.append(rank_b[ib])
team.append('B')
credit_a = sum(1 for i in clicks if i < len(out) and team[i] == 'A')
credit_b = sum(1 for i in clicks if i < len(out) and team[i] == 'B')
winner = 'A' if credit_a > credit_b else 'B' if credit_b > credit_a else 'tie'
return [out, team, winner]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('shared top document is placed once',
[['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
[['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
('coins are consumed only on ties',
[['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
[['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
('team behind picks next',
[['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
[['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
('shared document credits the team that placed it',
[['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
[['d2', 'd1'], ['B', 'A'], 'B']),
('exhausted ranking yields',
[['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
[['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
('query sample 1',
[['d4', 'd3', 'd2'], ['d5', 'd6'], [0], [], 6],
[['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']),
('query sample 2',
[['d6', 'd5', 'd1', 'd2'], ['d6', 'd5'], [1, 1, 1], [], 4],
[['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie']),
('query sample 3',
[['d5', 'd4'], ['d6', 'd3', 'd5', 'd1'], [0], [], 5],
[['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'])],
[('coins are consumed only on ties',
[['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
[['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
('team behind picks next',
[['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
[['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
('shared document credits the team that placed it',
[['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
[['d2', 'd1'], ['B', 'A'], 'B']),
('exhausted ranking yields',
[['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
[['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
('query sample 6',
[['d1', 'd4', 'd5'], ['d5', 'd3', 'd2'], [0], [1], 6],
[['d1', 'd5', 'd4', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'B']),
('query sample 7',
[['d4', 'd1', 'd6'], ['d1', 'd6', 'd2', 'd4', 'd3'], [0, 0, 1], [4, 5], 5],
[['d4', 'd1', 'd6', 'd2', 'd3'], ['A', 'B', 'A', 'B', 'B'], 'B']),
('query sample 31',
[['d2', 'd4', 'd5', 'd6', 'd3'], ['d1', 'd3'], [1], [2, 5], 5],
[['d1', 'd2', 'd3', 'd4', 'd5'], ['B', 'A', 'B', 'A', 'A'], 'B'])],
[('shared top document is placed once',
[['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
[['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
('team behind picks next',
[['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
[['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
('shared document credits the team that placed it',
[['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
[['d2', 'd1'], ['B', 'A'], 'B']),
('exhausted ranking yields',
[['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
[['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
('query sample 6',
[['d1', 'd4', 'd5'], ['d5', 'd3', 'd2'], [0], [1], 6],
[['d1', 'd5', 'd4', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'B']),
('query sample 11',
[['d4', 'd2', 'd5'], ['d5', 'd2', 'd1', 'd6'], [1, 0], [], 6],
[['d5', 'd4', 'd2', 'd1', 'd6'], ['B', 'A', 'A', 'B', 'B'], 'tie']),
('query sample 12',
[['d1', 'd4', 'd5', 'd6', 'd2'], ['d6', 'd2', 'd5'], [0, 1, 0], [], 3],
[['d1', 'd6', 'd2'], ['A', 'B', 'B'], 'tie'])],
[('shared top document is placed once',
[['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
[['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
('coins are consumed only on ties',
[['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
[['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
('shared document credits the team that placed it',
[['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
[['d2', 'd1'], ['B', 'A'], 'B']),
('exhausted ranking yields',
[['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
[['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
('query sample 16',
[['d1', 'd3', 'd4', 'd2'], ['d3', 'd2', 'd1', 'd5'], [0, 1, 1], [], 4],
[['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'B', 'A'], 'tie']),
('query sample 17',
[['d4', 'd2'], ['d3', 'd5', 'd6', 'd4', 'd2'], [1, 1], [], 5],
[['d3', 'd4', 'd5', 'd2', 'd6'], ['B', 'A', 'B', 'A', 'B'], 'tie']),
('query sample 45',
[['d4', 'd1'], ['d5', 'd2', 'd3', 'd4'], [0, 1], [0, 5], 4],
[['d4', 'd5', 'd2', 'd1'], ['A', 'B', 'B', 'A'], 'A'])],
[('shared top document is placed once',
[['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
[['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
('coins are consumed only on ties',
[['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
[['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
('team behind picks next',
[['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
[['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
('exhausted ranking yields',
[['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
[['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
('query sample 21',
[['d5', 'd3'], ['d2', 'd3', 'd6', 'd4', 'd5'], [1, 1], [], 5],
[['d2', 'd5', 'd3', 'd6', 'd4'], ['B', 'A', 'B', 'B', 'B'], 'tie']),
('query sample 22',
[['d1', 'd4'], ['d1', 'd6', 'd3', 'd5', 'd2'], [0], [4], 4],
[['d1', 'd6', 'd4', 'd3'], ['A', 'B', 'A', 'B'], 'tie']),
('query sample 25',
[['d5', 'd3', 'd2'], ['d2', 'd3', 'd4'], [1], [0, 2], 5],
[['d2', 'd5', 'd3', 'd4'], ['B', 'A', 'B', 'B'], 'B'])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| shared top document is placed once | [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B'] | [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B'] | Passed |
| coins are consumed only on ties | [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B'] | [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B'] | Passed |
| team behind picks next | [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A'] | [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A'] | Passed |
| shared document credits the team that placed it | [['d2', 'd1'], ['B', 'A'], 'B'] | [['d2', 'd1'], ['B', 'A'], 'B'] | Passed |
| exhausted ranking yields | [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B'] | [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B'] | Passed |
| query sample 1 | [['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie'] | [['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie'] | Passed |
| query sample 2 | [['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie'] | [['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie'] | Passed |
| query sample 3 | [['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'] | [['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'] | Passed |
SHA-256 / afa31581831271dcff445b66bf7f1cab674fc6fab2f19df7b93da800f20aca3f
Verification & scope
A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.
Observations recorded using Python 3.12.14 at 2026-09-29T14:48:59.851164+00:00.
Case digest / 5f2036c702e93516ea8349e608e7511ab689c0ee84c45d43b553638a83f60118