FAILURE MAP
← Case archive

FA-74771 / Experiment statistics / Open access

Team-draft interleaving credit: A coin is consumed every round · case 01

Tie-break draws no longer follow the logged coin sequence, so replays disagree.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The coin index advances even when the pick is decided by team counts.

VERIFIED REPAIR

Advance the coin index only when a coin decides a tie.

Unsuccessful approach: Indexing coins by list length ties the draw to position rather than to tie rounds.

Case contract

Build an interleaved list of up to length documents. Each round both teams skip documents already placed; when both teams have the same number of picks, the next coin (cycled, consumed only on ties) chooses A on 0 and B on 1; otherwise the team with fewer picks goes. An exhausted team yields to the other. Clicks are positions; each credits the team that placed that document. Return [documents, teams, winner A/B/tie].

Why this case matters

Interleaving compares ranking algorithms with far fewer users than an A/B test, if credit is fair.

1 / The failure

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(rank_a, rank_b, coins, clicks, length):
    out, team = [], []
    ia = ib = ci = 0
    while len(out) < length:
        while ia < len(rank_a) and rank_a[ia] in out:
            ia += 1
        while ib < len(rank_b) and rank_b[ib] in out:
            ib += 1
        has_a, has_b = ia < len(rank_a), ib < len(rank_b)
        if not has_a and not has_b:
            break
        na, nb = team.count('A'), team.count('B')
        if na == nb:
            pick_a = coins[ci % len(coins)] == 0
            ci += 1
        else:
            pick_a = na < nb
            ci += 1
        if pick_a and not has_a:
            pick_a = False
        elif not pick_a and not has_b:
            pick_a = True
        if pick_a:
            out.append(rank_a[ia])
            team.append('A')
        else:
            out.append(rank_b[ib])
            team.append('B')
    credit_a = sum(1 for i in clicks if i < len(out) and team[i] == 'A')
    credit_b = sum(1 for i in clicks if i < len(out) and team[i] == 'B')
    winner = 'A' if credit_a > credit_b else 'B' if credit_b > credit_a else 'tie'
    return [out, team, winner]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('query sample 1',
   [['d4', 'd3', 'd2'], ['d5', 'd6'], [0], [], 6],
   [['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']),
  ('query sample 2',
   [['d6', 'd5', 'd1', 'd2'], ['d6', 'd5'], [1, 1, 1], [], 4],
   [['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie']),
  ('query sample 3',
   [['d5', 'd4'], ['d6', 'd3', 'd5', 'd1'], [0], [], 5],
   [['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'])],
 [('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 6',
   [['d1', 'd4', 'd5'], ['d5', 'd3', 'd2'], [0], [1], 6],
   [['d1', 'd5', 'd4', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'B']),
  ('query sample 7',
   [['d4', 'd1', 'd6'], ['d1', 'd6', 'd2', 'd4', 'd3'], [0, 0, 1], [4, 5], 5],
   [['d4', 'd1', 'd6', 'd2', 'd3'], ['A', 'B', 'A', 'B', 'B'], 'B']),
  ('query sample 12',
   [['d1', 'd4', 'd5', 'd6', 'd2'], ['d6', 'd2', 'd5'], [0, 1, 0], [], 3],
   [['d1', 'd6', 'd2'], ['A', 'B', 'B'], 'tie'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 11',
   [['d4', 'd2', 'd5'], ['d5', 'd2', 'd1', 'd6'], [1, 0], [], 6],
   [['d5', 'd4', 'd2', 'd1', 'd6'], ['B', 'A', 'A', 'B', 'B'], 'tie']),
  ('query sample 12',
   [['d1', 'd4', 'd5', 'd6', 'd2'], ['d6', 'd2', 'd5'], [0, 1, 0], [], 3],
   [['d1', 'd6', 'd2'], ['A', 'B', 'B'], 'tie']),
  ('query sample 41',
   [['d5', 'd2', 'd3', 'd1', 'd4'], ['d5', 'd3'], [1, 0, 1], [1], 6],
   [['d5', 'd2', 'd3', 'd1', 'd4'], ['B', 'A', 'A', 'A', 'A'], 'A'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 16',
   [['d1', 'd3', 'd4', 'd2'], ['d3', 'd2', 'd1', 'd5'], [0, 1, 1], [], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'B', 'A'], 'tie']),
  ('query sample 17',
   [['d4', 'd2'], ['d3', 'd5', 'd6', 'd4', 'd2'], [1, 1], [], 5],
   [['d3', 'd4', 'd5', 'd2', 'd6'], ['B', 'A', 'B', 'A', 'B'], 'tie']),
  ('query sample 51',
   [['d1', 'd2'], ['d5', 'd1', 'd4'], [0, 0, 1], [], 3],
   [['d1', 'd5', 'd2'], ['A', 'B', 'A'], 'tie'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 10',
   [['d4', 'd2', 'd1', 'd3'], ['d2', 'd3', 'd1'], [0, 0, 1], [1, 2], 6],
   [['d4', 'd2', 'd1', 'd3'], ['A', 'B', 'A', 'B'], 'tie']),
  ('query sample 21',
   [['d5', 'd3'], ['d2', 'd3', 'd6', 'd4', 'd5'], [1, 1], [], 5],
   [['d2', 'd5', 'd3', 'd6', 'd4'], ['B', 'A', 'B', 'B', 'B'], 'tie']),
  ('query sample 22',
   [['d1', 'd4'], ['d1', 'd6', 'd3', 'd5', 'd2'], [0], [4], 4],
   [['d1', 'd6', 'd4', 'd3'], ['A', 'B', 'A', 'B'], 'tie'])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
shared top document is placed once[['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'B'][['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']Failed
coins are consumed only on ties[['d4', 'd1', 'd5', 'd2', 'd6', 'd3'], ['B', 'A', 'B', 'A', 'B', 'A'], 'tie'][['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']Failed
team behind picks next[['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A'][['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']Passed
shared document credits the team that placed it[['d2', 'd1'], ['B', 'A'], 'B'][['d2', 'd1'], ['B', 'A'], 'B']Passed
exhausted ranking yields[['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B'][['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']Passed
query sample 1[['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie'][['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']Passed
query sample 2[['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie'][['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie']Passed
query sample 3[['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'][['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie']Passed

SHA-256 / 9a1db9bc25fdc53e18fed6d8d112cc21157223da79b3be5be2f0eee417957b59

2 / The unsuccessful fix

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(rank_a, rank_b, coins, clicks, length):
    out, team = [], []
    ia = ib = ci = 0
    while len(out) < length:
        while ia < len(rank_a) and rank_a[ia] in out:
            ia += 1
        while ib < len(rank_b) and rank_b[ib] in out:
            ib += 1
        has_a, has_b = ia < len(rank_a), ib < len(rank_b)
        if not has_a and not has_b:
            break
        na, nb = team.count('A'), team.count('B')
        if na == nb:
            pick_a = coins[len(out) % len(coins)] == 0
            ci += 1
        else:
            pick_a = na < nb
        if pick_a and not has_a:
            pick_a = False
        elif not pick_a and not has_b:
            pick_a = True
        if pick_a:
            out.append(rank_a[ia])
            team.append('A')
        else:
            out.append(rank_b[ib])
            team.append('B')
    credit_a = sum(1 for i in clicks if i < len(out) and team[i] == 'A')
    credit_b = sum(1 for i in clicks if i < len(out) and team[i] == 'B')
    winner = 'A' if credit_a > credit_b else 'B' if credit_b > credit_a else 'tie'
    return [out, team, winner]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('query sample 1',
   [['d4', 'd3', 'd2'], ['d5', 'd6'], [0], [], 6],
   [['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']),
  ('query sample 2',
   [['d6', 'd5', 'd1', 'd2'], ['d6', 'd5'], [1, 1, 1], [], 4],
   [['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie']),
  ('query sample 3',
   [['d5', 'd4'], ['d6', 'd3', 'd5', 'd1'], [0], [], 5],
   [['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'])],
 [('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 6',
   [['d1', 'd4', 'd5'], ['d5', 'd3', 'd2'], [0], [1], 6],
   [['d1', 'd5', 'd4', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'B']),
  ('query sample 7',
   [['d4', 'd1', 'd6'], ['d1', 'd6', 'd2', 'd4', 'd3'], [0, 0, 1], [4, 5], 5],
   [['d4', 'd1', 'd6', 'd2', 'd3'], ['A', 'B', 'A', 'B', 'B'], 'B']),
  ('query sample 12',
   [['d1', 'd4', 'd5', 'd6', 'd2'], ['d6', 'd2', 'd5'], [0, 1, 0], [], 3],
   [['d1', 'd6', 'd2'], ['A', 'B', 'B'], 'tie'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 11',
   [['d4', 'd2', 'd5'], ['d5', 'd2', 'd1', 'd6'], [1, 0], [], 6],
   [['d5', 'd4', 'd2', 'd1', 'd6'], ['B', 'A', 'A', 'B', 'B'], 'tie']),
  ('query sample 12',
   [['d1', 'd4', 'd5', 'd6', 'd2'], ['d6', 'd2', 'd5'], [0, 1, 0], [], 3],
   [['d1', 'd6', 'd2'], ['A', 'B', 'B'], 'tie']),
  ('query sample 41',
   [['d5', 'd2', 'd3', 'd1', 'd4'], ['d5', 'd3'], [1, 0, 1], [1], 6],
   [['d5', 'd2', 'd3', 'd1', 'd4'], ['B', 'A', 'A', 'A', 'A'], 'A'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 16',
   [['d1', 'd3', 'd4', 'd2'], ['d3', 'd2', 'd1', 'd5'], [0, 1, 1], [], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'B', 'A'], 'tie']),
  ('query sample 17',
   [['d4', 'd2'], ['d3', 'd5', 'd6', 'd4', 'd2'], [1, 1], [], 5],
   [['d3', 'd4', 'd5', 'd2', 'd6'], ['B', 'A', 'B', 'A', 'B'], 'tie']),
  ('query sample 51',
   [['d1', 'd2'], ['d5', 'd1', 'd4'], [0, 0, 1], [], 3],
   [['d1', 'd5', 'd2'], ['A', 'B', 'A'], 'tie'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 10',
   [['d4', 'd2', 'd1', 'd3'], ['d2', 'd3', 'd1'], [0, 0, 1], [1, 2], 6],
   [['d4', 'd2', 'd1', 'd3'], ['A', 'B', 'A', 'B'], 'tie']),
  ('query sample 21',
   [['d5', 'd3'], ['d2', 'd3', 'd6', 'd4', 'd5'], [1, 1], [], 5],
   [['d2', 'd5', 'd3', 'd6', 'd4'], ['B', 'A', 'B', 'B', 'B'], 'tie']),
  ('query sample 22',
   [['d1', 'd4'], ['d1', 'd6', 'd3', 'd5', 'd2'], [0], [4], 4],
   [['d1', 'd6', 'd4', 'd3'], ['A', 'B', 'A', 'B'], 'tie'])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
shared top document is placed once[['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'B'][['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']Failed
coins are consumed only on ties[['d4', 'd1', 'd5', 'd2', 'd6', 'd3'], ['B', 'A', 'B', 'A', 'B', 'A'], 'tie'][['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']Failed
team behind picks next[['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A'][['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']Passed
shared document credits the team that placed it[['d2', 'd1'], ['B', 'A'], 'B'][['d2', 'd1'], ['B', 'A'], 'B']Passed
exhausted ranking yields[['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B'][['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']Passed
query sample 1[['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie'][['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']Passed
query sample 2[['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie'][['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie']Passed
query sample 3[['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'][['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie']Passed

SHA-256 / 865cc21251e82abface2dd4ca619d90f35e22db39e34dca711007b5d634a76a8

3 / The verified repair

Exit 0
"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(rank_a, rank_b, coins, clicks, length):
    out, team = [], []
    ia = ib = ci = 0
    while len(out) < length:
        while ia < len(rank_a) and rank_a[ia] in out:
            ia += 1
        while ib < len(rank_b) and rank_b[ib] in out:
            ib += 1
        has_a, has_b = ia < len(rank_a), ib < len(rank_b)
        if not has_a and not has_b:
            break
        na, nb = team.count('A'), team.count('B')
        if na == nb:
            pick_a = coins[ci % len(coins)] == 0
            ci += 1
        else:
            pick_a = na < nb
        if pick_a and not has_a:
            pick_a = False
        elif not pick_a and not has_b:
            pick_a = True
        if pick_a:
            out.append(rank_a[ia])
            team.append('A')
        else:
            out.append(rank_b[ib])
            team.append('B')
    credit_a = sum(1 for i in clicks if i < len(out) and team[i] == 'A')
    credit_b = sum(1 for i in clicks if i < len(out) and team[i] == 'B')
    winner = 'A' if credit_a > credit_b else 'B' if credit_b > credit_a else 'tie'
    return [out, team, winner]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('query sample 1',
   [['d4', 'd3', 'd2'], ['d5', 'd6'], [0], [], 6],
   [['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']),
  ('query sample 2',
   [['d6', 'd5', 'd1', 'd2'], ['d6', 'd5'], [1, 1, 1], [], 4],
   [['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie']),
  ('query sample 3',
   [['d5', 'd4'], ['d6', 'd3', 'd5', 'd1'], [0], [], 5],
   [['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'])],
 [('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 6',
   [['d1', 'd4', 'd5'], ['d5', 'd3', 'd2'], [0], [1], 6],
   [['d1', 'd5', 'd4', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'B']),
  ('query sample 7',
   [['d4', 'd1', 'd6'], ['d1', 'd6', 'd2', 'd4', 'd3'], [0, 0, 1], [4, 5], 5],
   [['d4', 'd1', 'd6', 'd2', 'd3'], ['A', 'B', 'A', 'B', 'B'], 'B']),
  ('query sample 12',
   [['d1', 'd4', 'd5', 'd6', 'd2'], ['d6', 'd2', 'd5'], [0, 1, 0], [], 3],
   [['d1', 'd6', 'd2'], ['A', 'B', 'B'], 'tie'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 11',
   [['d4', 'd2', 'd5'], ['d5', 'd2', 'd1', 'd6'], [1, 0], [], 6],
   [['d5', 'd4', 'd2', 'd1', 'd6'], ['B', 'A', 'A', 'B', 'B'], 'tie']),
  ('query sample 12',
   [['d1', 'd4', 'd5', 'd6', 'd2'], ['d6', 'd2', 'd5'], [0, 1, 0], [], 3],
   [['d1', 'd6', 'd2'], ['A', 'B', 'B'], 'tie']),
  ('query sample 41',
   [['d5', 'd2', 'd3', 'd1', 'd4'], ['d5', 'd3'], [1, 0, 1], [1], 6],
   [['d5', 'd2', 'd3', 'd1', 'd4'], ['B', 'A', 'A', 'A', 'A'], 'A'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 16',
   [['d1', 'd3', 'd4', 'd2'], ['d3', 'd2', 'd1', 'd5'], [0, 1, 1], [], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'B', 'A'], 'tie']),
  ('query sample 17',
   [['d4', 'd2'], ['d3', 'd5', 'd6', 'd4', 'd2'], [1, 1], [], 5],
   [['d3', 'd4', 'd5', 'd2', 'd6'], ['B', 'A', 'B', 'A', 'B'], 'tie']),
  ('query sample 51',
   [['d1', 'd2'], ['d5', 'd1', 'd4'], [0, 0, 1], [], 3],
   [['d1', 'd5', 'd2'], ['A', 'B', 'A'], 'tie'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 10',
   [['d4', 'd2', 'd1', 'd3'], ['d2', 'd3', 'd1'], [0, 0, 1], [1, 2], 6],
   [['d4', 'd2', 'd1', 'd3'], ['A', 'B', 'A', 'B'], 'tie']),
  ('query sample 21',
   [['d5', 'd3'], ['d2', 'd3', 'd6', 'd4', 'd5'], [1, 1], [], 5],
   [['d2', 'd5', 'd3', 'd6', 'd4'], ['B', 'A', 'B', 'B', 'B'], 'tie']),
  ('query sample 22',
   [['d1', 'd4'], ['d1', 'd6', 'd3', 'd5', 'd2'], [0], [4], 4],
   [['d1', 'd6', 'd4', 'd3'], ['A', 'B', 'A', 'B'], 'tie'])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
shared top document is placed once[['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B'][['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']Passed
coins are consumed only on ties[['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B'][['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']Passed
team behind picks next[['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A'][['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']Passed
shared document credits the team that placed it[['d2', 'd1'], ['B', 'A'], 'B'][['d2', 'd1'], ['B', 'A'], 'B']Passed
exhausted ranking yields[['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B'][['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']Passed
query sample 1[['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie'][['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']Passed
query sample 2[['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie'][['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie']Passed
query sample 3[['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'][['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie']Passed

SHA-256 / bfd66c90abc4a17b213edb9b19e37ca249530f7e6fdd4130437e8206196a6816

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:59.771611+00:00.

Case digest / c4cf0e69667057e16257ab4fd913a783600c231e695638e681d261dd06b0a81c