FAILURE MAP
← Case archive

FA-74766 / Experiment statistics / Open access

Team-draft interleaving credit: Team B can place a document already shown · case 01

The interleaved page shows the same result twice and B gets extra credit.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

Team B never skips documents that team A already placed.

VERIFIED REPAIR

Skip documents already in the interleaved list for both teams.

Unsuccessful approach: Checking only the most recently placed document misses earlier duplicates.

Case contract

Build an interleaved list of up to length documents. Each round both teams skip documents already placed; when both teams have the same number of picks, the next coin (cycled, consumed only on ties) chooses A on 0 and B on 1; otherwise the team with fewer picks goes. An exhausted team yields to the other. Clicks are positions; each credits the team that placed that document. Return [documents, teams, winner A/B/tie].

Why this case matters

Interleaving compares ranking algorithms with far fewer users than an A/B test, if credit is fair.

1 / The failure

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(rank_a, rank_b, coins, clicks, length):
    out, team = [], []
    ia = ib = ci = 0
    while len(out) < length:
        while ia < len(rank_a) and rank_a[ia] in out:
            ia += 1
        has_a, has_b = ia < len(rank_a), ib < len(rank_b)
        if not has_a and not has_b:
            break
        na, nb = team.count('A'), team.count('B')
        if na == nb:
            pick_a = coins[ci % len(coins)] == 0
            ci += 1
        else:
            pick_a = na < nb
        if pick_a and not has_a:
            pick_a = False
        elif not pick_a and not has_b:
            pick_a = True
        if pick_a:
            out.append(rank_a[ia])
            team.append('A')
        else:
            out.append(rank_b[ib])
            team.append('B')
    credit_a = sum(1 for i in clicks if i < len(out) and team[i] == 'A')
    credit_b = sum(1 for i in clicks if i < len(out) and team[i] == 'B')
    winner = 'A' if credit_a > credit_b else 'B' if credit_b > credit_a else 'tie'
    return [out, team, winner]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 1',
   [['d4', 'd3', 'd2'], ['d5', 'd6'], [0], [], 6],
   [['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']),
  ('query sample 3',
   [['d5', 'd4'], ['d6', 'd3', 'd5', 'd1'], [0], [], 5],
   [['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'])],
 [('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 3',
   [['d5', 'd4'], ['d6', 'd3', 'd5', 'd1'], [0], [], 5],
   [['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie']),
  ('query sample 6',
   [['d1', 'd4', 'd5'], ['d5', 'd3', 'd2'], [0], [1], 6],
   [['d1', 'd5', 'd4', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'B']),
  ('query sample 42',
   [['d5', 'd1'], ['d6', 'd5', 'd3', 'd1', 'd2'], [0, 0, 1], [], 6],
   [['d5', 'd6', 'd1', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'tie'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 10',
   [['d4', 'd2', 'd1', 'd3'], ['d2', 'd3', 'd1'], [0, 0, 1], [1, 2], 6],
   [['d4', 'd2', 'd1', 'd3'], ['A', 'B', 'A', 'B'], 'tie']),
  ('query sample 11',
   [['d4', 'd2', 'd5'], ['d5', 'd2', 'd1', 'd6'], [1, 0], [], 6],
   [['d5', 'd4', 'd2', 'd1', 'd6'], ['B', 'A', 'A', 'B', 'B'], 'tie']),
  ('query sample 26',
   [['d1', 'd3', 'd5'], ['d1', 'd4', 'd2', 'd5'], [1], [0], 6],
   [['d1', 'd3', 'd4', 'd5', 'd2'], ['B', 'A', 'B', 'A', 'B'], 'B'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 5',
   [['d3', 'd2', 'd4'], ['d4', 'd6', 'd2', 'd3'], [1], [], 6],
   [['d4', 'd3', 'd6', 'd2'], ['B', 'A', 'B', 'A'], 'tie']),
  ('query sample 16',
   [['d1', 'd3', 'd4', 'd2'], ['d3', 'd2', 'd1', 'd5'], [0, 1, 1], [], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'B', 'A'], 'tie']),
  ('query sample 17',
   [['d4', 'd2'], ['d3', 'd5', 'd6', 'd4', 'd2'], [1, 1], [], 5],
   [['d3', 'd4', 'd5', 'd2', 'd6'], ['B', 'A', 'B', 'A', 'B'], 'tie'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 21',
   [['d5', 'd3'], ['d2', 'd3', 'd6', 'd4', 'd5'], [1, 1], [], 5],
   [['d2', 'd5', 'd3', 'd6', 'd4'], ['B', 'A', 'B', 'B', 'B'], 'tie']),
  ('query sample 25',
   [['d5', 'd3', 'd2'], ['d2', 'd3', 'd4'], [1], [0, 2], 5],
   [['d2', 'd5', 'd3', 'd4'], ['B', 'A', 'B', 'B'], 'B']),
  ('query sample 54',
   [['d4', 'd1', 'd6'], ['d6', 'd5', 'd1', 'd4', 'd2'], [0, 0], [], 5],
   [['d4', 'd6', 'd1', 'd5', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'tie'])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
shared top document is placed once[['d1', 'd1', 'd1', 'd2'], ['A', 'B', 'B', 'A'], 'B'][['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']Failed
coins are consumed only on ties[['d4', 'd1', 'd2', 'd4', 'd4', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B'][['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']Failed
team behind picks next[['d1', 'd3', 'd2', 'd3'], ['A', 'B', 'A', 'B'], 'A'][['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']Failed
shared document credits the team that placed it[['d2', 'd1'], ['B', 'A'], 'B'][['d2', 'd1'], ['B', 'A'], 'B']Passed
exhausted ranking yields[['d1', 'd2', 'd2', 'd2'], ['A', 'B', 'B', 'B'], 'B'][['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']Failed
no clicks is a tie[['d3', 'd1', 'd3'], ['B', 'A', 'B'], 'tie'][['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']Failed
query sample 1[['d4', 'd5', 'd3', 'd5', 'd2', 'd5'], ['A', 'B', 'A', 'B', 'A', 'B'], 'tie'][['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']Failed
query sample 3[['d5', 'd6', 'd4', 'd6', 'd6'], ['A', 'B', 'A', 'B', 'B'], 'tie'][['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie']Failed

SHA-256 / 9171bfb080ce85eeadb7c83b929c39e27e73529607d150045eee9571bc3e6a9f

2 / The unsuccessful fix

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(rank_a, rank_b, coins, clicks, length):
    out, team = [], []
    ia = ib = ci = 0
    while len(out) < length:
        while ia < len(rank_a) and rank_a[ia] in out:
            ia += 1
        while ib < len(rank_b) and rank_b[ib] in out[-1:]:
            ib += 1
        has_a, has_b = ia < len(rank_a), ib < len(rank_b)
        if not has_a and not has_b:
            break
        na, nb = team.count('A'), team.count('B')
        if na == nb:
            pick_a = coins[ci % len(coins)] == 0
            ci += 1
        else:
            pick_a = na < nb
        if pick_a and not has_a:
            pick_a = False
        elif not pick_a and not has_b:
            pick_a = True
        if pick_a:
            out.append(rank_a[ia])
            team.append('A')
        else:
            out.append(rank_b[ib])
            team.append('B')
    credit_a = sum(1 for i in clicks if i < len(out) and team[i] == 'A')
    credit_b = sum(1 for i in clicks if i < len(out) and team[i] == 'B')
    winner = 'A' if credit_a > credit_b else 'B' if credit_b > credit_a else 'tie'
    return [out, team, winner]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 1',
   [['d4', 'd3', 'd2'], ['d5', 'd6'], [0], [], 6],
   [['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']),
  ('query sample 3',
   [['d5', 'd4'], ['d6', 'd3', 'd5', 'd1'], [0], [], 5],
   [['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'])],
 [('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 3',
   [['d5', 'd4'], ['d6', 'd3', 'd5', 'd1'], [0], [], 5],
   [['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie']),
  ('query sample 6',
   [['d1', 'd4', 'd5'], ['d5', 'd3', 'd2'], [0], [1], 6],
   [['d1', 'd5', 'd4', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'B']),
  ('query sample 42',
   [['d5', 'd1'], ['d6', 'd5', 'd3', 'd1', 'd2'], [0, 0, 1], [], 6],
   [['d5', 'd6', 'd1', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'tie'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 10',
   [['d4', 'd2', 'd1', 'd3'], ['d2', 'd3', 'd1'], [0, 0, 1], [1, 2], 6],
   [['d4', 'd2', 'd1', 'd3'], ['A', 'B', 'A', 'B'], 'tie']),
  ('query sample 11',
   [['d4', 'd2', 'd5'], ['d5', 'd2', 'd1', 'd6'], [1, 0], [], 6],
   [['d5', 'd4', 'd2', 'd1', 'd6'], ['B', 'A', 'A', 'B', 'B'], 'tie']),
  ('query sample 26',
   [['d1', 'd3', 'd5'], ['d1', 'd4', 'd2', 'd5'], [1], [0], 6],
   [['d1', 'd3', 'd4', 'd5', 'd2'], ['B', 'A', 'B', 'A', 'B'], 'B'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 5',
   [['d3', 'd2', 'd4'], ['d4', 'd6', 'd2', 'd3'], [1], [], 6],
   [['d4', 'd3', 'd6', 'd2'], ['B', 'A', 'B', 'A'], 'tie']),
  ('query sample 16',
   [['d1', 'd3', 'd4', 'd2'], ['d3', 'd2', 'd1', 'd5'], [0, 1, 1], [], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'B', 'A'], 'tie']),
  ('query sample 17',
   [['d4', 'd2'], ['d3', 'd5', 'd6', 'd4', 'd2'], [1, 1], [], 5],
   [['d3', 'd4', 'd5', 'd2', 'd6'], ['B', 'A', 'B', 'A', 'B'], 'tie'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 21',
   [['d5', 'd3'], ['d2', 'd3', 'd6', 'd4', 'd5'], [1, 1], [], 5],
   [['d2', 'd5', 'd3', 'd6', 'd4'], ['B', 'A', 'B', 'B', 'B'], 'tie']),
  ('query sample 25',
   [['d5', 'd3', 'd2'], ['d2', 'd3', 'd4'], [1], [0, 2], 5],
   [['d2', 'd5', 'd3', 'd4'], ['B', 'A', 'B', 'B'], 'B']),
  ('query sample 54',
   [['d4', 'd1', 'd6'], ['d6', 'd5', 'd1', 'd4', 'd2'], [0, 0], [], 5],
   [['d4', 'd6', 'd1', 'd5', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'tie'])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
shared top document is placed once[['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B'][['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']Passed
coins are consumed only on ties[['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B'][['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']Passed
team behind picks next[['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A'][['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']Passed
shared document credits the team that placed it[['d2', 'd1'], ['B', 'A'], 'B'][['d2', 'd1'], ['B', 'A'], 'B']Passed
exhausted ranking yields[['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B'][['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']Passed
no clicks is a tie[['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie'][['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']Passed
query sample 1[['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie'][['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']Passed
query sample 3[['d5', 'd6', 'd4', 'd3', 'd5'], ['A', 'B', 'A', 'B', 'B'], 'tie'][['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie']Failed

SHA-256 / 9f32bfd6415e85b364f2296f349350c53c710ce27289c6b79b758ef383f075e6

3 / The verified repair

Exit 0
"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(rank_a, rank_b, coins, clicks, length):
    out, team = [], []
    ia = ib = ci = 0
    while len(out) < length:
        while ia < len(rank_a) and rank_a[ia] in out:
            ia += 1
        while ib < len(rank_b) and rank_b[ib] in out:
            ib += 1
        has_a, has_b = ia < len(rank_a), ib < len(rank_b)
        if not has_a and not has_b:
            break
        na, nb = team.count('A'), team.count('B')
        if na == nb:
            pick_a = coins[ci % len(coins)] == 0
            ci += 1
        else:
            pick_a = na < nb
        if pick_a and not has_a:
            pick_a = False
        elif not pick_a and not has_b:
            pick_a = True
        if pick_a:
            out.append(rank_a[ia])
            team.append('A')
        else:
            out.append(rank_b[ib])
            team.append('B')
    credit_a = sum(1 for i in clicks if i < len(out) and team[i] == 'A')
    credit_b = sum(1 for i in clicks if i < len(out) and team[i] == 'B')
    winner = 'A' if credit_a > credit_b else 'B' if credit_b > credit_a else 'tie'
    return [out, team, winner]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 1',
   [['d4', 'd3', 'd2'], ['d5', 'd6'], [0], [], 6],
   [['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']),
  ('query sample 3',
   [['d5', 'd4'], ['d6', 'd3', 'd5', 'd1'], [0], [], 5],
   [['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'])],
 [('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 3',
   [['d5', 'd4'], ['d6', 'd3', 'd5', 'd1'], [0], [], 5],
   [['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie']),
  ('query sample 6',
   [['d1', 'd4', 'd5'], ['d5', 'd3', 'd2'], [0], [1], 6],
   [['d1', 'd5', 'd4', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'B']),
  ('query sample 42',
   [['d5', 'd1'], ['d6', 'd5', 'd3', 'd1', 'd2'], [0, 0, 1], [], 6],
   [['d5', 'd6', 'd1', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'tie'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 10',
   [['d4', 'd2', 'd1', 'd3'], ['d2', 'd3', 'd1'], [0, 0, 1], [1, 2], 6],
   [['d4', 'd2', 'd1', 'd3'], ['A', 'B', 'A', 'B'], 'tie']),
  ('query sample 11',
   [['d4', 'd2', 'd5'], ['d5', 'd2', 'd1', 'd6'], [1, 0], [], 6],
   [['d5', 'd4', 'd2', 'd1', 'd6'], ['B', 'A', 'A', 'B', 'B'], 'tie']),
  ('query sample 26',
   [['d1', 'd3', 'd5'], ['d1', 'd4', 'd2', 'd5'], [1], [0], 6],
   [['d1', 'd3', 'd4', 'd5', 'd2'], ['B', 'A', 'B', 'A', 'B'], 'B'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 5',
   [['d3', 'd2', 'd4'], ['d4', 'd6', 'd2', 'd3'], [1], [], 6],
   [['d4', 'd3', 'd6', 'd2'], ['B', 'A', 'B', 'A'], 'tie']),
  ('query sample 16',
   [['d1', 'd3', 'd4', 'd2'], ['d3', 'd2', 'd1', 'd5'], [0, 1, 1], [], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'B', 'A'], 'tie']),
  ('query sample 17',
   [['d4', 'd2'], ['d3', 'd5', 'd6', 'd4', 'd2'], [1, 1], [], 5],
   [['d3', 'd4', 'd5', 'd2', 'd6'], ['B', 'A', 'B', 'A', 'B'], 'tie'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 21',
   [['d5', 'd3'], ['d2', 'd3', 'd6', 'd4', 'd5'], [1, 1], [], 5],
   [['d2', 'd5', 'd3', 'd6', 'd4'], ['B', 'A', 'B', 'B', 'B'], 'tie']),
  ('query sample 25',
   [['d5', 'd3', 'd2'], ['d2', 'd3', 'd4'], [1], [0, 2], 5],
   [['d2', 'd5', 'd3', 'd4'], ['B', 'A', 'B', 'B'], 'B']),
  ('query sample 54',
   [['d4', 'd1', 'd6'], ['d6', 'd5', 'd1', 'd4', 'd2'], [0, 0], [], 5],
   [['d4', 'd6', 'd1', 'd5', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'tie'])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
shared top document is placed once[['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B'][['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']Passed
coins are consumed only on ties[['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B'][['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']Passed
team behind picks next[['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A'][['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']Passed
shared document credits the team that placed it[['d2', 'd1'], ['B', 'A'], 'B'][['d2', 'd1'], ['B', 'A'], 'B']Passed
exhausted ranking yields[['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B'][['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']Passed
no clicks is a tie[['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie'][['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']Passed
query sample 1[['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie'][['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']Passed
query sample 3[['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'][['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie']Passed

SHA-256 / 50eec6ba2c9fe3484688dd7fa7b24e0492dea8a790721486e8bbd9401ab03a1a

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:59.754264+00:00.

Case digest / 28790ffb6f0ae8f6e5b871dce5fbc695fa90dab5b922c5299637db25385da43b