FAILURE MAP
← Case archive

FA-74776 / Experiment statistics / Open access

Team-draft interleaving credit: The team ahead picks again · case 01

One ranker dominates the interleaved list whenever it wins the first coin.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

Without a tie the team with more picks goes next.

VERIFIED REPAIR

Let the team with fewer picks go next.

Unsuccessful approach: Strict alternation ignores the coin-decided starting team.

Case contract

Build an interleaved list of up to length documents. Each round both teams skip documents already placed; when both teams have the same number of picks, the next coin (cycled, consumed only on ties) chooses A on 0 and B on 1; otherwise the team with fewer picks goes. An exhausted team yields to the other. Clicks are positions; each credits the team that placed that document. Return [documents, teams, winner A/B/tie].

Why this case matters

Interleaving compares ranking algorithms with far fewer users than an A/B test, if credit is fair.

1 / The failure

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(rank_a, rank_b, coins, clicks, length):
    out, team = [], []
    ia = ib = ci = 0
    while len(out) < length:
        while ia < len(rank_a) and rank_a[ia] in out:
            ia += 1
        while ib < len(rank_b) and rank_b[ib] in out:
            ib += 1
        has_a, has_b = ia < len(rank_a), ib < len(rank_b)
        if not has_a and not has_b:
            break
        na, nb = team.count('A'), team.count('B')
        if na == nb:
            pick_a = coins[ci % len(coins)] == 0
            ci += 1
        else:
            pick_a = na > nb
        if pick_a and not has_a:
            pick_a = False
        elif not pick_a and not has_b:
            pick_a = True
        if pick_a:
            out.append(rank_a[ia])
            team.append('A')
        else:
            out.append(rank_b[ib])
            team.append('B')
    credit_a = sum(1 for i in clicks if i < len(out) and team[i] == 'A')
    credit_b = sum(1 for i in clicks if i < len(out) and team[i] == 'B')
    winner = 'A' if credit_a > credit_b else 'B' if credit_b > credit_a else 'tie'
    return [out, team, winner]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('query sample 1',
   [['d4', 'd3', 'd2'], ['d5', 'd6'], [0], [], 6],
   [['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']),
  ('query sample 2',
   [['d6', 'd5', 'd1', 'd2'], ['d6', 'd5'], [1, 1, 1], [], 4],
   [['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie']),
  ('query sample 3',
   [['d5', 'd4'], ['d6', 'd3', 'd5', 'd1'], [0], [], 5],
   [['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'])],
 [('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 4',
   [['d5', 'd6'], ['d3', 'd4'], [1, 0], [0, 4], 4],
   [['d3', 'd5', 'd6', 'd4'], ['B', 'A', 'A', 'B'], 'B']),
  ('query sample 6',
   [['d1', 'd4', 'd5'], ['d5', 'd3', 'd2'], [0], [1], 6],
   [['d1', 'd5', 'd4', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'B']),
  ('query sample 14',
   [['d2', 'd6', 'd3', 'd5'], ['d1', 'd4', 'd2', 'd5'], [0, 1], [5], 4],
   [['d2', 'd1', 'd4', 'd6'], ['A', 'B', 'B', 'A'], 'tie'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 11',
   [['d4', 'd2', 'd5'], ['d5', 'd2', 'd1', 'd6'], [1, 0], [], 6],
   [['d5', 'd4', 'd2', 'd1', 'd6'], ['B', 'A', 'A', 'B', 'B'], 'tie']),
  ('query sample 12',
   [['d1', 'd4', 'd5', 'd6', 'd2'], ['d6', 'd2', 'd5'], [0, 1, 0], [], 3],
   [['d1', 'd6', 'd2'], ['A', 'B', 'B'], 'tie']),
  ('query sample 26',
   [['d1', 'd3', 'd5'], ['d1', 'd4', 'd2', 'd5'], [1], [0], 6],
   [['d1', 'd3', 'd4', 'd5', 'd2'], ['B', 'A', 'B', 'A', 'B'], 'B'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 16',
   [['d1', 'd3', 'd4', 'd2'], ['d3', 'd2', 'd1', 'd5'], [0, 1, 1], [], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'B', 'A'], 'tie']),
  ('query sample 18',
   [['d2', 'd5'], ['d5', 'd4'], [1], [5], 4],
   [['d5', 'd2', 'd4'], ['B', 'A', 'B'], 'tie']),
  ('query sample 35',
   [['d5', 'd6', 'd1', 'd4'], ['d2', 'd5', 'd4', 'd3', 'd1'], [1, 0, 0], [0, 1], 3],
   [['d2', 'd5', 'd6'], ['B', 'A', 'A'], 'tie'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 21',
   [['d5', 'd3'], ['d2', 'd3', 'd6', 'd4', 'd5'], [1, 1], [], 5],
   [['d2', 'd5', 'd3', 'd6', 'd4'], ['B', 'A', 'B', 'B', 'B'], 'tie']),
  ('query sample 25',
   [['d5', 'd3', 'd2'], ['d2', 'd3', 'd4'], [1], [0, 2], 5],
   [['d2', 'd5', 'd3', 'd4'], ['B', 'A', 'B', 'B'], 'B']),
  ('query sample 46',
   [['d4', 'd2', 'd5', 'd6'], ['d2', 'd6'], [1, 1], [], 5],
   [['d2', 'd4', 'd6', 'd5'], ['B', 'A', 'B', 'A'], 'tie'])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
shared top document is placed once[['d1', 'd2', 'd3', 'd4'], ['A', 'A', 'A', 'B'], 'A'][['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']Failed
coins are consumed only on ties[['d4', 'd5', 'd6', 'd1', 'd2', 'd3'], ['B', 'B', 'B', 'A', 'A', 'A'], 'tie'][['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']Failed
team behind picks next[['d1', 'd2', 'd3', 'd4'], ['A', 'A', 'B', 'B'], 'B'][['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']Failed
shared document credits the team that placed it[['d2', 'd1'], ['B', 'B'], 'B'][['d2', 'd1'], ['B', 'A'], 'B']Failed
exhausted ranking yields[['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B'][['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']Passed
query sample 1[['d4', 'd3', 'd2', 'd5', 'd6'], ['A', 'A', 'A', 'B', 'B'], 'tie'][['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']Failed
query sample 2[['d6', 'd5', 'd1', 'd2'], ['B', 'B', 'A', 'A'], 'tie'][['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie']Failed
query sample 3[['d5', 'd4', 'd6', 'd3', 'd1'], ['A', 'A', 'B', 'B', 'B'], 'tie'][['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie']Failed

SHA-256 / 8ad69a1f1627a9f1dc9168674c99997728bc4f34bec6484a24de2e52ad27c710

2 / The unsuccessful fix

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(rank_a, rank_b, coins, clicks, length):
    out, team = [], []
    ia = ib = ci = 0
    while len(out) < length:
        while ia < len(rank_a) and rank_a[ia] in out:
            ia += 1
        while ib < len(rank_b) and rank_b[ib] in out:
            ib += 1
        has_a, has_b = ia < len(rank_a), ib < len(rank_b)
        if not has_a and not has_b:
            break
        na, nb = team.count('A'), team.count('B')
        if na == nb:
            pick_a = coins[ci % len(coins)] == 0
            ci += 1
        else:
            pick_a = len(out) % 2 == 0
        if pick_a and not has_a:
            pick_a = False
        elif not pick_a and not has_b:
            pick_a = True
        if pick_a:
            out.append(rank_a[ia])
            team.append('A')
        else:
            out.append(rank_b[ib])
            team.append('B')
    credit_a = sum(1 for i in clicks if i < len(out) and team[i] == 'A')
    credit_b = sum(1 for i in clicks if i < len(out) and team[i] == 'B')
    winner = 'A' if credit_a > credit_b else 'B' if credit_b > credit_a else 'tie'
    return [out, team, winner]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('query sample 1',
   [['d4', 'd3', 'd2'], ['d5', 'd6'], [0], [], 6],
   [['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']),
  ('query sample 2',
   [['d6', 'd5', 'd1', 'd2'], ['d6', 'd5'], [1, 1, 1], [], 4],
   [['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie']),
  ('query sample 3',
   [['d5', 'd4'], ['d6', 'd3', 'd5', 'd1'], [0], [], 5],
   [['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'])],
 [('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 4',
   [['d5', 'd6'], ['d3', 'd4'], [1, 0], [0, 4], 4],
   [['d3', 'd5', 'd6', 'd4'], ['B', 'A', 'A', 'B'], 'B']),
  ('query sample 6',
   [['d1', 'd4', 'd5'], ['d5', 'd3', 'd2'], [0], [1], 6],
   [['d1', 'd5', 'd4', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'B']),
  ('query sample 14',
   [['d2', 'd6', 'd3', 'd5'], ['d1', 'd4', 'd2', 'd5'], [0, 1], [5], 4],
   [['d2', 'd1', 'd4', 'd6'], ['A', 'B', 'B', 'A'], 'tie'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 11',
   [['d4', 'd2', 'd5'], ['d5', 'd2', 'd1', 'd6'], [1, 0], [], 6],
   [['d5', 'd4', 'd2', 'd1', 'd6'], ['B', 'A', 'A', 'B', 'B'], 'tie']),
  ('query sample 12',
   [['d1', 'd4', 'd5', 'd6', 'd2'], ['d6', 'd2', 'd5'], [0, 1, 0], [], 3],
   [['d1', 'd6', 'd2'], ['A', 'B', 'B'], 'tie']),
  ('query sample 26',
   [['d1', 'd3', 'd5'], ['d1', 'd4', 'd2', 'd5'], [1], [0], 6],
   [['d1', 'd3', 'd4', 'd5', 'd2'], ['B', 'A', 'B', 'A', 'B'], 'B'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 16',
   [['d1', 'd3', 'd4', 'd2'], ['d3', 'd2', 'd1', 'd5'], [0, 1, 1], [], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'B', 'A'], 'tie']),
  ('query sample 18',
   [['d2', 'd5'], ['d5', 'd4'], [1], [5], 4],
   [['d5', 'd2', 'd4'], ['B', 'A', 'B'], 'tie']),
  ('query sample 35',
   [['d5', 'd6', 'd1', 'd4'], ['d2', 'd5', 'd4', 'd3', 'd1'], [1, 0, 0], [0, 1], 3],
   [['d2', 'd5', 'd6'], ['B', 'A', 'A'], 'tie'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 21',
   [['d5', 'd3'], ['d2', 'd3', 'd6', 'd4', 'd5'], [1, 1], [], 5],
   [['d2', 'd5', 'd3', 'd6', 'd4'], ['B', 'A', 'B', 'B', 'B'], 'tie']),
  ('query sample 25',
   [['d5', 'd3', 'd2'], ['d2', 'd3', 'd4'], [1], [0, 2], 5],
   [['d2', 'd5', 'd3', 'd4'], ['B', 'A', 'B', 'B'], 'B']),
  ('query sample 46',
   [['d4', 'd2', 'd5', 'd6'], ['d2', 'd6'], [1, 1], [], 5],
   [['d2', 'd4', 'd6', 'd5'], ['B', 'A', 'B', 'A'], 'tie'])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
shared top document is placed once[['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B'][['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']Passed
coins are consumed only on ties[['d4', 'd5', 'd1', 'd6', 'd2', 'd3'], ['B', 'B', 'A', 'B', 'A', 'A'], 'B'][['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']Failed
team behind picks next[['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A'][['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']Passed
shared document credits the team that placed it[['d2', 'd1'], ['B', 'B'], 'B'][['d2', 'd1'], ['B', 'A'], 'B']Failed
exhausted ranking yields[['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B'][['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']Passed
query sample 1[['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie'][['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']Passed
query sample 2[['d6', 'd5', 'd1', 'd2'], ['B', 'B', 'A', 'A'], 'tie'][['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie']Failed
query sample 3[['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'][['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie']Passed

SHA-256 / 10455428c8bf1a2ce877342ae369ee37163dc27d0981da30889973f659ccba77

3 / The verified repair

Exit 0
"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(rank_a, rank_b, coins, clicks, length):
    out, team = [], []
    ia = ib = ci = 0
    while len(out) < length:
        while ia < len(rank_a) and rank_a[ia] in out:
            ia += 1
        while ib < len(rank_b) and rank_b[ib] in out:
            ib += 1
        has_a, has_b = ia < len(rank_a), ib < len(rank_b)
        if not has_a and not has_b:
            break
        na, nb = team.count('A'), team.count('B')
        if na == nb:
            pick_a = coins[ci % len(coins)] == 0
            ci += 1
        else:
            pick_a = na < nb
        if pick_a and not has_a:
            pick_a = False
        elif not pick_a and not has_b:
            pick_a = True
        if pick_a:
            out.append(rank_a[ia])
            team.append('A')
        else:
            out.append(rank_b[ib])
            team.append('B')
    credit_a = sum(1 for i in clicks if i < len(out) and team[i] == 'A')
    credit_b = sum(1 for i in clicks if i < len(out) and team[i] == 'B')
    winner = 'A' if credit_a > credit_b else 'B' if credit_b > credit_a else 'tie'
    return [out, team, winner]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('query sample 1',
   [['d4', 'd3', 'd2'], ['d5', 'd6'], [0], [], 6],
   [['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']),
  ('query sample 2',
   [['d6', 'd5', 'd1', 'd2'], ['d6', 'd5'], [1, 1, 1], [], 4],
   [['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie']),
  ('query sample 3',
   [['d5', 'd4'], ['d6', 'd3', 'd5', 'd1'], [0], [], 5],
   [['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'])],
 [('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 4',
   [['d5', 'd6'], ['d3', 'd4'], [1, 0], [0, 4], 4],
   [['d3', 'd5', 'd6', 'd4'], ['B', 'A', 'A', 'B'], 'B']),
  ('query sample 6',
   [['d1', 'd4', 'd5'], ['d5', 'd3', 'd2'], [0], [1], 6],
   [['d1', 'd5', 'd4', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'B']),
  ('query sample 14',
   [['d2', 'd6', 'd3', 'd5'], ['d1', 'd4', 'd2', 'd5'], [0, 1], [5], 4],
   [['d2', 'd1', 'd4', 'd6'], ['A', 'B', 'B', 'A'], 'tie'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 11',
   [['d4', 'd2', 'd5'], ['d5', 'd2', 'd1', 'd6'], [1, 0], [], 6],
   [['d5', 'd4', 'd2', 'd1', 'd6'], ['B', 'A', 'A', 'B', 'B'], 'tie']),
  ('query sample 12',
   [['d1', 'd4', 'd5', 'd6', 'd2'], ['d6', 'd2', 'd5'], [0, 1, 0], [], 3],
   [['d1', 'd6', 'd2'], ['A', 'B', 'B'], 'tie']),
  ('query sample 26',
   [['d1', 'd3', 'd5'], ['d1', 'd4', 'd2', 'd5'], [1], [0], 6],
   [['d1', 'd3', 'd4', 'd5', 'd2'], ['B', 'A', 'B', 'A', 'B'], 'B'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('shared document credits the team that placed it',
   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],
   [['d2', 'd1'], ['B', 'A'], 'B']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 16',
   [['d1', 'd3', 'd4', 'd2'], ['d3', 'd2', 'd1', 'd5'], [0, 1, 1], [], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'B', 'A'], 'tie']),
  ('query sample 18',
   [['d2', 'd5'], ['d5', 'd4'], [1], [5], 4],
   [['d5', 'd2', 'd4'], ['B', 'A', 'B'], 'tie']),
  ('query sample 35',
   [['d5', 'd6', 'd1', 'd4'], ['d2', 'd5', 'd4', 'd3', 'd1'], [1, 0, 0], [0, 1], 3],
   [['d2', 'd5', 'd6'], ['B', 'A', 'A'], 'tie'])],
 [('shared top document is placed once',
   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],
   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),
  ('coins are consumed only on ties',
   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],
   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),
  ('team behind picks next',
   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],
   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),
  ('exhausted ranking yields',
   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],
   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),
  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),
  ('query sample 21',
   [['d5', 'd3'], ['d2', 'd3', 'd6', 'd4', 'd5'], [1, 1], [], 5],
   [['d2', 'd5', 'd3', 'd6', 'd4'], ['B', 'A', 'B', 'B', 'B'], 'tie']),
  ('query sample 25',
   [['d5', 'd3', 'd2'], ['d2', 'd3', 'd4'], [1], [0, 2], 5],
   [['d2', 'd5', 'd3', 'd4'], ['B', 'A', 'B', 'B'], 'B']),
  ('query sample 46',
   [['d4', 'd2', 'd5', 'd6'], ['d2', 'd6'], [1, 1], [], 5],
   [['d2', 'd4', 'd6', 'd5'], ['B', 'A', 'B', 'A'], 'tie'])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
shared top document is placed once[['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B'][['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']Passed
coins are consumed only on ties[['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B'][['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']Passed
team behind picks next[['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A'][['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']Passed
shared document credits the team that placed it[['d2', 'd1'], ['B', 'A'], 'B'][['d2', 'd1'], ['B', 'A'], 'B']Passed
exhausted ranking yields[['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B'][['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']Passed
query sample 1[['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie'][['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']Passed
query sample 2[['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie'][['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie']Passed
query sample 3[['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'][['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie']Passed

SHA-256 / 665223cb84c1577d74d753d7c6dacd99a0a2c9ddc4d6b7bf714ec43c46f31368

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:59.809650+00:00.

Case digest / eb2a228213aea239a1a0988ae28f1580ce8d5cefdc16434b32d34bedcd6b0d11