{"abstract":"Clicks on documents both rankers returned count for both, masking real preferences.","category":"Experiment statistics","checks":8,"contract":"Build an interleaved list of up to length documents. Each round both teams skip documents already placed; when both teams have the same number of picks, the next coin (cycled, consumed only on ties) chooses A on 0 and B on 1; otherwise the team with fewer picks goes. An exhausted team yields to the other. Clicks are positions; each credits the team that placed that document. Return [documents, teams, winner A/B/tie].","evaluation_group":"w2-experiment-statistics-team-draft-interleaving","failed_approach":"Crediting only documents exclusive to one ranking ignores the team that actually placed shared ones.","family":"w2-experiment-statistics-team-draft-interleaving-click-credit","id":"FA-74781","implementations":{"attempt":{"sha256":"9abcd321f8454ec839361a4f43663fb9956e9703a3de0744d1bae3f8bb821651","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(rank_a, rank_b, coins, clicks, length):\n    out, team = [], []\n    ia = ib = ci = 0\n    while len(out) < length:\n        while ia < len(rank_a) and rank_a[ia] in out:\n            ia += 1\n        while ib < len(rank_b) and rank_b[ib] in out:\n            ib += 1\n        has_a, has_b = ia < len(rank_a), ib < len(rank_b)\n        if not has_a and not has_b:\n            break\n        na, nb = team.count('A'), team.count('B')\n        if na == nb:\n            pick_a = coins[ci % len(coins)] == 0\n            ci += 1\n        else:\n            pick_a = na < nb\n        if pick_a and not has_a:\n            pick_a = False\n        elif not pick_a and not has_b:\n            pick_a = True\n        if pick_a:\n            out.append(rank_a[ia])\n            team.append('A')\n        else:\n            out.append(rank_b[ib])\n            team.append('B')\n    credit_a = sum(1 for i in clicks if i < len(out) and out[i] in rank_a and out[i] not in rank_b)\n    credit_b = sum(1 for i in clicks if i < len(out) and out[i] in rank_b and out[i] not in rank_a)\n    winner = 'A' if credit_a > credit_b else 'B' if credit_b > credit_a else 'tie'\n    return [out, team, winner]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('shared top document is placed once',\n   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],\n   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),\n  ('coins are consumed only on ties',\n   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],\n   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),\n  ('team behind picks next',\n   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),\n  ('shared document credits the team that placed it',\n   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],\n   [['d2', 'd1'], ['B', 'A'], 'B']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('query sample 1',\n   [['d4', 'd3', 'd2'], ['d5', 'd6'], [0], [], 6],\n   [['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']),\n  ('query sample 2',\n   [['d6', 'd5', 'd1', 'd2'], ['d6', 'd5'], [1, 1, 1], [], 4],\n   [['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie']),\n  ('query sample 3',\n   [['d5', 'd4'], ['d6', 'd3', 'd5', 'd1'], [0], [], 5],\n   [['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'])],\n [('coins are consumed only on ties',\n   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],\n   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),\n  ('team behind picks next',\n   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),\n  ('shared document credits the team that placed it',\n   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],\n   [['d2', 'd1'], ['B', 'A'], 'B']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),\n  ('query sample 6',\n   [['d1', 'd4', 'd5'], ['d5', 'd3', 'd2'], [0], [1], 6],\n   [['d1', 'd5', 'd4', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'B']),\n  ('query sample 7',\n   [['d4', 'd1', 'd6'], ['d1', 'd6', 'd2', 'd4', 'd3'], [0, 0, 1], [4, 5], 5],\n   [['d4', 'd1', 'd6', 'd2', 'd3'], ['A', 'B', 'A', 'B', 'B'], 'B']),\n  ('query sample 31',\n   [['d2', 'd4', 'd5', 'd6', 'd3'], ['d1', 'd3'], [1], [2, 5], 5],\n   [['d1', 'd2', 'd3', 'd4', 'd5'], ['B', 'A', 'B', 'A', 'A'], 'B'])],\n [('shared top document is placed once',\n   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],\n   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),\n  ('team behind picks next',\n   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),\n  ('shared document credits the team that placed it',\n   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],\n   [['d2', 'd1'], ['B', 'A'], 'B']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),\n  ('query sample 6',\n   [['d1', 'd4', 'd5'], ['d5', 'd3', 'd2'], [0], [1], 6],\n   [['d1', 'd5', 'd4', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'B']),\n  ('query sample 11',\n   [['d4', 'd2', 'd5'], ['d5', 'd2', 'd1', 'd6'], [1, 0], [], 6],\n   [['d5', 'd4', 'd2', 'd1', 'd6'], ['B', 'A', 'A', 'B', 'B'], 'tie']),\n  ('query sample 12',\n   [['d1', 'd4', 'd5', 'd6', 'd2'], ['d6', 'd2', 'd5'], [0, 1, 0], [], 3],\n   [['d1', 'd6', 'd2'], ['A', 'B', 'B'], 'tie'])],\n [('shared top document is placed once',\n   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],\n   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),\n  ('coins are consumed only on ties',\n   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],\n   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),\n  ('shared document credits the team that placed it',\n   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],\n   [['d2', 'd1'], ['B', 'A'], 'B']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),\n  ('query sample 16',\n   [['d1', 'd3', 'd4', 'd2'], ['d3', 'd2', 'd1', 'd5'], [0, 1, 1], [], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'B', 'A'], 'tie']),\n  ('query sample 17',\n   [['d4', 'd2'], ['d3', 'd5', 'd6', 'd4', 'd2'], [1, 1], [], 5],\n   [['d3', 'd4', 'd5', 'd2', 'd6'], ['B', 'A', 'B', 'A', 'B'], 'tie']),\n  ('query sample 45',\n   [['d4', 'd1'], ['d5', 'd2', 'd3', 'd4'], [0, 1], [0, 5], 4],\n   [['d4', 'd5', 'd2', 'd1'], ['A', 'B', 'B', 'A'], 'A'])],\n [('shared top document is placed once',\n   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],\n   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),\n  ('coins are consumed only on ties',\n   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],\n   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),\n  ('team behind picks next',\n   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),\n  ('query sample 21',\n   [['d5', 'd3'], ['d2', 'd3', 'd6', 'd4', 'd5'], [1, 1], [], 5],\n   [['d2', 'd5', 'd3', 'd6', 'd4'], ['B', 'A', 'B', 'B', 'B'], 'tie']),\n  ('query sample 22',\n   [['d1', 'd4'], ['d1', 'd6', 'd3', 'd5', 'd2'], [0], [4], 4],\n   [['d1', 'd6', 'd4', 'd3'], ['A', 'B', 'A', 'B'], 'tie']),\n  ('query sample 25',\n   [['d5', 'd3', 'd2'], ['d2', 'd3', 'd4'], [1], [0, 2], 5],\n   [['d2', 'd5', 'd3', 'd4'], ['B', 'A', 'B', 'B'], 'B'])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"},"broken":{"sha256":"7f70b9a14b1771f850fddd07b1955f46fc9ca0cbf1db16ec5ca73d14073875a1","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(rank_a, rank_b, coins, clicks, length):\n    out, team = [], []\n    ia = ib = ci = 0\n    while len(out) < length:\n        while ia < len(rank_a) and rank_a[ia] in out:\n            ia += 1\n        while ib < len(rank_b) and rank_b[ib] in out:\n            ib += 1\n        has_a, has_b = ia < len(rank_a), ib < len(rank_b)\n        if not has_a and not has_b:\n            break\n        na, nb = team.count('A'), team.count('B')\n        if na == nb:\n            pick_a = coins[ci % len(coins)] == 0\n            ci += 1\n        else:\n            pick_a = na < nb\n        if pick_a and not has_a:\n            pick_a = False\n        elif not pick_a and not has_b:\n            pick_a = True\n        if pick_a:\n            out.append(rank_a[ia])\n            team.append('A')\n        else:\n            out.append(rank_b[ib])\n            team.append('B')\n    credit_a = sum(1 for i in clicks if i < len(out) and out[i] in rank_a)\n    credit_b = sum(1 for i in clicks if i < len(out) and out[i] in rank_b)\n    winner = 'A' if credit_a > credit_b else 'B' if credit_b > credit_a else 'tie'\n    return [out, team, winner]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('shared top document is placed once',\n   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],\n   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),\n  ('coins are consumed only on ties',\n   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],\n   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),\n  ('team behind picks next',\n   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),\n  ('shared document credits the team that placed it',\n   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],\n   [['d2', 'd1'], ['B', 'A'], 'B']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('query sample 1',\n   [['d4', 'd3', 'd2'], ['d5', 'd6'], [0], [], 6],\n   [['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']),\n  ('query sample 2',\n   [['d6', 'd5', 'd1', 'd2'], ['d6', 'd5'], [1, 1, 1], [], 4],\n   [['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie']),\n  ('query sample 3',\n   [['d5', 'd4'], ['d6', 'd3', 'd5', 'd1'], [0], [], 5],\n   [['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'])],\n [('coins are consumed only on ties',\n   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],\n   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),\n  ('team behind picks next',\n   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),\n  ('shared document credits the team that placed it',\n   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],\n   [['d2', 'd1'], ['B', 'A'], 'B']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),\n  ('query sample 6',\n   [['d1', 'd4', 'd5'], ['d5', 'd3', 'd2'], [0], [1], 6],\n   [['d1', 'd5', 'd4', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'B']),\n  ('query sample 7',\n   [['d4', 'd1', 'd6'], ['d1', 'd6', 'd2', 'd4', 'd3'], [0, 0, 1], [4, 5], 5],\n   [['d4', 'd1', 'd6', 'd2', 'd3'], ['A', 'B', 'A', 'B', 'B'], 'B']),\n  ('query sample 31',\n   [['d2', 'd4', 'd5', 'd6', 'd3'], ['d1', 'd3'], [1], [2, 5], 5],\n   [['d1', 'd2', 'd3', 'd4', 'd5'], ['B', 'A', 'B', 'A', 'A'], 'B'])],\n [('shared top document is placed once',\n   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],\n   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),\n  ('team behind picks next',\n   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),\n  ('shared document credits the team that placed it',\n   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],\n   [['d2', 'd1'], ['B', 'A'], 'B']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),\n  ('query sample 6',\n   [['d1', 'd4', 'd5'], ['d5', 'd3', 'd2'], [0], [1], 6],\n   [['d1', 'd5', 'd4', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'B']),\n  ('query sample 11',\n   [['d4', 'd2', 'd5'], ['d5', 'd2', 'd1', 'd6'], [1, 0], [], 6],\n   [['d5', 'd4', 'd2', 'd1', 'd6'], ['B', 'A', 'A', 'B', 'B'], 'tie']),\n  ('query sample 12',\n   [['d1', 'd4', 'd5', 'd6', 'd2'], ['d6', 'd2', 'd5'], [0, 1, 0], [], 3],\n   [['d1', 'd6', 'd2'], ['A', 'B', 'B'], 'tie'])],\n [('shared top document is placed once',\n   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],\n   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),\n  ('coins are consumed only on ties',\n   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],\n   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),\n  ('shared document credits the team that placed it',\n   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],\n   [['d2', 'd1'], ['B', 'A'], 'B']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),\n  ('query sample 16',\n   [['d1', 'd3', 'd4', 'd2'], ['d3', 'd2', 'd1', 'd5'], [0, 1, 1], [], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'B', 'A'], 'tie']),\n  ('query sample 17',\n   [['d4', 'd2'], ['d3', 'd5', 'd6', 'd4', 'd2'], [1, 1], [], 5],\n   [['d3', 'd4', 'd5', 'd2', 'd6'], ['B', 'A', 'B', 'A', 'B'], 'tie']),\n  ('query sample 45',\n   [['d4', 'd1'], ['d5', 'd2', 'd3', 'd4'], [0, 1], [0, 5], 4],\n   [['d4', 'd5', 'd2', 'd1'], ['A', 'B', 'B', 'A'], 'A'])],\n [('shared top document is placed once',\n   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],\n   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),\n  ('coins are consumed only on ties',\n   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],\n   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),\n  ('team behind picks next',\n   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),\n  ('query sample 21',\n   [['d5', 'd3'], ['d2', 'd3', 'd6', 'd4', 'd5'], [1, 1], [], 5],\n   [['d2', 'd5', 'd3', 'd6', 'd4'], ['B', 'A', 'B', 'B', 'B'], 'tie']),\n  ('query sample 22',\n   [['d1', 'd4'], ['d1', 'd6', 'd3', 'd5', 'd2'], [0], [4], 4],\n   [['d1', 'd6', 'd4', 'd3'], ['A', 'B', 'A', 'B'], 'tie']),\n  ('query sample 25',\n   [['d5', 'd3', 'd2'], ['d2', 'd3', 'd4'], [1], [0, 2], 5],\n   [['d2', 'd5', 'd3', 'd4'], ['B', 'A', 'B', 'B'], 'B'])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"},"fixed":{"sha256":"afa31581831271dcff445b66bf7f1cab674fc6fab2f19df7b93da800f20aca3f","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(rank_a, rank_b, coins, clicks, length):\n    out, team = [], []\n    ia = ib = ci = 0\n    while len(out) < length:\n        while ia < len(rank_a) and rank_a[ia] in out:\n            ia += 1\n        while ib < len(rank_b) and rank_b[ib] in out:\n            ib += 1\n        has_a, has_b = ia < len(rank_a), ib < len(rank_b)\n        if not has_a and not has_b:\n            break\n        na, nb = team.count('A'), team.count('B')\n        if na == nb:\n            pick_a = coins[ci % len(coins)] == 0\n            ci += 1\n        else:\n            pick_a = na < nb\n        if pick_a and not has_a:\n            pick_a = False\n        elif not pick_a and not has_b:\n            pick_a = True\n        if pick_a:\n            out.append(rank_a[ia])\n            team.append('A')\n        else:\n            out.append(rank_b[ib])\n            team.append('B')\n    credit_a = sum(1 for i in clicks if i < len(out) and team[i] == 'A')\n    credit_b = sum(1 for i in clicks if i < len(out) and team[i] == 'B')\n    winner = 'A' if credit_a > credit_b else 'B' if credit_b > credit_a else 'tie'\n    return [out, team, winner]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('shared top document is placed once',\n   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],\n   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),\n  ('coins are consumed only on ties',\n   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],\n   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),\n  ('team behind picks next',\n   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),\n  ('shared document credits the team that placed it',\n   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],\n   [['d2', 'd1'], ['B', 'A'], 'B']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('query sample 1',\n   [['d4', 'd3', 'd2'], ['d5', 'd6'], [0], [], 6],\n   [['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']),\n  ('query sample 2',\n   [['d6', 'd5', 'd1', 'd2'], ['d6', 'd5'], [1, 1, 1], [], 4],\n   [['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie']),\n  ('query sample 3',\n   [['d5', 'd4'], ['d6', 'd3', 'd5', 'd1'], [0], [], 5],\n   [['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'])],\n [('coins are consumed only on ties',\n   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],\n   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),\n  ('team behind picks next',\n   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),\n  ('shared document credits the team that placed it',\n   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],\n   [['d2', 'd1'], ['B', 'A'], 'B']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),\n  ('query sample 6',\n   [['d1', 'd4', 'd5'], ['d5', 'd3', 'd2'], [0], [1], 6],\n   [['d1', 'd5', 'd4', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'B']),\n  ('query sample 7',\n   [['d4', 'd1', 'd6'], ['d1', 'd6', 'd2', 'd4', 'd3'], [0, 0, 1], [4, 5], 5],\n   [['d4', 'd1', 'd6', 'd2', 'd3'], ['A', 'B', 'A', 'B', 'B'], 'B']),\n  ('query sample 31',\n   [['d2', 'd4', 'd5', 'd6', 'd3'], ['d1', 'd3'], [1], [2, 5], 5],\n   [['d1', 'd2', 'd3', 'd4', 'd5'], ['B', 'A', 'B', 'A', 'A'], 'B'])],\n [('shared top document is placed once',\n   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],\n   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),\n  ('team behind picks next',\n   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),\n  ('shared document credits the team that placed it',\n   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],\n   [['d2', 'd1'], ['B', 'A'], 'B']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),\n  ('query sample 6',\n   [['d1', 'd4', 'd5'], ['d5', 'd3', 'd2'], [0], [1], 6],\n   [['d1', 'd5', 'd4', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'B']),\n  ('query sample 11',\n   [['d4', 'd2', 'd5'], ['d5', 'd2', 'd1', 'd6'], [1, 0], [], 6],\n   [['d5', 'd4', 'd2', 'd1', 'd6'], ['B', 'A', 'A', 'B', 'B'], 'tie']),\n  ('query sample 12',\n   [['d1', 'd4', 'd5', 'd6', 'd2'], ['d6', 'd2', 'd5'], [0, 1, 0], [], 3],\n   [['d1', 'd6', 'd2'], ['A', 'B', 'B'], 'tie'])],\n [('shared top document is placed once',\n   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],\n   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),\n  ('coins are consumed only on ties',\n   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],\n   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),\n  ('shared document credits the team that placed it',\n   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],\n   [['d2', 'd1'], ['B', 'A'], 'B']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),\n  ('query sample 16',\n   [['d1', 'd3', 'd4', 'd2'], ['d3', 'd2', 'd1', 'd5'], [0, 1, 1], [], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'B', 'A'], 'tie']),\n  ('query sample 17',\n   [['d4', 'd2'], ['d3', 'd5', 'd6', 'd4', 'd2'], [1, 1], [], 5],\n   [['d3', 'd4', 'd5', 'd2', 'd6'], ['B', 'A', 'B', 'A', 'B'], 'tie']),\n  ('query sample 45',\n   [['d4', 'd1'], ['d5', 'd2', 'd3', 'd4'], [0, 1], [0, 5], 4],\n   [['d4', 'd5', 'd2', 'd1'], ['A', 'B', 'B', 'A'], 'A'])],\n [('shared top document is placed once',\n   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],\n   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),\n  ('coins are consumed only on ties',\n   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],\n   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),\n  ('team behind picks next',\n   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),\n  ('query sample 21',\n   [['d5', 'd3'], ['d2', 'd3', 'd6', 'd4', 'd5'], [1, 1], [], 5],\n   [['d2', 'd5', 'd3', 'd6', 'd4'], ['B', 'A', 'B', 'B', 'B'], 'tie']),\n  ('query sample 22',\n   [['d1', 'd4'], ['d1', 'd6', 'd3', 'd5', 'd2'], [0], [4], 4],\n   [['d1', 'd6', 'd4', 'd3'], ['A', 'B', 'A', 'B'], 'tie']),\n  ('query sample 25',\n   [['d5', 'd3', 'd2'], ['d2', 'd3', 'd4'], [1], [0, 2], 5],\n   [['d2', 'd5', 'd3', 'd4'], ['B', 'A', 'B', 'B'], 'B'])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"}},"limitations":"A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.","method":"Deterministic executable model with adversarial boundary fixtures.","provenance":{"created_by":"Failure Map","dependencies":"Python standard library","family":"w2-experiment-statistics-team-draft-interleaving-click-credit","generated_at":"2026-09-29T14:48:59.851164+00:00","license":"CC0-1.0","python":"3.12.14","seed":1,"split":"open-access"},"relevance":"Interleaving compares ranking algorithms with far fewer users than an A/B test, if credit is fair.","repair":"Credit only the team recorded for that position.","root_cause":"Credit checks whether the clicked document appears in each ranking rather than who placed it.","sha256":"5f2036c702e93516ea8349e608e7511ab689c0ee84c45d43b553638a83f60118","title":"Team-draft interleaving credit: Clicks credit every ranker containing the document · case 01","variant":1,"variant_policy":"Five numbered records share a model and may reuse boundary fixtures.","verification":{"attempt":{"elapsed_ms":40.517,"exit_code":1,"observations":[{"actual":[["d1","d3","d4","d2"],["A","B","B","A"],"tie"],"check":"shared top document is placed once","expected":[["d1","d3","d4","d2"],["A","B","B","A"],"B"],"passed":false},{"actual":[["d4","d1","d2","d5","d6","d3"],["B","A","A","B","B","A"],"B"],"check":"coins are consumed only on ties","expected":[["d4","d1","d2","d5","d6","d3"],["B","A","A","B","B","A"],"B"],"passed":true},{"actual":[["d1","d3","d2","d4"],["A","B","A","B"],"A"],"check":"team behind picks next","expected":[["d1","d3","d2","d4"],["A","B","A","B"],"A"],"passed":true},{"actual":[["d2","d1"],["B","A"],"tie"],"check":"shared document credits the team that placed it","expected":[["d2","d1"],["B","A"],"B"],"passed":false},{"actual":[["d1","d2","d3","d4"],["A","B","B","B"],"B"],"check":"exhausted ranking yields","expected":[["d1","d2","d3","d4"],["A","B","B","B"],"B"],"passed":true},{"actual":[["d4","d5","d3","d6","d2"],["A","B","A","B","A"],"tie"],"check":"query sample 1","expected":[["d4","d5","d3","d6","d2"],["A","B","A","B","A"],"tie"],"passed":true},{"actual":[["d6","d5","d1","d2"],["B","A","A","A"],"tie"],"check":"query sample 2","expected":[["d6","d5","d1","d2"],["B","A","A","A"],"tie"],"passed":true},{"actual":[["d5","d6","d4","d3","d1"],["A","B","A","B","B"],"tie"],"check":"query sample 3","expected":[["d5","d6","d4","d3","d1"],["A","B","A","B","B"],"tie"],"passed":true}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"shared top document is placed once\", \"actual\": [[\"d1\", \"d3\", \"d4\", \"d2\"], [\"A\", \"B\", \"B\", \"A\"], \"tie\"], \"expected\": [[\"d1\", \"d3\", \"d4\", \"d2\"], [\"A\", \"B\", \"B\", \"A\"], \"B\"], \"passed\": false}, {\"check\": \"coins are consumed only on ties\", \"actual\": [[\"d4\", \"d1\", \"d2\", \"d5\", \"d6\", \"d3\"], [\"B\", \"A\", \"A\", \"B\", \"B\", \"A\"], \"B\"], \"expected\": [[\"d4\", \"d1\", \"d2\", \"d5\", \"d6\", \"d3\"], [\"B\", \"A\", \"A\", \"B\", \"B\", \"A\"], \"B\"], \"passed\": true}, {\"check\": \"team behind picks next\", \"actual\": [[\"d1\", \"d3\", \"d2\", \"d4\"], [\"A\", \"B\", \"A\", \"B\"], \"A\"], \"expected\": [[\"d1\", \"d3\", \"d2\", \"d4\"], [\"A\", \"B\", \"A\", \"B\"], \"A\"], \"passed\": true}, {\"check\": \"shared document credits the team that placed it\", \"actual\": [[\"d2\", \"d1\"], [\"B\", \"A\"], \"tie\"], \"expected\": [[\"d2\", \"d1\"], [\"B\", \"A\"], \"B\"], \"passed\": false}, {\"check\": \"exhausted ranking yields\", \"actual\": [[\"d1\", \"d2\", \"d3\", \"d4\"], [\"A\", \"B\", \"B\", \"B\"], \"B\"], \"expected\": [[\"d1\", \"d2\", \"d3\", \"d4\"], [\"A\", \"B\", \"B\", \"B\"], \"B\"], \"passed\": true}, {\"check\": \"query sample 1\", \"actual\": [[\"d4\", \"d5\", \"d3\", \"d6\", \"d2\"], [\"A\", \"B\", \"A\", \"B\", \"A\"], \"tie\"], \"expected\": [[\"d4\", \"d5\", \"d3\", \"d6\", \"d2\"], [\"A\", \"B\", \"A\", \"B\", \"A\"], \"tie\"], \"passed\": true}, {\"check\": \"query sample 2\", \"actual\": [[\"d6\", \"d5\", \"d1\", \"d2\"], [\"B\", \"A\", \"A\", \"A\"], \"tie\"], \"expected\": [[\"d6\", \"d5\", \"d1\", \"d2\"], [\"B\", \"A\", \"A\", \"A\"], \"tie\"], \"passed\": true}, {\"check\": \"query sample 3\", \"actual\": [[\"d5\", \"d6\", \"d4\", \"d3\", \"d1\"], [\"A\", \"B\", \"A\", \"B\", \"B\"], \"tie\"], \"expected\": [[\"d5\", \"d6\", \"d4\", \"d3\", \"d1\"], [\"A\", \"B\", \"A\", \"B\", \"B\"], \"tie\"], \"passed\": true}], \"passed\": false}\n"},"broken":{"elapsed_ms":38.725,"exit_code":1,"observations":[{"actual":[["d1","d3","d4","d2"],["A","B","B","A"],"tie"],"check":"shared top document is placed once","expected":[["d1","d3","d4","d2"],["A","B","B","A"],"B"],"passed":false},{"actual":[["d4","d1","d2","d5","d6","d3"],["B","A","A","B","B","A"],"B"],"check":"coins are consumed only on ties","expected":[["d4","d1","d2","d5","d6","d3"],["B","A","A","B","B","A"],"B"],"passed":true},{"actual":[["d1","d3","d2","d4"],["A","B","A","B"],"A"],"check":"team behind picks next","expected":[["d1","d3","d2","d4"],["A","B","A","B"],"A"],"passed":true},{"actual":[["d2","d1"],["B","A"],"tie"],"check":"shared document credits the team that placed it","expected":[["d2","d1"],["B","A"],"B"],"passed":false},{"actual":[["d1","d2","d3","d4"],["A","B","B","B"],"B"],"check":"exhausted ranking yields","expected":[["d1","d2","d3","d4"],["A","B","B","B"],"B"],"passed":true},{"actual":[["d4","d5","d3","d6","d2"],["A","B","A","B","A"],"tie"],"check":"query sample 1","expected":[["d4","d5","d3","d6","d2"],["A","B","A","B","A"],"tie"],"passed":true},{"actual":[["d6","d5","d1","d2"],["B","A","A","A"],"tie"],"check":"query sample 2","expected":[["d6","d5","d1","d2"],["B","A","A","A"],"tie"],"passed":true},{"actual":[["d5","d6","d4","d3","d1"],["A","B","A","B","B"],"tie"],"check":"query sample 3","expected":[["d5","d6","d4","d3","d1"],["A","B","A","B","B"],"tie"],"passed":true}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"shared top document is placed once\", \"actual\": [[\"d1\", \"d3\", \"d4\", \"d2\"], [\"A\", \"B\", \"B\", \"A\"], \"tie\"], \"expected\": [[\"d1\", \"d3\", \"d4\", \"d2\"], [\"A\", \"B\", \"B\", \"A\"], \"B\"], \"passed\": false}, {\"check\": \"coins are consumed only on ties\", \"actual\": [[\"d4\", \"d1\", \"d2\", \"d5\", \"d6\", \"d3\"], [\"B\", \"A\", \"A\", \"B\", \"B\", \"A\"], \"B\"], \"expected\": [[\"d4\", \"d1\", \"d2\", \"d5\", \"d6\", \"d3\"], [\"B\", \"A\", \"A\", \"B\", \"B\", \"A\"], \"B\"], \"passed\": true}, {\"check\": \"team behind picks next\", \"actual\": [[\"d1\", \"d3\", \"d2\", \"d4\"], [\"A\", \"B\", \"A\", \"B\"], \"A\"], \"expected\": [[\"d1\", \"d3\", \"d2\", \"d4\"], [\"A\", \"B\", \"A\", \"B\"], \"A\"], \"passed\": true}, {\"check\": \"shared document credits the team that placed it\", \"actual\": [[\"d2\", \"d1\"], [\"B\", \"A\"], \"tie\"], \"expected\": [[\"d2\", \"d1\"], [\"B\", \"A\"], \"B\"], \"passed\": false}, {\"check\": \"exhausted ranking yields\", \"actual\": [[\"d1\", \"d2\", \"d3\", \"d4\"], [\"A\", \"B\", \"B\", \"B\"], \"B\"], \"expected\": [[\"d1\", \"d2\", \"d3\", \"d4\"], [\"A\", \"B\", \"B\", \"B\"], \"B\"], \"passed\": true}, {\"check\": \"query sample 1\", \"actual\": [[\"d4\", \"d5\", \"d3\", \"d6\", \"d2\"], [\"A\", \"B\", \"A\", \"B\", \"A\"], \"tie\"], \"expected\": [[\"d4\", \"d5\", \"d3\", \"d6\", \"d2\"], [\"A\", \"B\", \"A\", \"B\", \"A\"], \"tie\"], \"passed\": true}, {\"check\": \"query sample 2\", \"actual\": [[\"d6\", \"d5\", \"d1\", \"d2\"], [\"B\", \"A\", \"A\", \"A\"], \"tie\"], \"expected\": [[\"d6\", \"d5\", \"d1\", \"d2\"], [\"B\", \"A\", \"A\", \"A\"], \"tie\"], \"passed\": true}, {\"check\": \"query sample 3\", \"actual\": [[\"d5\", \"d6\", \"d4\", \"d3\", \"d1\"], [\"A\", \"B\", \"A\", \"B\", \"B\"], \"tie\"], \"expected\": [[\"d5\", \"d6\", \"d4\", \"d3\", \"d1\"], [\"A\", \"B\", \"A\", \"B\", \"B\"], \"tie\"], \"passed\": true}], \"passed\": false}\n"},"fixed":{"elapsed_ms":40.485,"exit_code":0,"observations":[{"actual":[["d1","d3","d4","d2"],["A","B","B","A"],"B"],"check":"shared top document is placed once","expected":[["d1","d3","d4","d2"],["A","B","B","A"],"B"],"passed":true},{"actual":[["d4","d1","d2","d5","d6","d3"],["B","A","A","B","B","A"],"B"],"check":"coins are consumed only on ties","expected":[["d4","d1","d2","d5","d6","d3"],["B","A","A","B","B","A"],"B"],"passed":true},{"actual":[["d1","d3","d2","d4"],["A","B","A","B"],"A"],"check":"team behind picks next","expected":[["d1","d3","d2","d4"],["A","B","A","B"],"A"],"passed":true},{"actual":[["d2","d1"],["B","A"],"B"],"check":"shared document credits the team that placed it","expected":[["d2","d1"],["B","A"],"B"],"passed":true},{"actual":[["d1","d2","d3","d4"],["A","B","B","B"],"B"],"check":"exhausted ranking yields","expected":[["d1","d2","d3","d4"],["A","B","B","B"],"B"],"passed":true},{"actual":[["d4","d5","d3","d6","d2"],["A","B","A","B","A"],"tie"],"check":"query sample 1","expected":[["d4","d5","d3","d6","d2"],["A","B","A","B","A"],"tie"],"passed":true},{"actual":[["d6","d5","d1","d2"],["B","A","A","A"],"tie"],"check":"query sample 2","expected":[["d6","d5","d1","d2"],["B","A","A","A"],"tie"],"passed":true},{"actual":[["d5","d6","d4","d3","d1"],["A","B","A","B","B"],"tie"],"check":"query sample 3","expected":[["d5","d6","d4","d3","d1"],["A","B","A","B","B"],"tie"],"passed":true}],"passed":true,"stderr":"","stdout":"{\"observations\": [{\"check\": \"shared top document is placed once\", \"actual\": [[\"d1\", \"d3\", \"d4\", \"d2\"], [\"A\", \"B\", \"B\", \"A\"], \"B\"], \"expected\": [[\"d1\", \"d3\", \"d4\", \"d2\"], [\"A\", \"B\", \"B\", \"A\"], \"B\"], \"passed\": true}, {\"check\": \"coins are consumed only on ties\", \"actual\": [[\"d4\", \"d1\", \"d2\", \"d5\", \"d6\", \"d3\"], [\"B\", \"A\", \"A\", \"B\", \"B\", \"A\"], \"B\"], \"expected\": [[\"d4\", \"d1\", \"d2\", \"d5\", \"d6\", \"d3\"], [\"B\", \"A\", \"A\", \"B\", \"B\", \"A\"], \"B\"], \"passed\": true}, {\"check\": \"team behind picks next\", \"actual\": [[\"d1\", \"d3\", \"d2\", \"d4\"], [\"A\", \"B\", \"A\", \"B\"], \"A\"], \"expected\": [[\"d1\", \"d3\", \"d2\", \"d4\"], [\"A\", \"B\", \"A\", \"B\"], \"A\"], \"passed\": true}, {\"check\": \"shared document credits the team that placed it\", \"actual\": [[\"d2\", \"d1\"], [\"B\", \"A\"], \"B\"], \"expected\": [[\"d2\", \"d1\"], [\"B\", \"A\"], \"B\"], \"passed\": true}, {\"check\": \"exhausted ranking yields\", \"actual\": [[\"d1\", \"d2\", \"d3\", \"d4\"], [\"A\", \"B\", \"B\", \"B\"], \"B\"], \"expected\": [[\"d1\", \"d2\", \"d3\", \"d4\"], [\"A\", \"B\", \"B\", \"B\"], \"B\"], \"passed\": true}, {\"check\": \"query sample 1\", \"actual\": [[\"d4\", \"d5\", \"d3\", \"d6\", \"d2\"], [\"A\", \"B\", \"A\", \"B\", \"A\"], \"tie\"], \"expected\": [[\"d4\", \"d5\", \"d3\", \"d6\", \"d2\"], [\"A\", \"B\", \"A\", \"B\", \"A\"], \"tie\"], \"passed\": true}, {\"check\": \"query sample 2\", \"actual\": [[\"d6\", \"d5\", \"d1\", \"d2\"], [\"B\", \"A\", \"A\", \"A\"], \"tie\"], \"expected\": [[\"d6\", \"d5\", \"d1\", \"d2\"], [\"B\", \"A\", \"A\", \"A\"], \"tie\"], \"passed\": true}, {\"check\": \"query sample 3\", \"actual\": [[\"d5\", \"d6\", \"d4\", \"d3\", \"d1\"], [\"A\", \"B\", \"A\", \"B\", \"B\"], \"tie\"], \"expected\": [[\"d5\", \"d6\", \"d4\", \"d3\", \"d1\"], [\"A\", \"B\", \"A\", \"B\", \"B\"], \"tie\"], \"passed\": true}], \"passed\": true}\n"}},"verified":true,"visibility":"public"}