{"abstract":"One ranker dominates the interleaved list whenever it wins the first coin.","category":"Experiment statistics","checks":8,"contract":"Build an interleaved list of up to length documents. Each round both teams skip documents already placed; when both teams have the same number of picks, the next coin (cycled, consumed only on ties) chooses A on 0 and B on 1; otherwise the team with fewer picks goes. An exhausted team yields to the other. Clicks are positions; each credits the team that placed that document. Return [documents, teams, winner A/B/tie].","evaluation_group":"w2-experiment-statistics-team-draft-interleaving","failed_approach":"Strict alternation ignores the coin-decided starting team.","family":"w2-experiment-statistics-team-draft-interleaving-behind-team-picks","id":"FA-74776","implementations":{"attempt":{"sha256":"10455428c8bf1a2ce877342ae369ee37163dc27d0981da30889973f659ccba77","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(rank_a, rank_b, coins, clicks, length):\n    out, team = [], []\n    ia = ib = ci = 0\n    while len(out) < length:\n        while ia < len(rank_a) and rank_a[ia] in out:\n            ia += 1\n        while ib < len(rank_b) and rank_b[ib] in out:\n            ib += 1\n        has_a, has_b = ia < len(rank_a), ib < len(rank_b)\n        if not has_a and not has_b:\n            break\n        na, nb = team.count('A'), team.count('B')\n        if na == nb:\n            pick_a = coins[ci % len(coins)] == 0\n            ci += 1\n        else:\n            pick_a = len(out) % 2 == 0\n        if pick_a and not has_a:\n            pick_a = False\n        elif not pick_a and not has_b:\n            pick_a = True\n        if pick_a:\n            out.append(rank_a[ia])\n            team.append('A')\n        else:\n            out.append(rank_b[ib])\n            team.append('B')\n    credit_a = sum(1 for i in clicks if i < len(out) and team[i] == 'A')\n    credit_b = sum(1 for i in clicks if i < len(out) and team[i] == 'B')\n    winner = 'A' if credit_a > credit_b else 'B' if credit_b > credit_a else 'tie'\n    return [out, team, winner]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('shared top document is placed once',\n   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],\n   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),\n  ('coins are consumed only on ties',\n   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],\n   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),\n  ('team behind picks next',\n   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),\n  ('shared document credits the team that placed it',\n   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],\n   [['d2', 'd1'], ['B', 'A'], 'B']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('query sample 1',\n   [['d4', 'd3', 'd2'], ['d5', 'd6'], [0], [], 6],\n   [['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']),\n  ('query sample 2',\n   [['d6', 'd5', 'd1', 'd2'], ['d6', 'd5'], [1, 1, 1], [], 4],\n   [['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie']),\n  ('query sample 3',\n   [['d5', 'd4'], ['d6', 'd3', 'd5', 'd1'], [0], [], 5],\n   [['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'])],\n [('coins are consumed only on ties',\n   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],\n   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),\n  ('team behind picks next',\n   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),\n  ('shared document credits the team that placed it',\n   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],\n   [['d2', 'd1'], ['B', 'A'], 'B']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),\n  ('query sample 4',\n   [['d5', 'd6'], ['d3', 'd4'], [1, 0], [0, 4], 4],\n   [['d3', 'd5', 'd6', 'd4'], ['B', 'A', 'A', 'B'], 'B']),\n  ('query sample 6',\n   [['d1', 'd4', 'd5'], ['d5', 'd3', 'd2'], [0], [1], 6],\n   [['d1', 'd5', 'd4', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'B']),\n  ('query sample 14',\n   [['d2', 'd6', 'd3', 'd5'], ['d1', 'd4', 'd2', 'd5'], [0, 1], [5], 4],\n   [['d2', 'd1', 'd4', 'd6'], ['A', 'B', 'B', 'A'], 'tie'])],\n [('shared top document is placed once',\n   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],\n   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),\n  ('team behind picks next',\n   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),\n  ('shared document credits the team that placed it',\n   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],\n   [['d2', 'd1'], ['B', 'A'], 'B']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),\n  ('query sample 11',\n   [['d4', 'd2', 'd5'], ['d5', 'd2', 'd1', 'd6'], [1, 0], [], 6],\n   [['d5', 'd4', 'd2', 'd1', 'd6'], ['B', 'A', 'A', 'B', 'B'], 'tie']),\n  ('query sample 12',\n   [['d1', 'd4', 'd5', 'd6', 'd2'], ['d6', 'd2', 'd5'], [0, 1, 0], [], 3],\n   [['d1', 'd6', 'd2'], ['A', 'B', 'B'], 'tie']),\n  ('query sample 26',\n   [['d1', 'd3', 'd5'], ['d1', 'd4', 'd2', 'd5'], [1], [0], 6],\n   [['d1', 'd3', 'd4', 'd5', 'd2'], ['B', 'A', 'B', 'A', 'B'], 'B'])],\n [('shared top document is placed once',\n   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],\n   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),\n  ('coins are consumed only on ties',\n   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],\n   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),\n  ('shared document credits the team that placed it',\n   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],\n   [['d2', 'd1'], ['B', 'A'], 'B']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),\n  ('query sample 16',\n   [['d1', 'd3', 'd4', 'd2'], ['d3', 'd2', 'd1', 'd5'], [0, 1, 1], [], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'B', 'A'], 'tie']),\n  ('query sample 18',\n   [['d2', 'd5'], ['d5', 'd4'], [1], [5], 4],\n   [['d5', 'd2', 'd4'], ['B', 'A', 'B'], 'tie']),\n  ('query sample 35',\n   [['d5', 'd6', 'd1', 'd4'], ['d2', 'd5', 'd4', 'd3', 'd1'], [1, 0, 0], [0, 1], 3],\n   [['d2', 'd5', 'd6'], ['B', 'A', 'A'], 'tie'])],\n [('shared top document is placed once',\n   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],\n   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),\n  ('coins are consumed only on ties',\n   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],\n   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),\n  ('team behind picks next',\n   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),\n  ('query sample 21',\n   [['d5', 'd3'], ['d2', 'd3', 'd6', 'd4', 'd5'], [1, 1], [], 5],\n   [['d2', 'd5', 'd3', 'd6', 'd4'], ['B', 'A', 'B', 'B', 'B'], 'tie']),\n  ('query sample 25',\n   [['d5', 'd3', 'd2'], ['d2', 'd3', 'd4'], [1], [0, 2], 5],\n   [['d2', 'd5', 'd3', 'd4'], ['B', 'A', 'B', 'B'], 'B']),\n  ('query sample 46',\n   [['d4', 'd2', 'd5', 'd6'], ['d2', 'd6'], [1, 1], [], 5],\n   [['d2', 'd4', 'd6', 'd5'], ['B', 'A', 'B', 'A'], 'tie'])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"},"broken":{"sha256":"8ad69a1f1627a9f1dc9168674c99997728bc4f34bec6484a24de2e52ad27c710","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(rank_a, rank_b, coins, clicks, length):\n    out, team = [], []\n    ia = ib = ci = 0\n    while len(out) < length:\n        while ia < len(rank_a) and rank_a[ia] in out:\n            ia += 1\n        while ib < len(rank_b) and rank_b[ib] in out:\n            ib += 1\n        has_a, has_b = ia < len(rank_a), ib < len(rank_b)\n        if not has_a and not has_b:\n            break\n        na, nb = team.count('A'), team.count('B')\n        if na == nb:\n            pick_a = coins[ci % len(coins)] == 0\n            ci += 1\n        else:\n            pick_a = na > nb\n        if pick_a and not has_a:\n            pick_a = False\n        elif not pick_a and not has_b:\n            pick_a = True\n        if pick_a:\n            out.append(rank_a[ia])\n            team.append('A')\n        else:\n            out.append(rank_b[ib])\n            team.append('B')\n    credit_a = sum(1 for i in clicks if i < len(out) and team[i] == 'A')\n    credit_b = sum(1 for i in clicks if i < len(out) and team[i] == 'B')\n    winner = 'A' if credit_a > credit_b else 'B' if credit_b > credit_a else 'tie'\n    return [out, team, winner]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('shared top document is placed once',\n   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],\n   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),\n  ('coins are consumed only on ties',\n   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],\n   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),\n  ('team behind picks next',\n   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),\n  ('shared document credits the team that placed it',\n   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],\n   [['d2', 'd1'], ['B', 'A'], 'B']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('query sample 1',\n   [['d4', 'd3', 'd2'], ['d5', 'd6'], [0], [], 6],\n   [['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']),\n  ('query sample 2',\n   [['d6', 'd5', 'd1', 'd2'], ['d6', 'd5'], [1, 1, 1], [], 4],\n   [['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie']),\n  ('query sample 3',\n   [['d5', 'd4'], ['d6', 'd3', 'd5', 'd1'], [0], [], 5],\n   [['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'])],\n [('coins are consumed only on ties',\n   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],\n   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),\n  ('team behind picks next',\n   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),\n  ('shared document credits the team that placed it',\n   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],\n   [['d2', 'd1'], ['B', 'A'], 'B']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),\n  ('query sample 4',\n   [['d5', 'd6'], ['d3', 'd4'], [1, 0], [0, 4], 4],\n   [['d3', 'd5', 'd6', 'd4'], ['B', 'A', 'A', 'B'], 'B']),\n  ('query sample 6',\n   [['d1', 'd4', 'd5'], ['d5', 'd3', 'd2'], [0], [1], 6],\n   [['d1', 'd5', 'd4', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'B']),\n  ('query sample 14',\n   [['d2', 'd6', 'd3', 'd5'], ['d1', 'd4', 'd2', 'd5'], [0, 1], [5], 4],\n   [['d2', 'd1', 'd4', 'd6'], ['A', 'B', 'B', 'A'], 'tie'])],\n [('shared top document is placed once',\n   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],\n   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),\n  ('team behind picks next',\n   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),\n  ('shared document credits the team that placed it',\n   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],\n   [['d2', 'd1'], ['B', 'A'], 'B']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),\n  ('query sample 11',\n   [['d4', 'd2', 'd5'], ['d5', 'd2', 'd1', 'd6'], [1, 0], [], 6],\n   [['d5', 'd4', 'd2', 'd1', 'd6'], ['B', 'A', 'A', 'B', 'B'], 'tie']),\n  ('query sample 12',\n   [['d1', 'd4', 'd5', 'd6', 'd2'], ['d6', 'd2', 'd5'], [0, 1, 0], [], 3],\n   [['d1', 'd6', 'd2'], ['A', 'B', 'B'], 'tie']),\n  ('query sample 26',\n   [['d1', 'd3', 'd5'], ['d1', 'd4', 'd2', 'd5'], [1], [0], 6],\n   [['d1', 'd3', 'd4', 'd5', 'd2'], ['B', 'A', 'B', 'A', 'B'], 'B'])],\n [('shared top document is placed once',\n   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],\n   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),\n  ('coins are consumed only on ties',\n   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],\n   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),\n  ('shared document credits the team that placed it',\n   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],\n   [['d2', 'd1'], ['B', 'A'], 'B']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),\n  ('query sample 16',\n   [['d1', 'd3', 'd4', 'd2'], ['d3', 'd2', 'd1', 'd5'], [0, 1, 1], [], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'B', 'A'], 'tie']),\n  ('query sample 18',\n   [['d2', 'd5'], ['d5', 'd4'], [1], [5], 4],\n   [['d5', 'd2', 'd4'], ['B', 'A', 'B'], 'tie']),\n  ('query sample 35',\n   [['d5', 'd6', 'd1', 'd4'], ['d2', 'd5', 'd4', 'd3', 'd1'], [1, 0, 0], [0, 1], 3],\n   [['d2', 'd5', 'd6'], ['B', 'A', 'A'], 'tie'])],\n [('shared top document is placed once',\n   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],\n   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),\n  ('coins are consumed only on ties',\n   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],\n   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),\n  ('team behind picks next',\n   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),\n  ('query sample 21',\n   [['d5', 'd3'], ['d2', 'd3', 'd6', 'd4', 'd5'], [1, 1], [], 5],\n   [['d2', 'd5', 'd3', 'd6', 'd4'], ['B', 'A', 'B', 'B', 'B'], 'tie']),\n  ('query sample 25',\n   [['d5', 'd3', 'd2'], ['d2', 'd3', 'd4'], [1], [0, 2], 5],\n   [['d2', 'd5', 'd3', 'd4'], ['B', 'A', 'B', 'B'], 'B']),\n  ('query sample 46',\n   [['d4', 'd2', 'd5', 'd6'], ['d2', 'd6'], [1, 1], [], 5],\n   [['d2', 'd4', 'd6', 'd5'], ['B', 'A', 'B', 'A'], 'tie'])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"},"fixed":{"sha256":"665223cb84c1577d74d753d7c6dacd99a0a2c9ddc4d6b7bf714ec43c46f31368","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(rank_a, rank_b, coins, clicks, length):\n    out, team = [], []\n    ia = ib = ci = 0\n    while len(out) < length:\n        while ia < len(rank_a) and rank_a[ia] in out:\n            ia += 1\n        while ib < len(rank_b) and rank_b[ib] in out:\n            ib += 1\n        has_a, has_b = ia < len(rank_a), ib < len(rank_b)\n        if not has_a and not has_b:\n            break\n        na, nb = team.count('A'), team.count('B')\n        if na == nb:\n            pick_a = coins[ci % len(coins)] == 0\n            ci += 1\n        else:\n            pick_a = na < nb\n        if pick_a and not has_a:\n            pick_a = False\n        elif not pick_a and not has_b:\n            pick_a = True\n        if pick_a:\n            out.append(rank_a[ia])\n            team.append('A')\n        else:\n            out.append(rank_b[ib])\n            team.append('B')\n    credit_a = sum(1 for i in clicks if i < len(out) and team[i] == 'A')\n    credit_b = sum(1 for i in clicks if i < len(out) and team[i] == 'B')\n    winner = 'A' if credit_a > credit_b else 'B' if credit_b > credit_a else 'tie'\n    return [out, team, winner]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('shared top document is placed once',\n   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],\n   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),\n  ('coins are consumed only on ties',\n   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],\n   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),\n  ('team behind picks next',\n   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),\n  ('shared document credits the team that placed it',\n   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],\n   [['d2', 'd1'], ['B', 'A'], 'B']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('query sample 1',\n   [['d4', 'd3', 'd2'], ['d5', 'd6'], [0], [], 6],\n   [['d4', 'd5', 'd3', 'd6', 'd2'], ['A', 'B', 'A', 'B', 'A'], 'tie']),\n  ('query sample 2',\n   [['d6', 'd5', 'd1', 'd2'], ['d6', 'd5'], [1, 1, 1], [], 4],\n   [['d6', 'd5', 'd1', 'd2'], ['B', 'A', 'A', 'A'], 'tie']),\n  ('query sample 3',\n   [['d5', 'd4'], ['d6', 'd3', 'd5', 'd1'], [0], [], 5],\n   [['d5', 'd6', 'd4', 'd3', 'd1'], ['A', 'B', 'A', 'B', 'B'], 'tie'])],\n [('coins are consumed only on ties',\n   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],\n   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),\n  ('team behind picks next',\n   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),\n  ('shared document credits the team that placed it',\n   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],\n   [['d2', 'd1'], ['B', 'A'], 'B']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),\n  ('query sample 4',\n   [['d5', 'd6'], ['d3', 'd4'], [1, 0], [0, 4], 4],\n   [['d3', 'd5', 'd6', 'd4'], ['B', 'A', 'A', 'B'], 'B']),\n  ('query sample 6',\n   [['d1', 'd4', 'd5'], ['d5', 'd3', 'd2'], [0], [1], 6],\n   [['d1', 'd5', 'd4', 'd3', 'd2'], ['A', 'B', 'A', 'B', 'B'], 'B']),\n  ('query sample 14',\n   [['d2', 'd6', 'd3', 'd5'], ['d1', 'd4', 'd2', 'd5'], [0, 1], [5], 4],\n   [['d2', 'd1', 'd4', 'd6'], ['A', 'B', 'B', 'A'], 'tie'])],\n [('shared top document is placed once',\n   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],\n   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),\n  ('team behind picks next',\n   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),\n  ('shared document credits the team that placed it',\n   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],\n   [['d2', 'd1'], ['B', 'A'], 'B']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),\n  ('query sample 11',\n   [['d4', 'd2', 'd5'], ['d5', 'd2', 'd1', 'd6'], [1, 0], [], 6],\n   [['d5', 'd4', 'd2', 'd1', 'd6'], ['B', 'A', 'A', 'B', 'B'], 'tie']),\n  ('query sample 12',\n   [['d1', 'd4', 'd5', 'd6', 'd2'], ['d6', 'd2', 'd5'], [0, 1, 0], [], 3],\n   [['d1', 'd6', 'd2'], ['A', 'B', 'B'], 'tie']),\n  ('query sample 26',\n   [['d1', 'd3', 'd5'], ['d1', 'd4', 'd2', 'd5'], [1], [0], 6],\n   [['d1', 'd3', 'd4', 'd5', 'd2'], ['B', 'A', 'B', 'A', 'B'], 'B'])],\n [('shared top document is placed once',\n   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],\n   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),\n  ('coins are consumed only on ties',\n   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],\n   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),\n  ('shared document credits the team that placed it',\n   [['d1', 'd2'], ['d2', 'd1'], [1], [0], 2],\n   [['d2', 'd1'], ['B', 'A'], 'B']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),\n  ('query sample 16',\n   [['d1', 'd3', 'd4', 'd2'], ['d3', 'd2', 'd1', 'd5'], [0, 1, 1], [], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'B', 'A'], 'tie']),\n  ('query sample 18',\n   [['d2', 'd5'], ['d5', 'd4'], [1], [5], 4],\n   [['d5', 'd2', 'd4'], ['B', 'A', 'B'], 'tie']),\n  ('query sample 35',\n   [['d5', 'd6', 'd1', 'd4'], ['d2', 'd5', 'd4', 'd3', 'd1'], [1, 0, 0], [0, 1], 3],\n   [['d2', 'd5', 'd6'], ['B', 'A', 'A'], 'tie'])],\n [('shared top document is placed once',\n   [['d1', 'd2', 'd3'], ['d1', 'd3', 'd4'], [0, 1], [1], 4],\n   [['d1', 'd3', 'd4', 'd2'], ['A', 'B', 'B', 'A'], 'B']),\n  ('coins are consumed only on ties',\n   [['d1', 'd2', 'd3'], ['d4', 'd5', 'd6'], [1, 0], [0, 3], 6],\n   [['d4', 'd1', 'd2', 'd5', 'd6', 'd3'], ['B', 'A', 'A', 'B', 'B', 'A'], 'B']),\n  ('team behind picks next',\n   [['d1', 'd2'], ['d3', 'd4'], [0], [2], 4],\n   [['d1', 'd3', 'd2', 'd4'], ['A', 'B', 'A', 'B'], 'A']),\n  ('exhausted ranking yields',\n   [['d1'], ['d2', 'd3', 'd4'], [0], [3], 4],\n   [['d1', 'd2', 'd3', 'd4'], ['A', 'B', 'B', 'B'], 'B']),\n  ('no clicks is a tie', [['d1', 'd2'], ['d3'], [1], [], 3], [['d3', 'd1', 'd2'], ['B', 'A', 'A'], 'tie']),\n  ('query sample 21',\n   [['d5', 'd3'], ['d2', 'd3', 'd6', 'd4', 'd5'], [1, 1], [], 5],\n   [['d2', 'd5', 'd3', 'd6', 'd4'], ['B', 'A', 'B', 'B', 'B'], 'tie']),\n  ('query sample 25',\n   [['d5', 'd3', 'd2'], ['d2', 'd3', 'd4'], [1], [0, 2], 5],\n   [['d2', 'd5', 'd3', 'd4'], ['B', 'A', 'B', 'B'], 'B']),\n  ('query sample 46',\n   [['d4', 'd2', 'd5', 'd6'], ['d2', 'd6'], [1, 1], [], 5],\n   [['d2', 'd4', 'd6', 'd5'], ['B', 'A', 'B', 'A'], 'tie'])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"}},"limitations":"A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.","method":"Deterministic executable model with adversarial boundary fixtures.","provenance":{"created_by":"Failure Map","dependencies":"Python standard library","family":"w2-experiment-statistics-team-draft-interleaving-behind-team-picks","generated_at":"2026-09-29T14:48:59.809650+00:00","license":"CC0-1.0","python":"3.12.14","seed":1,"split":"open-access"},"relevance":"Interleaving compares ranking algorithms with far fewer users than an A/B test, if credit is fair.","repair":"Let the team with fewer picks go next.","root_cause":"Without a tie the team with more picks goes next.","sha256":"eb2a228213aea239a1a0988ae28f1580ce8d5cefdc16434b32d34bedcd6b0d11","title":"Team-draft interleaving credit: The team ahead picks again · case 01","variant":1,"variant_policy":"Five numbered records share a model and may reuse boundary fixtures.","verification":{"attempt":{"elapsed_ms":38.939,"exit_code":1,"observations":[{"actual":[["d1","d3","d4","d2"],["A","B","B","A"],"B"],"check":"shared top document is placed once","expected":[["d1","d3","d4","d2"],["A","B","B","A"],"B"],"passed":true},{"actual":[["d4","d5","d1","d6","d2","d3"],["B","B","A","B","A","A"],"B"],"check":"coins are consumed only on ties","expected":[["d4","d1","d2","d5","d6","d3"],["B","A","A","B","B","A"],"B"],"passed":false},{"actual":[["d1","d3","d2","d4"],["A","B","A","B"],"A"],"check":"team behind picks next","expected":[["d1","d3","d2","d4"],["A","B","A","B"],"A"],"passed":true},{"actual":[["d2","d1"],["B","B"],"B"],"check":"shared document credits the team that placed it","expected":[["d2","d1"],["B","A"],"B"],"passed":false},{"actual":[["d1","d2","d3","d4"],["A","B","B","B"],"B"],"check":"exhausted ranking yields","expected":[["d1","d2","d3","d4"],["A","B","B","B"],"B"],"passed":true},{"actual":[["d4","d5","d3","d6","d2"],["A","B","A","B","A"],"tie"],"check":"query sample 1","expected":[["d4","d5","d3","d6","d2"],["A","B","A","B","A"],"tie"],"passed":true},{"actual":[["d6","d5","d1","d2"],["B","B","A","A"],"tie"],"check":"query sample 2","expected":[["d6","d5","d1","d2"],["B","A","A","A"],"tie"],"passed":false},{"actual":[["d5","d6","d4","d3","d1"],["A","B","A","B","B"],"tie"],"check":"query sample 3","expected":[["d5","d6","d4","d3","d1"],["A","B","A","B","B"],"tie"],"passed":true}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"shared top document is placed once\", \"actual\": [[\"d1\", \"d3\", \"d4\", \"d2\"], [\"A\", \"B\", \"B\", \"A\"], \"B\"], \"expected\": [[\"d1\", \"d3\", \"d4\", \"d2\"], [\"A\", \"B\", \"B\", \"A\"], \"B\"], \"passed\": true}, {\"check\": \"coins are consumed only on ties\", \"actual\": [[\"d4\", \"d5\", \"d1\", \"d6\", \"d2\", \"d3\"], [\"B\", \"B\", \"A\", \"B\", \"A\", \"A\"], \"B\"], \"expected\": [[\"d4\", \"d1\", \"d2\", \"d5\", \"d6\", \"d3\"], [\"B\", \"A\", \"A\", \"B\", \"B\", \"A\"], \"B\"], \"passed\": false}, {\"check\": \"team behind picks next\", \"actual\": [[\"d1\", \"d3\", \"d2\", \"d4\"], [\"A\", \"B\", \"A\", \"B\"], \"A\"], \"expected\": [[\"d1\", \"d3\", \"d2\", \"d4\"], [\"A\", \"B\", \"A\", \"B\"], \"A\"], \"passed\": true}, {\"check\": \"shared document credits the team that placed it\", \"actual\": [[\"d2\", \"d1\"], [\"B\", \"B\"], \"B\"], \"expected\": [[\"d2\", \"d1\"], [\"B\", \"A\"], \"B\"], \"passed\": false}, {\"check\": \"exhausted ranking yields\", \"actual\": [[\"d1\", \"d2\", \"d3\", \"d4\"], [\"A\", \"B\", \"B\", \"B\"], \"B\"], \"expected\": [[\"d1\", \"d2\", \"d3\", \"d4\"], [\"A\", \"B\", \"B\", \"B\"], \"B\"], \"passed\": true}, {\"check\": \"query sample 1\", \"actual\": [[\"d4\", \"d5\", \"d3\", \"d6\", \"d2\"], [\"A\", \"B\", \"A\", \"B\", \"A\"], \"tie\"], \"expected\": [[\"d4\", \"d5\", \"d3\", \"d6\", \"d2\"], [\"A\", \"B\", \"A\", \"B\", \"A\"], \"tie\"], \"passed\": true}, {\"check\": \"query sample 2\", \"actual\": [[\"d6\", \"d5\", \"d1\", \"d2\"], [\"B\", \"B\", \"A\", \"A\"], \"tie\"], \"expected\": [[\"d6\", \"d5\", \"d1\", \"d2\"], [\"B\", \"A\", \"A\", \"A\"], \"tie\"], \"passed\": false}, {\"check\": \"query sample 3\", \"actual\": [[\"d5\", \"d6\", \"d4\", \"d3\", \"d1\"], [\"A\", \"B\", \"A\", \"B\", \"B\"], \"tie\"], \"expected\": [[\"d5\", \"d6\", \"d4\", \"d3\", \"d1\"], [\"A\", \"B\", \"A\", \"B\", \"B\"], \"tie\"], \"passed\": true}], \"passed\": false}\n"},"broken":{"elapsed_ms":38.288,"exit_code":1,"observations":[{"actual":[["d1","d2","d3","d4"],["A","A","A","B"],"A"],"check":"shared top document is placed once","expected":[["d1","d3","d4","d2"],["A","B","B","A"],"B"],"passed":false},{"actual":[["d4","d5","d6","d1","d2","d3"],["B","B","B","A","A","A"],"tie"],"check":"coins are consumed only on ties","expected":[["d4","d1","d2","d5","d6","d3"],["B","A","A","B","B","A"],"B"],"passed":false},{"actual":[["d1","d2","d3","d4"],["A","A","B","B"],"B"],"check":"team behind picks next","expected":[["d1","d3","d2","d4"],["A","B","A","B"],"A"],"passed":false},{"actual":[["d2","d1"],["B","B"],"B"],"check":"shared document credits the team that placed it","expected":[["d2","d1"],["B","A"],"B"],"passed":false},{"actual":[["d1","d2","d3","d4"],["A","B","B","B"],"B"],"check":"exhausted ranking yields","expected":[["d1","d2","d3","d4"],["A","B","B","B"],"B"],"passed":true},{"actual":[["d4","d3","d2","d5","d6"],["A","A","A","B","B"],"tie"],"check":"query sample 1","expected":[["d4","d5","d3","d6","d2"],["A","B","A","B","A"],"tie"],"passed":false},{"actual":[["d6","d5","d1","d2"],["B","B","A","A"],"tie"],"check":"query sample 2","expected":[["d6","d5","d1","d2"],["B","A","A","A"],"tie"],"passed":false},{"actual":[["d5","d4","d6","d3","d1"],["A","A","B","B","B"],"tie"],"check":"query sample 3","expected":[["d5","d6","d4","d3","d1"],["A","B","A","B","B"],"tie"],"passed":false}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"shared top document is placed once\", \"actual\": [[\"d1\", \"d2\", \"d3\", \"d4\"], [\"A\", \"A\", \"A\", \"B\"], \"A\"], \"expected\": [[\"d1\", \"d3\", \"d4\", \"d2\"], [\"A\", \"B\", \"B\", \"A\"], \"B\"], \"passed\": false}, {\"check\": \"coins are consumed only on ties\", \"actual\": [[\"d4\", \"d5\", \"d6\", \"d1\", \"d2\", \"d3\"], [\"B\", \"B\", \"B\", \"A\", \"A\", \"A\"], \"tie\"], \"expected\": [[\"d4\", \"d1\", \"d2\", \"d5\", \"d6\", \"d3\"], [\"B\", \"A\", \"A\", \"B\", \"B\", \"A\"], \"B\"], \"passed\": false}, {\"check\": \"team behind picks next\", \"actual\": [[\"d1\", \"d2\", \"d3\", \"d4\"], [\"A\", \"A\", \"B\", \"B\"], \"B\"], \"expected\": [[\"d1\", \"d3\", \"d2\", \"d4\"], [\"A\", \"B\", \"A\", \"B\"], \"A\"], \"passed\": false}, {\"check\": \"shared document credits the team that placed it\", \"actual\": [[\"d2\", \"d1\"], [\"B\", \"B\"], \"B\"], \"expected\": [[\"d2\", \"d1\"], [\"B\", \"A\"], \"B\"], \"passed\": false}, {\"check\": \"exhausted ranking yields\", \"actual\": [[\"d1\", \"d2\", \"d3\", \"d4\"], [\"A\", \"B\", \"B\", \"B\"], \"B\"], \"expected\": [[\"d1\", \"d2\", \"d3\", \"d4\"], [\"A\", \"B\", \"B\", \"B\"], \"B\"], \"passed\": true}, {\"check\": \"query sample 1\", \"actual\": [[\"d4\", \"d3\", \"d2\", \"d5\", \"d6\"], [\"A\", \"A\", \"A\", \"B\", \"B\"], \"tie\"], \"expected\": [[\"d4\", \"d5\", \"d3\", \"d6\", \"d2\"], [\"A\", \"B\", \"A\", \"B\", \"A\"], \"tie\"], \"passed\": false}, {\"check\": \"query sample 2\", \"actual\": [[\"d6\", \"d5\", \"d1\", \"d2\"], [\"B\", \"B\", \"A\", \"A\"], \"tie\"], \"expected\": [[\"d6\", \"d5\", \"d1\", \"d2\"], [\"B\", \"A\", \"A\", \"A\"], \"tie\"], \"passed\": false}, {\"check\": \"query sample 3\", \"actual\": [[\"d5\", \"d4\", \"d6\", \"d3\", \"d1\"], [\"A\", \"A\", \"B\", \"B\", \"B\"], \"tie\"], \"expected\": [[\"d5\", \"d6\", \"d4\", \"d3\", \"d1\"], [\"A\", \"B\", \"A\", \"B\", \"B\"], \"tie\"], \"passed\": false}], \"passed\": false}\n"},"fixed":{"elapsed_ms":41.214,"exit_code":0,"observations":[{"actual":[["d1","d3","d4","d2"],["A","B","B","A"],"B"],"check":"shared top document is placed once","expected":[["d1","d3","d4","d2"],["A","B","B","A"],"B"],"passed":true},{"actual":[["d4","d1","d2","d5","d6","d3"],["B","A","A","B","B","A"],"B"],"check":"coins are consumed only on ties","expected":[["d4","d1","d2","d5","d6","d3"],["B","A","A","B","B","A"],"B"],"passed":true},{"actual":[["d1","d3","d2","d4"],["A","B","A","B"],"A"],"check":"team behind picks next","expected":[["d1","d3","d2","d4"],["A","B","A","B"],"A"],"passed":true},{"actual":[["d2","d1"],["B","A"],"B"],"check":"shared document credits the team that placed it","expected":[["d2","d1"],["B","A"],"B"],"passed":true},{"actual":[["d1","d2","d3","d4"],["A","B","B","B"],"B"],"check":"exhausted ranking yields","expected":[["d1","d2","d3","d4"],["A","B","B","B"],"B"],"passed":true},{"actual":[["d4","d5","d3","d6","d2"],["A","B","A","B","A"],"tie"],"check":"query sample 1","expected":[["d4","d5","d3","d6","d2"],["A","B","A","B","A"],"tie"],"passed":true},{"actual":[["d6","d5","d1","d2"],["B","A","A","A"],"tie"],"check":"query sample 2","expected":[["d6","d5","d1","d2"],["B","A","A","A"],"tie"],"passed":true},{"actual":[["d5","d6","d4","d3","d1"],["A","B","A","B","B"],"tie"],"check":"query sample 3","expected":[["d5","d6","d4","d3","d1"],["A","B","A","B","B"],"tie"],"passed":true}],"passed":true,"stderr":"","stdout":"{\"observations\": [{\"check\": \"shared top document is placed once\", \"actual\": [[\"d1\", \"d3\", \"d4\", \"d2\"], [\"A\", \"B\", \"B\", \"A\"], \"B\"], \"expected\": [[\"d1\", \"d3\", \"d4\", \"d2\"], [\"A\", \"B\", \"B\", \"A\"], \"B\"], \"passed\": true}, {\"check\": \"coins are consumed only on ties\", \"actual\": [[\"d4\", \"d1\", \"d2\", \"d5\", \"d6\", \"d3\"], [\"B\", \"A\", \"A\", \"B\", \"B\", \"A\"], \"B\"], \"expected\": [[\"d4\", \"d1\", \"d2\", \"d5\", \"d6\", \"d3\"], [\"B\", \"A\", \"A\", \"B\", \"B\", \"A\"], \"B\"], \"passed\": true}, {\"check\": \"team behind picks next\", \"actual\": [[\"d1\", \"d3\", \"d2\", \"d4\"], [\"A\", \"B\", \"A\", \"B\"], \"A\"], \"expected\": [[\"d1\", \"d3\", \"d2\", \"d4\"], [\"A\", \"B\", \"A\", \"B\"], \"A\"], \"passed\": true}, {\"check\": \"shared document credits the team that placed it\", \"actual\": [[\"d2\", \"d1\"], [\"B\", \"A\"], \"B\"], \"expected\": [[\"d2\", \"d1\"], [\"B\", \"A\"], \"B\"], \"passed\": true}, {\"check\": \"exhausted ranking yields\", \"actual\": [[\"d1\", \"d2\", \"d3\", \"d4\"], [\"A\", \"B\", \"B\", \"B\"], \"B\"], \"expected\": [[\"d1\", \"d2\", \"d3\", \"d4\"], [\"A\", \"B\", \"B\", \"B\"], \"B\"], \"passed\": true}, {\"check\": \"query sample 1\", \"actual\": [[\"d4\", \"d5\", \"d3\", \"d6\", \"d2\"], [\"A\", \"B\", \"A\", \"B\", \"A\"], \"tie\"], \"expected\": [[\"d4\", \"d5\", \"d3\", \"d6\", \"d2\"], [\"A\", \"B\", \"A\", \"B\", \"A\"], \"tie\"], \"passed\": true}, {\"check\": \"query sample 2\", \"actual\": [[\"d6\", \"d5\", \"d1\", \"d2\"], [\"B\", \"A\", \"A\", \"A\"], \"tie\"], \"expected\": [[\"d6\", \"d5\", \"d1\", \"d2\"], [\"B\", \"A\", \"A\", \"A\"], \"tie\"], \"passed\": true}, {\"check\": \"query sample 3\", \"actual\": [[\"d5\", \"d6\", \"d4\", \"d3\", \"d1\"], [\"A\", \"B\", \"A\", \"B\", \"B\"], \"tie\"], \"expected\": [[\"d5\", \"d6\", \"d4\", \"d3\", \"d1\"], [\"A\", \"B\", \"A\", \"B\", \"B\"], \"tie\"], \"passed\": true}], \"passed\": true}\n"}},"verified":true,"visibility":"public"}