{"abstract":"Intervals are far too narrow because repeated sessions of one user are assumed independent.","category":"Experiment statistics","checks":8,"contract":"Each user contributes [clicks, sessions]; the unit of randomisation is the user. The arm ratio is sum(clicks) / sum(sessions). Its delta-method variance is (var_x - 2 R cov_xy + R^2 var_y) / (n mean_y^2) with sample (n - 1) moments over users. Arms with fewer than two users or no sessions -> None. Return [R_control, R_treatment, standard error of the difference] rounded to 6.","evaluation_group":"w2-experiment-statistics-ratio-metric","failed_approach":"Keeping the delta-method numerator but dividing by session count still counts sessions as units.","family":"w2-experiment-statistics-ratio-metric-unit-of-analysis","id":"FA-74531","implementations":{"attempt":{"sha256":"8d4b20283fea3529641a72d4e3fafe61cc8764f8f086b8771a5893d1f93b8034","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\nimport math\nN = 1\nobservations = []\ndef solve(users_c, users_t):\n    def stats(users):\n        n = len(users)\n        if n < 2:\n            return None\n        xs = [u[0] for u in users]\n        ys = [u[1] for u in users]\n        mx, my = sum(xs) / n, sum(ys) / n\n        if my == 0:\n            return None\n        r = mx / my\n        vx = sum((x - mx) ** 2 for x in xs) / (n - 1)\n        vy = sum((y - my) ** 2 for y in ys) / (n - 1)\n        cxy = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / (n - 1)\n        var = (vx - 2 * r * cxy + r * r * vy) / (sum(ys) * my * my)\n        return r, var\n    a, b = stats(users_c), stats(users_t)\n    if a is None or b is None:\n        return None\n    return [round(a[0], 6), round(b[0], 6), round(math.sqrt(max(a[1] + b[1], 0.0)), 6)]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('heavy user dominates the ratio of sums',\n   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],\n   [0.52381, 0.318182, 0.132002]),\n  ('correlated clicks and sessions',\n   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],\n   [0.538462, 0.7, 0.172146]),\n  ('users with zero sessions count',\n   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],\n   [0.625, 0.4, 0.161613]),\n  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),\n  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),\n  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),\n  ('user aggregate sample 1', [[[1, 2], [9, 10]], [[0, 0], [0, 2], [7, 10]]], [0.833333, 0.583333, 0.201748]),\n  ('user aggregate sample 2',\n   [[[0, 0], [1, 5], [1, 2], [0, 0]], [[0, 0], [1, 1], [7, 10], [10, 10]]],\n   [0.285714, 0.857143, 0.153928])],\n [('correlated clicks and sessions',\n   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],\n   [0.538462, 0.7, 0.172146]),\n  ('users with zero sessions count',\n   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],\n   [0.625, 0.4, 0.161613]),\n  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),\n  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),\n  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),\n  ('user aggregate sample 5',\n   [[[6, 10], [0, 1], [0, 0], [0, 2], [8, 10]], [[1, 2], [1, 5], [0, 0], [0, 5], [0, 0]]],\n   [0.608696, 0.166667, 0.152218]),\n  ('user aggregate sample 6',\n   [[[1, 1], [5, 5], [1, 1], [3, 5], [10, 10]], [[0, 1], [1, 5], [0, 0], [0, 10], [1, 5], [1, 1]]],\n   [0.909091, 0.136364, 0.126043]),\n  ('user aggregate sample 7',\n   [[[0, 1], [5, 10], [1, 5]], [[1, 2], [1, 2], [0, 1], [3, 5], [5, 5]]],\n   [0.375, 0.666667, 0.184982])],\n [('heavy user dominates the ratio of sums',\n   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],\n   [0.52381, 0.318182, 0.132002]),\n  ('users with zero sessions count',\n   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],\n   [0.625, 0.4, 0.161613]),\n  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),\n  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),\n  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),\n  ('user aggregate sample 11',\n   [[[7, 10], [0, 0]], [[1, 5], [0, 1], [0, 5], [1, 1], [1, 1], [2, 10]]],\n   [0.7, 0.217391, 0.075178]),\n  ('user aggregate sample 12',\n   [[[0, 5], [5, 10], [0, 2], [0, 0]], [[5, 5], [0, 1], [10, 10], [7, 10], [1, 1], [0, 1]]],\n   [0.294118, 0.821429, 0.203966]),\n  ('user aggregate sample 13',\n   [[[0, 0], [1, 5], [0, 2], [0, 2], [1, 2], [7, 10]], [[2, 10], [2, 10], [1, 1], [0, 0], [1, 1]]],\n   [0.428571, 0.272727, 0.181987])],\n [('heavy user dominates the ratio of sums',\n   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],\n   [0.52381, 0.318182, 0.132002]),\n  ('correlated clicks and sessions',\n   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],\n   [0.538462, 0.7, 0.172146]),\n  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),\n  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),\n  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),\n  ('user aggregate sample 16',\n   [[[0, 0], [1, 1], [5, 5], [1, 1], [0, 1], [0, 10]], [[0, 0], [0, 2], [0, 0], [0, 10]]],\n   [0.388889, 0.0, 0.306461]),\n  ('user aggregate sample 17',\n   [[[0, 10], [0, 2]], [[3, 10], [6, 10], [0, 0], [5, 10]]],\n   [0.0, 0.466667, 0.083148]),\n  ('user aggregate sample 21',\n   [[[4, 5], [2, 2], [2, 2]], [[2, 5], [2, 2], [5, 5], [7, 10]]],\n   [0.888889, 0.727273, 0.137879])],\n [('heavy user dominates the ratio of sums',\n   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],\n   [0.52381, 0.318182, 0.132002]),\n  ('correlated clicks and sessions',\n   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],\n   [0.538462, 0.7, 0.172146]),\n  ('users with zero sessions count',\n   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],\n   [0.625, 0.4, 0.161613]),\n  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),\n  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),\n  ('user aggregate sample 21',\n   [[[4, 5], [2, 2], [2, 2]], [[2, 5], [2, 2], [5, 5], [7, 10]]],\n   [0.888889, 0.727273, 0.137879]),\n  ('user aggregate sample 22',\n   [[[0, 0], [2, 2], [8, 10], [0, 1], [2, 10]], [[0, 0], [1, 1], [0, 2], [0, 0]]],\n   [0.521739, 0.333333, 0.421009]),\n  ('user aggregate sample 28',\n   [[[0, 2], [0, 1], [1, 2], [3, 5], [2, 5], [0, 1]], [[4, 5], [1, 1], [1, 1]]],\n   [0.375, 0.857143, 0.118332])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"},"broken":{"sha256":"52d614cb7bcd12ab1aa7c0e34c52555126bfc0739b5043d33c4a3d0d09f19d15","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\nimport math\nN = 1\nobservations = []\ndef solve(users_c, users_t):\n    def stats(users):\n        n = len(users)\n        if n < 2:\n            return None\n        xs = [u[0] for u in users]\n        ys = [u[1] for u in users]\n        mx, my = sum(xs) / n, sum(ys) / n\n        if my == 0:\n            return None\n        r = mx / my\n        vx = sum((x - mx) ** 2 for x in xs) / (n - 1)\n        vy = sum((y - my) ** 2 for y in ys) / (n - 1)\n        cxy = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / (n - 1)\n        var = r * (1 - r) / sum(ys)\n        return r, var\n    a, b = stats(users_c), stats(users_t)\n    if a is None or b is None:\n        return None\n    return [round(a[0], 6), round(b[0], 6), round(math.sqrt(max(a[1] + b[1], 0.0)), 6)]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('heavy user dominates the ratio of sums',\n   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],\n   [0.52381, 0.318182, 0.132002]),\n  ('correlated clicks and sessions',\n   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],\n   [0.538462, 0.7, 0.172146]),\n  ('users with zero sessions count',\n   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],\n   [0.625, 0.4, 0.161613]),\n  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),\n  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),\n  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),\n  ('user aggregate sample 1', [[[1, 2], [9, 10]], [[0, 0], [0, 2], [7, 10]]], [0.833333, 0.583333, 0.201748]),\n  ('user aggregate sample 2',\n   [[[0, 0], [1, 5], [1, 2], [0, 0]], [[0, 0], [1, 1], [7, 10], [10, 10]]],\n   [0.285714, 0.857143, 0.153928])],\n [('correlated clicks and sessions',\n   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],\n   [0.538462, 0.7, 0.172146]),\n  ('users with zero sessions count',\n   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],\n   [0.625, 0.4, 0.161613]),\n  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),\n  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),\n  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),\n  ('user aggregate sample 5',\n   [[[6, 10], [0, 1], [0, 0], [0, 2], [8, 10]], [[1, 2], [1, 5], [0, 0], [0, 5], [0, 0]]],\n   [0.608696, 0.166667, 0.152218]),\n  ('user aggregate sample 6',\n   [[[1, 1], [5, 5], [1, 1], [3, 5], [10, 10]], [[0, 1], [1, 5], [0, 0], [0, 10], [1, 5], [1, 1]]],\n   [0.909091, 0.136364, 0.126043]),\n  ('user aggregate sample 7',\n   [[[0, 1], [5, 10], [1, 5]], [[1, 2], [1, 2], [0, 1], [3, 5], [5, 5]]],\n   [0.375, 0.666667, 0.184982])],\n [('heavy user dominates the ratio of sums',\n   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],\n   [0.52381, 0.318182, 0.132002]),\n  ('users with zero sessions count',\n   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],\n   [0.625, 0.4, 0.161613]),\n  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),\n  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),\n  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),\n  ('user aggregate sample 11',\n   [[[7, 10], [0, 0]], [[1, 5], [0, 1], [0, 5], [1, 1], [1, 1], [2, 10]]],\n   [0.7, 0.217391, 0.075178]),\n  ('user aggregate sample 12',\n   [[[0, 5], [5, 10], [0, 2], [0, 0]], [[5, 5], [0, 1], [10, 10], [7, 10], [1, 1], [0, 1]]],\n   [0.294118, 0.821429, 0.203966]),\n  ('user aggregate sample 13',\n   [[[0, 0], [1, 5], [0, 2], [0, 2], [1, 2], [7, 10]], [[2, 10], [2, 10], [1, 1], [0, 0], [1, 1]]],\n   [0.428571, 0.272727, 0.181987])],\n [('heavy user dominates the ratio of sums',\n   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],\n   [0.52381, 0.318182, 0.132002]),\n  ('correlated clicks and sessions',\n   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],\n   [0.538462, 0.7, 0.172146]),\n  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),\n  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),\n  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),\n  ('user aggregate sample 16',\n   [[[0, 0], [1, 1], [5, 5], [1, 1], [0, 1], [0, 10]], [[0, 0], [0, 2], [0, 0], [0, 10]]],\n   [0.388889, 0.0, 0.306461]),\n  ('user aggregate sample 17',\n   [[[0, 10], [0, 2]], [[3, 10], [6, 10], [0, 0], [5, 10]]],\n   [0.0, 0.466667, 0.083148]),\n  ('user aggregate sample 21',\n   [[[4, 5], [2, 2], [2, 2]], [[2, 5], [2, 2], [5, 5], [7, 10]]],\n   [0.888889, 0.727273, 0.137879])],\n [('heavy user dominates the ratio of sums',\n   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],\n   [0.52381, 0.318182, 0.132002]),\n  ('correlated clicks and sessions',\n   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],\n   [0.538462, 0.7, 0.172146]),\n  ('users with zero sessions count',\n   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],\n   [0.625, 0.4, 0.161613]),\n  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),\n  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),\n  ('user aggregate sample 21',\n   [[[4, 5], [2, 2], [2, 2]], [[2, 5], [2, 2], [5, 5], [7, 10]]],\n   [0.888889, 0.727273, 0.137879]),\n  ('user aggregate sample 22',\n   [[[0, 0], [2, 2], [8, 10], [0, 1], [2, 10]], [[0, 0], [1, 1], [0, 2], [0, 0]]],\n   [0.521739, 0.333333, 0.421009]),\n  ('user aggregate sample 28',\n   [[[0, 2], [0, 1], [1, 2], [3, 5], [2, 5], [0, 1]], [[4, 5], [1, 1], [1, 1]]],\n   [0.375, 0.857143, 0.118332])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"},"fixed":{"sha256":"1e036cd52a55db5c0ebe3d666e058f91f98e3daf5f19c0536d28fba4e1ea7763","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\nimport math\nN = 1\nobservations = []\ndef solve(users_c, users_t):\n    def stats(users):\n        n = len(users)\n        if n < 2:\n            return None\n        xs = [u[0] for u in users]\n        ys = [u[1] for u in users]\n        mx, my = sum(xs) / n, sum(ys) / n\n        if my == 0:\n            return None\n        r = mx / my\n        vx = sum((x - mx) ** 2 for x in xs) / (n - 1)\n        vy = sum((y - my) ** 2 for y in ys) / (n - 1)\n        cxy = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / (n - 1)\n        var = (vx - 2 * r * cxy + r * r * vy) / (n * my * my)\n        return r, var\n    a, b = stats(users_c), stats(users_t)\n    if a is None or b is None:\n        return None\n    return [round(a[0], 6), round(b[0], 6), round(math.sqrt(max(a[1] + b[1], 0.0)), 6)]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('heavy user dominates the ratio of sums',\n   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],\n   [0.52381, 0.318182, 0.132002]),\n  ('correlated clicks and sessions',\n   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],\n   [0.538462, 0.7, 0.172146]),\n  ('users with zero sessions count',\n   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],\n   [0.625, 0.4, 0.161613]),\n  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),\n  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),\n  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),\n  ('user aggregate sample 1', [[[1, 2], [9, 10]], [[0, 0], [0, 2], [7, 10]]], [0.833333, 0.583333, 0.201748]),\n  ('user aggregate sample 2',\n   [[[0, 0], [1, 5], [1, 2], [0, 0]], [[0, 0], [1, 1], [7, 10], [10, 10]]],\n   [0.285714, 0.857143, 0.153928])],\n [('correlated clicks and sessions',\n   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],\n   [0.538462, 0.7, 0.172146]),\n  ('users with zero sessions count',\n   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],\n   [0.625, 0.4, 0.161613]),\n  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),\n  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),\n  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),\n  ('user aggregate sample 5',\n   [[[6, 10], [0, 1], [0, 0], [0, 2], [8, 10]], [[1, 2], [1, 5], [0, 0], [0, 5], [0, 0]]],\n   [0.608696, 0.166667, 0.152218]),\n  ('user aggregate sample 6',\n   [[[1, 1], [5, 5], [1, 1], [3, 5], [10, 10]], [[0, 1], [1, 5], [0, 0], [0, 10], [1, 5], [1, 1]]],\n   [0.909091, 0.136364, 0.126043]),\n  ('user aggregate sample 7',\n   [[[0, 1], [5, 10], [1, 5]], [[1, 2], [1, 2], [0, 1], [3, 5], [5, 5]]],\n   [0.375, 0.666667, 0.184982])],\n [('heavy user dominates the ratio of sums',\n   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],\n   [0.52381, 0.318182, 0.132002]),\n  ('users with zero sessions count',\n   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],\n   [0.625, 0.4, 0.161613]),\n  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),\n  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),\n  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),\n  ('user aggregate sample 11',\n   [[[7, 10], [0, 0]], [[1, 5], [0, 1], [0, 5], [1, 1], [1, 1], [2, 10]]],\n   [0.7, 0.217391, 0.075178]),\n  ('user aggregate sample 12',\n   [[[0, 5], [5, 10], [0, 2], [0, 0]], [[5, 5], [0, 1], [10, 10], [7, 10], [1, 1], [0, 1]]],\n   [0.294118, 0.821429, 0.203966]),\n  ('user aggregate sample 13',\n   [[[0, 0], [1, 5], [0, 2], [0, 2], [1, 2], [7, 10]], [[2, 10], [2, 10], [1, 1], [0, 0], [1, 1]]],\n   [0.428571, 0.272727, 0.181987])],\n [('heavy user dominates the ratio of sums',\n   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],\n   [0.52381, 0.318182, 0.132002]),\n  ('correlated clicks and sessions',\n   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],\n   [0.538462, 0.7, 0.172146]),\n  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),\n  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),\n  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),\n  ('user aggregate sample 16',\n   [[[0, 0], [1, 1], [5, 5], [1, 1], [0, 1], [0, 10]], [[0, 0], [0, 2], [0, 0], [0, 10]]],\n   [0.388889, 0.0, 0.306461]),\n  ('user aggregate sample 17',\n   [[[0, 10], [0, 2]], [[3, 10], [6, 10], [0, 0], [5, 10]]],\n   [0.0, 0.466667, 0.083148]),\n  ('user aggregate sample 21',\n   [[[4, 5], [2, 2], [2, 2]], [[2, 5], [2, 2], [5, 5], [7, 10]]],\n   [0.888889, 0.727273, 0.137879])],\n [('heavy user dominates the ratio of sums',\n   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],\n   [0.52381, 0.318182, 0.132002]),\n  ('correlated clicks and sessions',\n   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],\n   [0.538462, 0.7, 0.172146]),\n  ('users with zero sessions count',\n   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],\n   [0.625, 0.4, 0.161613]),\n  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),\n  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),\n  ('user aggregate sample 21',\n   [[[4, 5], [2, 2], [2, 2]], [[2, 5], [2, 2], [5, 5], [7, 10]]],\n   [0.888889, 0.727273, 0.137879]),\n  ('user aggregate sample 22',\n   [[[0, 0], [2, 2], [8, 10], [0, 1], [2, 10]], [[0, 0], [1, 1], [0, 2], [0, 0]]],\n   [0.521739, 0.333333, 0.421009]),\n  ('user aggregate sample 28',\n   [[[0, 2], [0, 1], [1, 2], [3, 5], [2, 5], [0, 1]], [[4, 5], [1, 1], [1, 1]]],\n   [0.375, 0.857143, 0.118332])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"}},"limitations":"A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.","method":"Deterministic executable model with adversarial boundary fixtures.","provenance":{"created_by":"Failure Map","dependencies":"Python standard library","family":"w2-experiment-statistics-ratio-metric-unit-of-analysis","generated_at":"2026-09-29T14:48:57.730782+00:00","license":"CC0-1.0","python":"3.12.14","seed":1,"split":"open-access"},"relevance":"Click-through per session is analysed per user; treating sessions as independent understates variance.","repair":"Use the per-user delta-method variance.","root_cause":"The variance is the binomial R(1 - R) / total sessions.","sha256":"ee3b48618732fc8cf0382a46bb59ee741922bec9f33cc7ff732e4e388ff393f4","title":"Per-user ratio metric variance: Sessions are treated as independent trials · case 01","variant":1,"variant_policy":"Five numbered records share a model and may reuse boundary fixtures.","verification":{"attempt":{"elapsed_ms":41.523,"exit_code":1,"observations":[{"actual":[0.52381,0.318182,0.039912],"check":"heavy user dominates the ratio of sums","expected":[0.52381,0.318182,0.132002],"passed":false},{"actual":[0.538462,0.7,0.094383],"check":"correlated clicks and sessions","expected":[0.538462,0.7,0.172146],"passed":false},{"actual":[0.625,0.4,0.114169],"check":"users with zero sessions count","expected":[0.625,0.4,0.161613],"passed":false},{"actual":null,"check":"single user arm","expected":null,"passed":true},{"actual":null,"check":"no sessions in control","expected":null,"passed":true},{"actual":[0.5,0.5,0.176777],"check":"identical arms","expected":[0.5,0.5,0.353553],"passed":false},{"actual":[0.833333,0.583333,0.095639],"check":"user aggregate sample 1","expected":[0.833333,0.583333,0.201748],"passed":false},{"actual":[0.285714,0.857143,0.09122],"check":"user aggregate sample 2","expected":[0.285714,0.857143,0.153928],"passed":false}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"heavy user dominates the ratio of sums\", \"actual\": [0.52381, 0.318182, 0.039912], \"expected\": [0.52381, 0.318182, 0.132002], \"passed\": false}, {\"check\": \"correlated clicks and sessions\", \"actual\": [0.538462, 0.7, 0.094383], \"expected\": [0.538462, 0.7, 0.172146], \"passed\": false}, {\"check\": \"users with zero sessions count\", \"actual\": [0.625, 0.4, 0.114169], \"expected\": [0.625, 0.4, 0.161613], \"passed\": false}, {\"check\": \"single user arm\", \"actual\": null, \"expected\": null, \"passed\": true}, {\"check\": \"no sessions in control\", \"actual\": null, \"expected\": null, \"passed\": true}, {\"check\": \"identical arms\", \"actual\": [0.5, 0.5, 0.176777], \"expected\": [0.5, 0.5, 0.353553], \"passed\": false}, {\"check\": \"user aggregate sample 1\", \"actual\": [0.833333, 0.583333, 0.095639], \"expected\": [0.833333, 0.583333, 0.201748], \"passed\": false}, {\"check\": \"user aggregate sample 2\", \"actual\": [0.285714, 0.857143, 0.09122], \"expected\": [0.285714, 0.857143, 0.153928], \"passed\": false}], \"passed\": false}\n"},"broken":{"elapsed_ms":40.851,"exit_code":1,"observations":[{"actual":[0.52381,0.318182,0.147441],"check":"heavy user dominates the ratio of sums","expected":[0.52381,0.318182,0.132002],"passed":false},{"actual":[0.538462,0.7,0.200292],"check":"correlated clicks and sessions","expected":[0.538462,0.7,0.172146],"passed":false},{"actual":[0.625,0.4,0.278023],"check":"users with zero sessions count","expected":[0.625,0.4,0.161613],"passed":false},{"actual":null,"check":"single user arm","expected":null,"passed":true},{"actual":null,"check":"no sessions in control","expected":null,"passed":true},{"actual":[0.5,0.5,0.25],"check":"identical arms","expected":[0.5,0.5,0.353553],"passed":false},{"actual":[0.833333,0.583333,0.178406],"check":"user aggregate sample 1","expected":[0.833333,0.583333,0.201748],"passed":false},{"actual":[0.285714,0.857143,0.187044],"check":"user aggregate sample 2","expected":[0.285714,0.857143,0.153928],"passed":false}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"heavy user dominates the ratio of sums\", \"actual\": [0.52381, 0.318182, 0.147441], \"expected\": [0.52381, 0.318182, 0.132002], \"passed\": false}, {\"check\": \"correlated clicks and sessions\", \"actual\": [0.538462, 0.7, 0.200292], \"expected\": [0.538462, 0.7, 0.172146], \"passed\": false}, {\"check\": \"users with zero sessions count\", \"actual\": [0.625, 0.4, 0.278023], \"expected\": [0.625, 0.4, 0.161613], \"passed\": false}, {\"check\": \"single user arm\", \"actual\": null, \"expected\": null, \"passed\": true}, {\"check\": \"no sessions in control\", \"actual\": null, \"expected\": null, \"passed\": true}, {\"check\": \"identical arms\", \"actual\": [0.5, 0.5, 0.25], \"expected\": [0.5, 0.5, 0.353553], \"passed\": false}, {\"check\": \"user aggregate sample 1\", \"actual\": [0.833333, 0.583333, 0.178406], \"expected\": [0.833333, 0.583333, 0.201748], \"passed\": false}, {\"check\": \"user aggregate sample 2\", \"actual\": [0.285714, 0.857143, 0.187044], \"expected\": [0.285714, 0.857143, 0.153928], \"passed\": false}], \"passed\": false}\n"},"fixed":{"elapsed_ms":36.434,"exit_code":0,"observations":[{"actual":[0.52381,0.318182,0.132002],"check":"heavy user dominates the ratio of sums","expected":[0.52381,0.318182,0.132002],"passed":true},{"actual":[0.538462,0.7,0.172146],"check":"correlated clicks and sessions","expected":[0.538462,0.7,0.172146],"passed":true},{"actual":[0.625,0.4,0.161613],"check":"users with zero sessions count","expected":[0.625,0.4,0.161613],"passed":true},{"actual":null,"check":"single user arm","expected":null,"passed":true},{"actual":null,"check":"no sessions in control","expected":null,"passed":true},{"actual":[0.5,0.5,0.353553],"check":"identical arms","expected":[0.5,0.5,0.353553],"passed":true},{"actual":[0.833333,0.583333,0.201748],"check":"user aggregate sample 1","expected":[0.833333,0.583333,0.201748],"passed":true},{"actual":[0.285714,0.857143,0.153928],"check":"user aggregate sample 2","expected":[0.285714,0.857143,0.153928],"passed":true}],"passed":true,"stderr":"","stdout":"{\"observations\": [{\"check\": \"heavy user dominates the ratio of sums\", \"actual\": [0.52381, 0.318182, 0.132002], \"expected\": [0.52381, 0.318182, 0.132002], \"passed\": true}, {\"check\": \"correlated clicks and sessions\", \"actual\": [0.538462, 0.7, 0.172146], \"expected\": [0.538462, 0.7, 0.172146], \"passed\": true}, {\"check\": \"users with zero sessions count\", \"actual\": [0.625, 0.4, 0.161613], \"expected\": [0.625, 0.4, 0.161613], \"passed\": true}, {\"check\": \"single user arm\", \"actual\": null, \"expected\": null, \"passed\": true}, {\"check\": \"no sessions in control\", \"actual\": null, \"expected\": null, \"passed\": true}, {\"check\": \"identical arms\", \"actual\": [0.5, 0.5, 0.353553], \"expected\": [0.5, 0.5, 0.353553], \"passed\": true}, {\"check\": \"user aggregate sample 1\", \"actual\": [0.833333, 0.583333, 0.201748], \"expected\": [0.833333, 0.583333, 0.201748], \"passed\": true}, {\"check\": \"user aggregate sample 2\", \"actual\": [0.285714, 0.857143, 0.153928], \"expected\": [0.285714, 0.857143, 0.153928], \"passed\": true}], \"passed\": true}\n"}},"verified":true,"visibility":"public"}