FAILURE MAP
← Case archive

FA-74531 / Experiment statistics / Open access

Per-user ratio metric variance: Sessions are treated as independent trials · case 01

Intervals are far too narrow because repeated sessions of one user are assumed independent.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The variance is the binomial R(1 - R) / total sessions.

VERIFIED REPAIR

Use the per-user delta-method variance.

Unsuccessful approach: Keeping the delta-method numerator but dividing by session count still counts sessions as units.

Case contract

Each user contributes [clicks, sessions]; the unit of randomisation is the user. The arm ratio is sum(clicks) / sum(sessions). Its delta-method variance is (var_x - 2 R cov_xy + R^2 var_y) / (n mean_y^2) with sample (n - 1) moments over users. Arms with fewer than two users or no sessions -> None. Return [R_control, R_treatment, standard error of the difference] rounded to 6.

Why this case matters

Click-through per session is analysed per user; treating sessions as independent understates variance.

1 / The failure

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(users_c, users_t):
    def stats(users):
        n = len(users)
        if n < 2:
            return None
        xs = [u[0] for u in users]
        ys = [u[1] for u in users]
        mx, my = sum(xs) / n, sum(ys) / n
        if my == 0:
            return None
        r = mx / my
        vx = sum((x - mx) ** 2 for x in xs) / (n - 1)
        vy = sum((y - my) ** 2 for y in ys) / (n - 1)
        cxy = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / (n - 1)
        var = r * (1 - r) / sum(ys)
        return r, var
    a, b = stats(users_c), stats(users_t)
    if a is None or b is None:
        return None
    return [round(a[0], 6), round(b[0], 6), round(math.sqrt(max(a[1] + b[1], 0.0)), 6)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('heavy user dominates the ratio of sums',
   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],
   [0.52381, 0.318182, 0.132002]),
  ('correlated clicks and sessions',
   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],
   [0.538462, 0.7, 0.172146]),
  ('users with zero sessions count',
   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],
   [0.625, 0.4, 0.161613]),
  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 1', [[[1, 2], [9, 10]], [[0, 0], [0, 2], [7, 10]]], [0.833333, 0.583333, 0.201748]),
  ('user aggregate sample 2',
   [[[0, 0], [1, 5], [1, 2], [0, 0]], [[0, 0], [1, 1], [7, 10], [10, 10]]],
   [0.285714, 0.857143, 0.153928])],
 [('correlated clicks and sessions',
   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],
   [0.538462, 0.7, 0.172146]),
  ('users with zero sessions count',
   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],
   [0.625, 0.4, 0.161613]),
  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 5',
   [[[6, 10], [0, 1], [0, 0], [0, 2], [8, 10]], [[1, 2], [1, 5], [0, 0], [0, 5], [0, 0]]],
   [0.608696, 0.166667, 0.152218]),
  ('user aggregate sample 6',
   [[[1, 1], [5, 5], [1, 1], [3, 5], [10, 10]], [[0, 1], [1, 5], [0, 0], [0, 10], [1, 5], [1, 1]]],
   [0.909091, 0.136364, 0.126043]),
  ('user aggregate sample 7',
   [[[0, 1], [5, 10], [1, 5]], [[1, 2], [1, 2], [0, 1], [3, 5], [5, 5]]],
   [0.375, 0.666667, 0.184982])],
 [('heavy user dominates the ratio of sums',
   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],
   [0.52381, 0.318182, 0.132002]),
  ('users with zero sessions count',
   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],
   [0.625, 0.4, 0.161613]),
  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 11',
   [[[7, 10], [0, 0]], [[1, 5], [0, 1], [0, 5], [1, 1], [1, 1], [2, 10]]],
   [0.7, 0.217391, 0.075178]),
  ('user aggregate sample 12',
   [[[0, 5], [5, 10], [0, 2], [0, 0]], [[5, 5], [0, 1], [10, 10], [7, 10], [1, 1], [0, 1]]],
   [0.294118, 0.821429, 0.203966]),
  ('user aggregate sample 13',
   [[[0, 0], [1, 5], [0, 2], [0, 2], [1, 2], [7, 10]], [[2, 10], [2, 10], [1, 1], [0, 0], [1, 1]]],
   [0.428571, 0.272727, 0.181987])],
 [('heavy user dominates the ratio of sums',
   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],
   [0.52381, 0.318182, 0.132002]),
  ('correlated clicks and sessions',
   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],
   [0.538462, 0.7, 0.172146]),
  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 16',
   [[[0, 0], [1, 1], [5, 5], [1, 1], [0, 1], [0, 10]], [[0, 0], [0, 2], [0, 0], [0, 10]]],
   [0.388889, 0.0, 0.306461]),
  ('user aggregate sample 17',
   [[[0, 10], [0, 2]], [[3, 10], [6, 10], [0, 0], [5, 10]]],
   [0.0, 0.466667, 0.083148]),
  ('user aggregate sample 21',
   [[[4, 5], [2, 2], [2, 2]], [[2, 5], [2, 2], [5, 5], [7, 10]]],
   [0.888889, 0.727273, 0.137879])],
 [('heavy user dominates the ratio of sums',
   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],
   [0.52381, 0.318182, 0.132002]),
  ('correlated clicks and sessions',
   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],
   [0.538462, 0.7, 0.172146]),
  ('users with zero sessions count',
   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],
   [0.625, 0.4, 0.161613]),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 21',
   [[[4, 5], [2, 2], [2, 2]], [[2, 5], [2, 2], [5, 5], [7, 10]]],
   [0.888889, 0.727273, 0.137879]),
  ('user aggregate sample 22',
   [[[0, 0], [2, 2], [8, 10], [0, 1], [2, 10]], [[0, 0], [1, 1], [0, 2], [0, 0]]],
   [0.521739, 0.333333, 0.421009]),
  ('user aggregate sample 28',
   [[[0, 2], [0, 1], [1, 2], [3, 5], [2, 5], [0, 1]], [[4, 5], [1, 1], [1, 1]]],
   [0.375, 0.857143, 0.118332])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
heavy user dominates the ratio of sums[0.52381, 0.318182, 0.147441][0.52381, 0.318182, 0.132002]Failed
correlated clicks and sessions[0.538462, 0.7, 0.200292][0.538462, 0.7, 0.172146]Failed
users with zero sessions count[0.625, 0.4, 0.278023][0.625, 0.4, 0.161613]Failed
single user armNoneNonePassed
no sessions in controlNoneNonePassed
identical arms[0.5, 0.5, 0.25][0.5, 0.5, 0.353553]Failed
user aggregate sample 1[0.833333, 0.583333, 0.178406][0.833333, 0.583333, 0.201748]Failed
user aggregate sample 2[0.285714, 0.857143, 0.187044][0.285714, 0.857143, 0.153928]Failed

SHA-256 / 52d614cb7bcd12ab1aa7c0e34c52555126bfc0739b5043d33c4a3d0d09f19d15

2 / The unsuccessful fix

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(users_c, users_t):
    def stats(users):
        n = len(users)
        if n < 2:
            return None
        xs = [u[0] for u in users]
        ys = [u[1] for u in users]
        mx, my = sum(xs) / n, sum(ys) / n
        if my == 0:
            return None
        r = mx / my
        vx = sum((x - mx) ** 2 for x in xs) / (n - 1)
        vy = sum((y - my) ** 2 for y in ys) / (n - 1)
        cxy = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / (n - 1)
        var = (vx - 2 * r * cxy + r * r * vy) / (sum(ys) * my * my)
        return r, var
    a, b = stats(users_c), stats(users_t)
    if a is None or b is None:
        return None
    return [round(a[0], 6), round(b[0], 6), round(math.sqrt(max(a[1] + b[1], 0.0)), 6)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('heavy user dominates the ratio of sums',
   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],
   [0.52381, 0.318182, 0.132002]),
  ('correlated clicks and sessions',
   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],
   [0.538462, 0.7, 0.172146]),
  ('users with zero sessions count',
   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],
   [0.625, 0.4, 0.161613]),
  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 1', [[[1, 2], [9, 10]], [[0, 0], [0, 2], [7, 10]]], [0.833333, 0.583333, 0.201748]),
  ('user aggregate sample 2',
   [[[0, 0], [1, 5], [1, 2], [0, 0]], [[0, 0], [1, 1], [7, 10], [10, 10]]],
   [0.285714, 0.857143, 0.153928])],
 [('correlated clicks and sessions',
   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],
   [0.538462, 0.7, 0.172146]),
  ('users with zero sessions count',
   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],
   [0.625, 0.4, 0.161613]),
  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 5',
   [[[6, 10], [0, 1], [0, 0], [0, 2], [8, 10]], [[1, 2], [1, 5], [0, 0], [0, 5], [0, 0]]],
   [0.608696, 0.166667, 0.152218]),
  ('user aggregate sample 6',
   [[[1, 1], [5, 5], [1, 1], [3, 5], [10, 10]], [[0, 1], [1, 5], [0, 0], [0, 10], [1, 5], [1, 1]]],
   [0.909091, 0.136364, 0.126043]),
  ('user aggregate sample 7',
   [[[0, 1], [5, 10], [1, 5]], [[1, 2], [1, 2], [0, 1], [3, 5], [5, 5]]],
   [0.375, 0.666667, 0.184982])],
 [('heavy user dominates the ratio of sums',
   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],
   [0.52381, 0.318182, 0.132002]),
  ('users with zero sessions count',
   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],
   [0.625, 0.4, 0.161613]),
  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 11',
   [[[7, 10], [0, 0]], [[1, 5], [0, 1], [0, 5], [1, 1], [1, 1], [2, 10]]],
   [0.7, 0.217391, 0.075178]),
  ('user aggregate sample 12',
   [[[0, 5], [5, 10], [0, 2], [0, 0]], [[5, 5], [0, 1], [10, 10], [7, 10], [1, 1], [0, 1]]],
   [0.294118, 0.821429, 0.203966]),
  ('user aggregate sample 13',
   [[[0, 0], [1, 5], [0, 2], [0, 2], [1, 2], [7, 10]], [[2, 10], [2, 10], [1, 1], [0, 0], [1, 1]]],
   [0.428571, 0.272727, 0.181987])],
 [('heavy user dominates the ratio of sums',
   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],
   [0.52381, 0.318182, 0.132002]),
  ('correlated clicks and sessions',
   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],
   [0.538462, 0.7, 0.172146]),
  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 16',
   [[[0, 0], [1, 1], [5, 5], [1, 1], [0, 1], [0, 10]], [[0, 0], [0, 2], [0, 0], [0, 10]]],
   [0.388889, 0.0, 0.306461]),
  ('user aggregate sample 17',
   [[[0, 10], [0, 2]], [[3, 10], [6, 10], [0, 0], [5, 10]]],
   [0.0, 0.466667, 0.083148]),
  ('user aggregate sample 21',
   [[[4, 5], [2, 2], [2, 2]], [[2, 5], [2, 2], [5, 5], [7, 10]]],
   [0.888889, 0.727273, 0.137879])],
 [('heavy user dominates the ratio of sums',
   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],
   [0.52381, 0.318182, 0.132002]),
  ('correlated clicks and sessions',
   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],
   [0.538462, 0.7, 0.172146]),
  ('users with zero sessions count',
   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],
   [0.625, 0.4, 0.161613]),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 21',
   [[[4, 5], [2, 2], [2, 2]], [[2, 5], [2, 2], [5, 5], [7, 10]]],
   [0.888889, 0.727273, 0.137879]),
  ('user aggregate sample 22',
   [[[0, 0], [2, 2], [8, 10], [0, 1], [2, 10]], [[0, 0], [1, 1], [0, 2], [0, 0]]],
   [0.521739, 0.333333, 0.421009]),
  ('user aggregate sample 28',
   [[[0, 2], [0, 1], [1, 2], [3, 5], [2, 5], [0, 1]], [[4, 5], [1, 1], [1, 1]]],
   [0.375, 0.857143, 0.118332])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
heavy user dominates the ratio of sums[0.52381, 0.318182, 0.039912][0.52381, 0.318182, 0.132002]Failed
correlated clicks and sessions[0.538462, 0.7, 0.094383][0.538462, 0.7, 0.172146]Failed
users with zero sessions count[0.625, 0.4, 0.114169][0.625, 0.4, 0.161613]Failed
single user armNoneNonePassed
no sessions in controlNoneNonePassed
identical arms[0.5, 0.5, 0.176777][0.5, 0.5, 0.353553]Failed
user aggregate sample 1[0.833333, 0.583333, 0.095639][0.833333, 0.583333, 0.201748]Failed
user aggregate sample 2[0.285714, 0.857143, 0.09122][0.285714, 0.857143, 0.153928]Failed

SHA-256 / 8d4b20283fea3529641a72d4e3fafe61cc8764f8f086b8771a5893d1f93b8034

3 / The verified repair

Exit 0
"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(users_c, users_t):
    def stats(users):
        n = len(users)
        if n < 2:
            return None
        xs = [u[0] for u in users]
        ys = [u[1] for u in users]
        mx, my = sum(xs) / n, sum(ys) / n
        if my == 0:
            return None
        r = mx / my
        vx = sum((x - mx) ** 2 for x in xs) / (n - 1)
        vy = sum((y - my) ** 2 for y in ys) / (n - 1)
        cxy = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / (n - 1)
        var = (vx - 2 * r * cxy + r * r * vy) / (n * my * my)
        return r, var
    a, b = stats(users_c), stats(users_t)
    if a is None or b is None:
        return None
    return [round(a[0], 6), round(b[0], 6), round(math.sqrt(max(a[1] + b[1], 0.0)), 6)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('heavy user dominates the ratio of sums',
   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],
   [0.52381, 0.318182, 0.132002]),
  ('correlated clicks and sessions',
   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],
   [0.538462, 0.7, 0.172146]),
  ('users with zero sessions count',
   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],
   [0.625, 0.4, 0.161613]),
  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 1', [[[1, 2], [9, 10]], [[0, 0], [0, 2], [7, 10]]], [0.833333, 0.583333, 0.201748]),
  ('user aggregate sample 2',
   [[[0, 0], [1, 5], [1, 2], [0, 0]], [[0, 0], [1, 1], [7, 10], [10, 10]]],
   [0.285714, 0.857143, 0.153928])],
 [('correlated clicks and sessions',
   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],
   [0.538462, 0.7, 0.172146]),
  ('users with zero sessions count',
   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],
   [0.625, 0.4, 0.161613]),
  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 5',
   [[[6, 10], [0, 1], [0, 0], [0, 2], [8, 10]], [[1, 2], [1, 5], [0, 0], [0, 5], [0, 0]]],
   [0.608696, 0.166667, 0.152218]),
  ('user aggregate sample 6',
   [[[1, 1], [5, 5], [1, 1], [3, 5], [10, 10]], [[0, 1], [1, 5], [0, 0], [0, 10], [1, 5], [1, 1]]],
   [0.909091, 0.136364, 0.126043]),
  ('user aggregate sample 7',
   [[[0, 1], [5, 10], [1, 5]], [[1, 2], [1, 2], [0, 1], [3, 5], [5, 5]]],
   [0.375, 0.666667, 0.184982])],
 [('heavy user dominates the ratio of sums',
   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],
   [0.52381, 0.318182, 0.132002]),
  ('users with zero sessions count',
   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],
   [0.625, 0.4, 0.161613]),
  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 11',
   [[[7, 10], [0, 0]], [[1, 5], [0, 1], [0, 5], [1, 1], [1, 1], [2, 10]]],
   [0.7, 0.217391, 0.075178]),
  ('user aggregate sample 12',
   [[[0, 5], [5, 10], [0, 2], [0, 0]], [[5, 5], [0, 1], [10, 10], [7, 10], [1, 1], [0, 1]]],
   [0.294118, 0.821429, 0.203966]),
  ('user aggregate sample 13',
   [[[0, 0], [1, 5], [0, 2], [0, 2], [1, 2], [7, 10]], [[2, 10], [2, 10], [1, 1], [0, 0], [1, 1]]],
   [0.428571, 0.272727, 0.181987])],
 [('heavy user dominates the ratio of sums',
   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],
   [0.52381, 0.318182, 0.132002]),
  ('correlated clicks and sessions',
   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],
   [0.538462, 0.7, 0.172146]),
  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 16',
   [[[0, 0], [1, 1], [5, 5], [1, 1], [0, 1], [0, 10]], [[0, 0], [0, 2], [0, 0], [0, 10]]],
   [0.388889, 0.0, 0.306461]),
  ('user aggregate sample 17',
   [[[0, 10], [0, 2]], [[3, 10], [6, 10], [0, 0], [5, 10]]],
   [0.0, 0.466667, 0.083148]),
  ('user aggregate sample 21',
   [[[4, 5], [2, 2], [2, 2]], [[2, 5], [2, 2], [5, 5], [7, 10]]],
   [0.888889, 0.727273, 0.137879])],
 [('heavy user dominates the ratio of sums',
   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],
   [0.52381, 0.318182, 0.132002]),
  ('correlated clicks and sessions',
   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],
   [0.538462, 0.7, 0.172146]),
  ('users with zero sessions count',
   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],
   [0.625, 0.4, 0.161613]),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 21',
   [[[4, 5], [2, 2], [2, 2]], [[2, 5], [2, 2], [5, 5], [7, 10]]],
   [0.888889, 0.727273, 0.137879]),
  ('user aggregate sample 22',
   [[[0, 0], [2, 2], [8, 10], [0, 1], [2, 10]], [[0, 0], [1, 1], [0, 2], [0, 0]]],
   [0.521739, 0.333333, 0.421009]),
  ('user aggregate sample 28',
   [[[0, 2], [0, 1], [1, 2], [3, 5], [2, 5], [0, 1]], [[4, 5], [1, 1], [1, 1]]],
   [0.375, 0.857143, 0.118332])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
heavy user dominates the ratio of sums[0.52381, 0.318182, 0.132002][0.52381, 0.318182, 0.132002]Passed
correlated clicks and sessions[0.538462, 0.7, 0.172146][0.538462, 0.7, 0.172146]Passed
users with zero sessions count[0.625, 0.4, 0.161613][0.625, 0.4, 0.161613]Passed
single user armNoneNonePassed
no sessions in controlNoneNonePassed
identical arms[0.5, 0.5, 0.353553][0.5, 0.5, 0.353553]Passed
user aggregate sample 1[0.833333, 0.583333, 0.201748][0.833333, 0.583333, 0.201748]Passed
user aggregate sample 2[0.285714, 0.857143, 0.153928][0.285714, 0.857143, 0.153928]Passed

SHA-256 / 1e036cd52a55db5c0ebe3d666e058f91f98e3daf5f19c0536d28fba4e1ea7763

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:57.730782+00:00.

Case digest / ee3b48618732fc8cf0382a46bb59ee741922bec9f33cc7ff732e4e388ff393f4