FAILURE MAP
← Case archive

FA-74536 / Experiment statistics / Open access

Per-user ratio metric variance: The delta-method variance is divided by mean sessions once · case 01

Standard errors scale with the session rate instead of its square.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The denominator is n * mean_y instead of n * mean_y^2.

THE FAILURE

The denominator is n * mean_y instead of n * mean_y^2.

Unsuccessful approach: Dropping n gives the variance of a single user rather than of the arm ratio.

Case contract

Each user contributes [clicks, sessions]; the unit of randomisation is the user. The arm ratio is sum(clicks) / sum(sessions). Its delta-method variance is (var_x - 2 R cov_xy + R^2 var_y) / (n mean_y^2) with sample (n - 1) moments over users. Arms with fewer than two users or no sessions -> None. Return [R_control, R_treatment, standard error of the difference] rounded to 6.

Why this case matters

Click-through per session is analysed per user; treating sessions as independent understates variance.

1 / The failure

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(users_c, users_t):
    def stats(users):
        n = len(users)
        if n < 2:
            return None
        xs = [u[0] for u in users]
        ys = [u[1] for u in users]
        mx, my = sum(xs) / n, sum(ys) / n
        if my == 0:
            return None
        r = mx / my
        vx = sum((x - mx) ** 2 for x in xs) / (n - 1)
        vy = sum((y - my) ** 2 for y in ys) / (n - 1)
        cxy = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / (n - 1)
        var = (vx - 2 * r * cxy + r * r * vy) / (n * my)
        return r, var
    a, b = stats(users_c), stats(users_t)
    if a is None or b is None:
        return None
    return [round(a[0], 6), round(b[0], 6), round(math.sqrt(max(a[1] + b[1], 0.0)), 6)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('heavy user dominates the ratio of sums',
   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],
   [0.52381, 0.318182, 0.132002]),
  ('correlated clicks and sessions',
   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],
   [0.538462, 0.7, 0.172146]),
  ('users with zero sessions count',
   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],
   [0.625, 0.4, 0.161613]),
  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 1', [[[1, 2], [9, 10]], [[0, 0], [0, 2], [7, 10]]], [0.833333, 0.583333, 0.201748]),
  ('user aggregate sample 2',
   [[[0, 0], [1, 5], [1, 2], [0, 0]], [[0, 0], [1, 1], [7, 10], [10, 10]]],
   [0.285714, 0.857143, 0.153928])],
 [('correlated clicks and sessions',
   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],
   [0.538462, 0.7, 0.172146]),
  ('users with zero sessions count',
   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],
   [0.625, 0.4, 0.161613]),
  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 5',
   [[[6, 10], [0, 1], [0, 0], [0, 2], [8, 10]], [[1, 2], [1, 5], [0, 0], [0, 5], [0, 0]]],
   [0.608696, 0.166667, 0.152218]),
  ('user aggregate sample 6',
   [[[1, 1], [5, 5], [1, 1], [3, 5], [10, 10]], [[0, 1], [1, 5], [0, 0], [0, 10], [1, 5], [1, 1]]],
   [0.909091, 0.136364, 0.126043]),
  ('user aggregate sample 7',
   [[[0, 1], [5, 10], [1, 5]], [[1, 2], [1, 2], [0, 1], [3, 5], [5, 5]]],
   [0.375, 0.666667, 0.184982])],
 [('heavy user dominates the ratio of sums',
   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],
   [0.52381, 0.318182, 0.132002]),
  ('users with zero sessions count',
   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],
   [0.625, 0.4, 0.161613]),
  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 11',
   [[[7, 10], [0, 0]], [[1, 5], [0, 1], [0, 5], [1, 1], [1, 1], [2, 10]]],
   [0.7, 0.217391, 0.075178]),
  ('user aggregate sample 12',
   [[[0, 5], [5, 10], [0, 2], [0, 0]], [[5, 5], [0, 1], [10, 10], [7, 10], [1, 1], [0, 1]]],
   [0.294118, 0.821429, 0.203966]),
  ('user aggregate sample 13',
   [[[0, 0], [1, 5], [0, 2], [0, 2], [1, 2], [7, 10]], [[2, 10], [2, 10], [1, 1], [0, 0], [1, 1]]],
   [0.428571, 0.272727, 0.181987])],
 [('heavy user dominates the ratio of sums',
   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],
   [0.52381, 0.318182, 0.132002]),
  ('correlated clicks and sessions',
   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],
   [0.538462, 0.7, 0.172146]),
  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 16',
   [[[0, 0], [1, 1], [5, 5], [1, 1], [0, 1], [0, 10]], [[0, 0], [0, 2], [0, 0], [0, 10]]],
   [0.388889, 0.0, 0.306461]),
  ('user aggregate sample 17',
   [[[0, 10], [0, 2]], [[3, 10], [6, 10], [0, 0], [5, 10]]],
   [0.0, 0.466667, 0.083148]),
  ('user aggregate sample 21',
   [[[4, 5], [2, 2], [2, 2]], [[2, 5], [2, 2], [5, 5], [7, 10]]],
   [0.888889, 0.727273, 0.137879])],
 [('heavy user dominates the ratio of sums',
   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],
   [0.52381, 0.318182, 0.132002]),
  ('correlated clicks and sessions',
   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],
   [0.538462, 0.7, 0.172146]),
  ('users with zero sessions count',
   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],
   [0.625, 0.4, 0.161613]),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 21',
   [[[4, 5], [2, 2], [2, 2]], [[2, 5], [2, 2], [5, 5], [7, 10]]],
   [0.888889, 0.727273, 0.137879]),
  ('user aggregate sample 22',
   [[[0, 0], [2, 2], [8, 10], [0, 1], [2, 10]], [[0, 0], [1, 1], [0, 2], [0, 0]]],
   [0.521739, 0.333333, 0.421009]),
  ('user aggregate sample 28',
   [[[0, 2], [0, 1], [1, 2], [3, 5], [2, 5], [0, 1]], [[4, 5], [1, 1], [1, 1]]],
   [0.375, 0.857143, 0.118332])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
heavy user dominates the ratio of sums[0.52381, 0.318182, 0.436625][0.52381, 0.318182, 0.132002]Failed
correlated clicks and sessions[0.538462, 0.7, 0.313984][0.538462, 0.7, 0.172146]Failed
users with zero sessions count[0.625, 0.4, 0.235053][0.625, 0.4, 0.161613]Failed
single user armNoneNonePassed
no sessions in controlNoneNonePassed
identical arms[0.5, 0.5, 0.707107][0.5, 0.5, 0.353553]Failed
user aggregate sample 1[0.833333, 0.583333, 0.433013][0.833333, 0.583333, 0.201748]Failed
user aggregate sample 2[0.285714, 0.857143, 0.29901][0.285714, 0.857143, 0.153928]Failed

SHA-256 / d945ac57453a246851bf247b258fbdc685cbd4ec23e5573605af3db8777948bd

2 / The unsuccessful fix

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(users_c, users_t):
    def stats(users):
        n = len(users)
        if n < 2:
            return None
        xs = [u[0] for u in users]
        ys = [u[1] for u in users]
        mx, my = sum(xs) / n, sum(ys) / n
        if my == 0:
            return None
        r = mx / my
        vx = sum((x - mx) ** 2 for x in xs) / (n - 1)
        vy = sum((y - my) ** 2 for y in ys) / (n - 1)
        cxy = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / (n - 1)
        var = (vx - 2 * r * cxy + r * r * vy) / (my * my)
        return r, var
    a, b = stats(users_c), stats(users_t)
    if a is None or b is None:
        return None
    return [round(a[0], 6), round(b[0], 6), round(math.sqrt(max(a[1] + b[1], 0.0)), 6)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('heavy user dominates the ratio of sums',
   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],
   [0.52381, 0.318182, 0.132002]),
  ('correlated clicks and sessions',
   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],
   [0.538462, 0.7, 0.172146]),
  ('users with zero sessions count',
   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],
   [0.625, 0.4, 0.161613]),
  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 1', [[[1, 2], [9, 10]], [[0, 0], [0, 2], [7, 10]]], [0.833333, 0.583333, 0.201748]),
  ('user aggregate sample 2',
   [[[0, 0], [1, 5], [1, 2], [0, 0]], [[0, 0], [1, 1], [7, 10], [10, 10]]],
   [0.285714, 0.857143, 0.153928])],
 [('correlated clicks and sessions',
   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],
   [0.538462, 0.7, 0.172146]),
  ('users with zero sessions count',
   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],
   [0.625, 0.4, 0.161613]),
  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 5',
   [[[6, 10], [0, 1], [0, 0], [0, 2], [8, 10]], [[1, 2], [1, 5], [0, 0], [0, 5], [0, 0]]],
   [0.608696, 0.166667, 0.152218]),
  ('user aggregate sample 6',
   [[[1, 1], [5, 5], [1, 1], [3, 5], [10, 10]], [[0, 1], [1, 5], [0, 0], [0, 10], [1, 5], [1, 1]]],
   [0.909091, 0.136364, 0.126043]),
  ('user aggregate sample 7',
   [[[0, 1], [5, 10], [1, 5]], [[1, 2], [1, 2], [0, 1], [3, 5], [5, 5]]],
   [0.375, 0.666667, 0.184982])],
 [('heavy user dominates the ratio of sums',
   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],
   [0.52381, 0.318182, 0.132002]),
  ('users with zero sessions count',
   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],
   [0.625, 0.4, 0.161613]),
  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 11',
   [[[7, 10], [0, 0]], [[1, 5], [0, 1], [0, 5], [1, 1], [1, 1], [2, 10]]],
   [0.7, 0.217391, 0.075178]),
  ('user aggregate sample 12',
   [[[0, 5], [5, 10], [0, 2], [0, 0]], [[5, 5], [0, 1], [10, 10], [7, 10], [1, 1], [0, 1]]],
   [0.294118, 0.821429, 0.203966]),
  ('user aggregate sample 13',
   [[[0, 0], [1, 5], [0, 2], [0, 2], [1, 2], [7, 10]], [[2, 10], [2, 10], [1, 1], [0, 0], [1, 1]]],
   [0.428571, 0.272727, 0.181987])],
 [('heavy user dominates the ratio of sums',
   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],
   [0.52381, 0.318182, 0.132002]),
  ('correlated clicks and sessions',
   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],
   [0.538462, 0.7, 0.172146]),
  ('single user arm', [[[1, 2]], [[1, 2], [2, 3]]], None),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 16',
   [[[0, 0], [1, 1], [5, 5], [1, 1], [0, 1], [0, 10]], [[0, 0], [0, 2], [0, 0], [0, 10]]],
   [0.388889, 0.0, 0.306461]),
  ('user aggregate sample 17',
   [[[0, 10], [0, 2]], [[3, 10], [6, 10], [0, 0], [5, 10]]],
   [0.0, 0.466667, 0.083148]),
  ('user aggregate sample 21',
   [[[4, 5], [2, 2], [2, 2]], [[2, 5], [2, 2], [5, 5], [7, 10]]],
   [0.888889, 0.727273, 0.137879])],
 [('heavy user dominates the ratio of sums',
   [[[1, 1], [10, 20]], [[2, 2], [5, 20]]],
   [0.52381, 0.318182, 0.132002]),
  ('correlated clicks and sessions',
   [[[1, 2], [2, 4], [3, 6], [1, 1]], [[2, 2], [4, 5], [1, 3]]],
   [0.538462, 0.7, 0.172146]),
  ('users with zero sessions count',
   [[[0, 0], [2, 4], [3, 4]], [[1, 2], [1, 2], [0, 1]]],
   [0.625, 0.4, 0.161613]),
  ('no sessions in control', [[[0, 0], [0, 0]], [[1, 2], [2, 3]]], None),
  ('identical arms', [[[1, 4], [3, 4]], [[1, 4], [3, 4]]], [0.5, 0.5, 0.353553]),
  ('user aggregate sample 21',
   [[[4, 5], [2, 2], [2, 2]], [[2, 5], [2, 2], [5, 5], [7, 10]]],
   [0.888889, 0.727273, 0.137879]),
  ('user aggregate sample 22',
   [[[0, 0], [2, 2], [8, 10], [0, 1], [2, 10]], [[0, 0], [1, 1], [0, 2], [0, 0]]],
   [0.521739, 0.333333, 0.421009]),
  ('user aggregate sample 28',
   [[[0, 2], [0, 1], [1, 2], [3, 5], [2, 5], [0, 1]], [[4, 5], [1, 1], [1, 1]]],
   [0.375, 0.857143, 0.118332])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
heavy user dominates the ratio of sums[0.52381, 0.318182, 0.186679][0.52381, 0.318182, 0.132002]Failed
correlated clicks and sessions[0.538462, 0.7, 0.302054][0.538462, 0.7, 0.172146]Failed
users with zero sessions count[0.625, 0.4, 0.279922][0.625, 0.4, 0.161613]Failed
single user armNoneNonePassed
no sessions in controlNoneNonePassed
identical arms[0.5, 0.5, 0.5][0.5, 0.5, 0.353553]Failed
user aggregate sample 1[0.833333, 0.583333, 0.331302][0.833333, 0.583333, 0.201748]Failed
user aggregate sample 2[0.285714, 0.857143, 0.307856][0.285714, 0.857143, 0.153928]Failed

SHA-256 / 7079fd434bc43dffdf92584a4dd58c72972d95b86ac5967da5142d3ec641923f

HELD IN THE MEMBER ARCHIVE

The verified repair and its recorded checks are member-only.

This mechanism has 8 recorded checks per implementation. The open-access tier publishes the failure and the unsuccessful fix; the repaired source that passes every check, and the observations that prove it, are available to members.

Every case sharing this mechanism uses the same contract and the same repair, so this one record is held back for all of them.

Member access is invitation-based. Sign in with your invited account to inspect the repair.

Sign in to the archive ↗

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:57.778355+00:00.

Case digest / 988a698c59e532705081535c27ad144408a59f95dd1dcf61716a752464c2f419