FA-74525 / Experiment statistics / Member archive

Per-user ratio metric variance: The metric averages per-user ratios · case 05

Light users with one session swing the metric as much as heavy users.

Member previewVariant 5 · 3 implementations · 8 checks per implementation

Case contract

Each user contributes [clicks, sessions]; the unit of randomisation is the user. The arm ratio is sum(clicks) / sum(sessions). Its delta-method variance is (var_x - 2 R cov_xy + R^2 var_y) / (n mean_y^2) with sample (n - 1) moments over users. Arms with fewer than two users or no sessions -> None. Return [R_control, R_treatment, standard error of the difference] rounded to 6.

Why this case matters

Click-through per session is analysed per user; treating sessions as independent understates variance.

One recorded failure

Sample boundary fixture

This sample comes from the broken implementation of a controlled reproducer.

Boundary fixture	Actual	Expected	Outcome
heavy user dominates the ratio of sums	`[0.75, 0.625, 0.450694]`	`[0.52381, 0.318182, 0.132002]`	Failed

MEMBER ARCHIVE

The complete case is available to members.

This record includes three runnable implementations, regression fixtures, execution results, and source hashes.

Member access is invitation-based. Sign in with your invited account to inspect the sources.