FA-74379 / Experiment statistics / Member archive

Conversion rate z-test: Two-sided p-values are halved · case 04

Twice as many experiments are declared significant as the alpha level allows.

Member previewVariant 4 · 3 implementations · 8 checks per implementation

Case contract

Pooled two-proportion z-test: pool = (conv_a + conv_b) / (n_a + n_b), se = sqrt(pool(1 - pool)(1/n_a + 1/n_b)), z = (p_b - p_a) / se, two-sided p = erfc(|z| / sqrt 2). Nonpositive n -> None; se = 0 -> [0.0, 1.0]. Return [round(z, 6), round(p, 6)].

Why this case matters

Online experiment readouts drive launch decisions; a silent formula slip flips conclusions.

One recorded failure

Sample boundary fixture

This sample comes from the broken implementation of a controlled reproducer.

Boundary fixture	Actual	Expected	Outcome
unequal arm sizes weight the pooled rate	`[-2.98032, 0.00144]`	`[-2.98032, 0.002879]`	Failed

MEMBER ARCHIVE

The complete case is available to members.

This record includes three runnable implementations, regression fixtures, execution results, and source hashes.

Member access is invitation-based. Sign in with your invited account to inspect the sources.