FA-74496 / Experiment statistics / Open access

Per-arm sample size for conversion tests: Alpha is not split across tails · case 01

Planned sample sizes are too small for the two-sided test that is actually run.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

z_alpha is the 1 - alpha quantile.

THE FAILURE

z_alpha is the 1 - alpha quantile.

Unsuccessful approach: Using the alpha / 2 quantile gives a negative z and a far too small sample.

Case contract

p2 = p_base (1 + mde) when relative else p_base + mde; both rates must lie in (0, 1) and differ (else None). n per arm = ceil((z_{1-alpha/2} + z_power)^2 (p1(1-p1) + p2(1-p2)) / (p2 - p1)^2) using statistics.NormalDist quantiles.

Why this case matters

Underpowered experiments waste traffic; overestimates delay launches.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json
import math
import statistics
N = 1
observations = []
def solve(p_base, mde, alpha, power, relative):
    nd = statistics.NormalDist()
    p1 = p_base
    p2 = p_base * (1 + mde) if relative else p_base + mde
    if not (0 < p1 < 1 and 0 < p2 < 1) or p1 == p2:
        return None
    za = nd.inv_cdf(1 - alpha)
    zb = nd.inv_cdf(power)
    n = (za + zb) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) / (p2 - p1) ** 2
    return math.ceil(n)
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('planning sample 1', [0.02, 0.1, 0.01, 0.8, False], 147),
  ('planning sample 2', [0.2, 0.2, 0.05, 0.9, True], 2249),
  ('planning sample 3', [0.05, 0.01, 0.1, 0.9, True], 3269669)],
 [('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 4', [0.1, 0.01, 0.1, 0.9, False], 16092),
  ('planning sample 6', [0.2, 0.1, 0.01, 0.9, True], 12336),
  ('planning sample 7', [0.5, 0.05, 0.1, 0.9, False], 1705)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 11', [0.02, 0.05, 0.1, 0.9, False], 291),
  ('planning sample 12', [0.5, 0.2, 0.01, 0.8, True], 573),
  ('planning sample 13', [0.5, 0.01, 0.05, 0.9, False], 52527)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 16', [0.2, 0.05, 0.1, 0.8, True], 20149),
  ('planning sample 17', [0.1, 0.02, 0.1, 0.8, False], 3024),
  ('planning sample 18', [0.2, 0.1, 0.1, 0.9, True], 7100)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 21', [0.02, 0.1, 0.01, 0.9, False], 187),
  ('planning sample 22', [0.2, 0.05, 0.1, 0.9, True], 27910),
  ('planning sample 25', [0.1, 0.2, 0.01, 0.9, True], 7277)]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
ten percent relative lift on five percent	`24601`	`31231`	Failed
one point absolute lift	`11618`	`14749`	Failed
higher power needs more users	`27910`	`34244`	Failed
stricter alpha	`12535`	`14588`	Failed
lift beyond one is invalid	`None`	`None`	Passed
planning sample 1	`126`	`147`	Failed
planning sample 2	`1833`	`2249`	Failed
planning sample 3	`2508228`	`3269669`	Failed

SHA-256 / 23cbc79fc29b5b27d2a8715da6733080ada07ea94a95ee731556320a1f2b38ac

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json
import math
import statistics
N = 1
observations = []
def solve(p_base, mde, alpha, power, relative):
    nd = statistics.NormalDist()
    p1 = p_base
    p2 = p_base * (1 + mde) if relative else p_base + mde
    if not (0 < p1 < 1 and 0 < p2 < 1) or p1 == p2:
        return None
    za = nd.inv_cdf(alpha / 2)
    zb = nd.inv_cdf(power)
    n = (za + zb) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) / (p2 - p1) ** 2
    return math.ceil(n)
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('planning sample 1', [0.02, 0.1, 0.01, 0.8, False], 147),
  ('planning sample 2', [0.2, 0.2, 0.05, 0.9, True], 2249),
  ('planning sample 3', [0.05, 0.01, 0.1, 0.9, True], 3269669)],
 [('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 4', [0.1, 0.01, 0.1, 0.9, False], 16092),
  ('planning sample 6', [0.2, 0.1, 0.01, 0.9, True], 12336),
  ('planning sample 7', [0.5, 0.05, 0.1, 0.9, False], 1705)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 11', [0.02, 0.05, 0.1, 0.9, False], 291),
  ('planning sample 12', [0.5, 0.2, 0.01, 0.8, True], 573),
  ('planning sample 13', [0.5, 0.01, 0.05, 0.9, False], 52527)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 16', [0.2, 0.05, 0.1, 0.8, True], 20149),
  ('planning sample 17', [0.1, 0.02, 0.1, 0.8, False], 3024),
  ('planning sample 18', [0.2, 0.1, 0.1, 0.9, True], 7100)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 21', [0.02, 0.1, 0.01, 0.9, False], 187),
  ('planning sample 22', [0.2, 0.05, 0.1, 0.9, True], 27910),
  ('planning sample 25', [0.1, 0.2, 0.01, 0.9, True], 7277)]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
ten percent relative lift on five percent	`4977`	`31231`	Failed
one point absolute lift	`2351`	`14749`	Failed
higher power needs more users	`1500`	`34244`	Failed
stricter alpha	`3757`	`14588`	Failed
lift beyond one is invalid	`None`	`None`	Passed
planning sample 1	`38`	`147`	Failed
planning sample 2	`99`	`2249`	Failed
planning sample 3	`50394`	`3269669`	Failed

SHA-256 / 07c6b40f2d53f152d0a6673501204f1d51767ac8e20800c8c988b619e623b98e

HELD IN THE MEMBER ARCHIVE

The verified repair and its recorded checks are member-only.

This mechanism has 8 recorded checks per implementation. The open-access tier publishes the failure and the unsuccessful fix; the repaired source that passes every check, and the observations that prove it, are available to members.

Every case sharing this mechanism uses the same contract and the same repair, so this one record is held back for all of them.

Member access is invitation-based. Sign in with your invited account to inspect the repair.

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:57.538328+00:00.

Case digest / e97692e93eb30fe09739bf618a3f84155c21ea0034cc180810dbb0d9c9841cf4