FA-74511 / Experiment statistics / Open access

Per-arm sample size for conversion tests: Only the baseline variance is used · case 01

Plans ignore that the treatment rate has a different variance.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The variance sum is replaced by twice the baseline variance.

VERIFIED REPAIR

Sum p1(1-p1) and p2(1-p2).

Unsuccessful approach: A single pooled variance term is half of what two arms contribute.

Case contract

p2 = p_base (1 + mde) when relative else p_base + mde; both rates must lie in (0, 1) and differ (else None). n per arm = ceil((z_{1-alpha/2} + z_power)^2 (p1(1-p1) + p2(1-p2)) / (p2 - p1)^2) using statistics.NormalDist quantiles.

Why this case matters

Underpowered experiments waste traffic; overestimates delay launches.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json
import math
import statistics
N = 1
observations = []
def solve(p_base, mde, alpha, power, relative):
    nd = statistics.NormalDist()
    p1 = p_base
    p2 = p_base * (1 + mde) if relative else p_base + mde
    if not (0 < p1 < 1 and 0 < p2 < 1) or p1 == p2:
        return None
    za = nd.inv_cdf(1 - alpha / 2)
    zb = nd.inv_cdf(power)
    n = (za + zb) ** 2 * 2 * p1 * (1 - p1) / (p2 - p1) ** 2
    return math.ceil(n)
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('planning sample 1', [0.02, 0.1, 0.01, 0.8, False], 147),
  ('planning sample 2', [0.2, 0.2, 0.05, 0.9, True], 2249),
  ('planning sample 3', [0.05, 0.01, 0.1, 0.9, True], 3269669)],
 [('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 4', [0.1, 0.01, 0.1, 0.9, False], 16092),
  ('planning sample 6', [0.2, 0.1, 0.01, 0.9, True], 12336),
  ('planning sample 7', [0.5, 0.05, 0.1, 0.9, False], 1705)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 11', [0.02, 0.05, 0.1, 0.9, False], 291),
  ('planning sample 12', [0.5, 0.2, 0.01, 0.8, True], 573),
  ('planning sample 13', [0.5, 0.01, 0.05, 0.9, False], 52527)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 16', [0.2, 0.05, 0.1, 0.8, True], 20149),
  ('planning sample 17', [0.1, 0.02, 0.1, 0.8, False], 3024),
  ('planning sample 18', [0.2, 0.1, 0.1, 0.9, True], 7100)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 21', [0.02, 0.1, 0.01, 0.9, False], 187),
  ('planning sample 22', [0.2, 0.05, 0.1, 0.9, True], 27910),
  ('planning sample 25', [0.1, 0.2, 0.01, 0.9, True], 7277)]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
ten percent relative lift on five percent	`29826`	`31231`	Failed
one point absolute lift	`14128`	`14749`	Failed
higher power needs more users	`33624`	`34244`	Failed
stricter alpha	`14599`	`14588`	Failed
lift beyond one is invalid	`None`	`None`	Passed
planning sample 1	`46`	`147`	Failed
planning sample 2	`2102`	`2249`	Failed
planning sample 3	`3254262`	`3269669`	Failed

SHA-256 / df362add144991f6e053dc520314f5b26ddcd7cfb5a88350cb9a8d883a09aad0

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json
import math
import statistics
N = 1
observations = []
def solve(p_base, mde, alpha, power, relative):
    nd = statistics.NormalDist()
    p1 = p_base
    p2 = p_base * (1 + mde) if relative else p_base + mde
    if not (0 < p1 < 1 and 0 < p2 < 1) or p1 == p2:
        return None
    za = nd.inv_cdf(1 - alpha / 2)
    zb = nd.inv_cdf(power)
    n = (za + zb) ** 2 * ((p1 + p2) / 2 * (1 - (p1 + p2) / 2)) / (p2 - p1) ** 2
    return math.ceil(n)
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('planning sample 1', [0.02, 0.1, 0.01, 0.8, False], 147),
  ('planning sample 2', [0.2, 0.2, 0.05, 0.9, True], 2249),
  ('planning sample 3', [0.05, 0.01, 0.1, 0.9, True], 3269669)],
 [('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 4', [0.1, 0.01, 0.1, 0.9, False], 16092),
  ('planning sample 6', [0.2, 0.1, 0.01, 0.9, True], 12336),
  ('planning sample 7', [0.5, 0.05, 0.1, 0.9, False], 1705)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 11', [0.02, 0.05, 0.1, 0.9, False], 291),
  ('planning sample 12', [0.5, 0.2, 0.01, 0.8, True], 573),
  ('planning sample 13', [0.5, 0.01, 0.05, 0.9, False], 52527)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 16', [0.2, 0.05, 0.1, 0.8, True], 20149),
  ('planning sample 17', [0.1, 0.02, 0.1, 0.8, False], 3024),
  ('planning sample 18', [0.2, 0.1, 0.1, 0.9, True], 7100)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 21', [0.02, 0.1, 0.01, 0.9, False], 187),
  ('planning sample 22', [0.2, 0.05, 0.1, 0.9, True], 27910),
  ('planning sample 25', [0.1, 0.2, 0.01, 0.9, True], 7277)]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
ten percent relative lift on five percent	`15618`	`31231`	Failed
one point absolute lift	`7376`	`14749`	Failed
higher power needs more users	`17125`	`34244`	Failed
stricter alpha	`7297`	`14588`	Failed
lift beyond one is invalid	`None`	`None`	Passed
planning sample 1	`77`	`147`	Failed
planning sample 2	`1127`	`2249`	Failed
planning sample 3	`1634837`	`3269669`	Failed

SHA-256 / c40c4f35c2140f51af16565e0606b7c3c13f816792b8cf80ac8bc2596b7161ae

3 / The verified repair

Exit 0

"""Failure Map reference implementation. Python standard library only."""
import json
import math
import statistics
N = 1
observations = []
def solve(p_base, mde, alpha, power, relative):
    nd = statistics.NormalDist()
    p1 = p_base
    p2 = p_base * (1 + mde) if relative else p_base + mde
    if not (0 < p1 < 1 and 0 < p2 < 1) or p1 == p2:
        return None
    za = nd.inv_cdf(1 - alpha / 2)
    zb = nd.inv_cdf(power)
    n = (za + zb) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) / (p2 - p1) ** 2
    return math.ceil(n)
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('planning sample 1', [0.02, 0.1, 0.01, 0.8, False], 147),
  ('planning sample 2', [0.2, 0.2, 0.05, 0.9, True], 2249),
  ('planning sample 3', [0.05, 0.01, 0.1, 0.9, True], 3269669)],
 [('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 4', [0.1, 0.01, 0.1, 0.9, False], 16092),
  ('planning sample 6', [0.2, 0.1, 0.01, 0.9, True], 12336),
  ('planning sample 7', [0.5, 0.05, 0.1, 0.9, False], 1705)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 11', [0.02, 0.05, 0.1, 0.9, False], 291),
  ('planning sample 12', [0.5, 0.2, 0.01, 0.8, True], 573),
  ('planning sample 13', [0.5, 0.01, 0.05, 0.9, False], 52527)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 16', [0.2, 0.05, 0.1, 0.8, True], 20149),
  ('planning sample 17', [0.1, 0.02, 0.1, 0.8, False], 3024),
  ('planning sample 18', [0.2, 0.1, 0.1, 0.9, True], 7100)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 21', [0.02, 0.1, 0.01, 0.9, False], 187),
  ('planning sample 22', [0.2, 0.05, 0.1, 0.9, True], 27910),
  ('planning sample 25', [0.1, 0.2, 0.01, 0.9, True], 7277)]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
ten percent relative lift on five percent	`31231`	`31231`	Passed
one point absolute lift	`14749`	`14749`	Passed
higher power needs more users	`34244`	`34244`	Passed
stricter alpha	`14588`	`14588`	Passed
lift beyond one is invalid	`None`	`None`	Passed
planning sample 1	`147`	`147`	Passed
planning sample 2	`2249`	`2249`	Passed
planning sample 3	`3269669`	`3269669`	Passed

SHA-256 / ef9860d0ebe995a1d59eafd0da5b8a020c7bb398b89a79afc810c6eb4b26ae11

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:57.488529+00:00.

Case digest / c38ba79dcc1bd194bbb4e9587dab6ed8de0f3002663d19f4cf3ed10b6cb8cfd6