FA-74516 / Experiment statistics / Open access

Per-arm sample size for conversion tests: The power quantile is taken from the wrong tail · case 01

Higher requested power produces smaller planned samples.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

z_beta is the (1 - power) quantile.

VERIFIED REPAIR

Use the power quantile directly.

Unsuccessful approach: Halving the power probability also lands in the lower tail.

Case contract

p2 = p_base (1 + mde) when relative else p_base + mde; both rates must lie in (0, 1) and differ (else None). n per arm = ceil((z_{1-alpha/2} + z_power)^2 (p1(1-p1) + p2(1-p2)) / (p2 - p1)^2) using statistics.NormalDist quantiles.

Why this case matters

Underpowered experiments waste traffic; overestimates delay launches.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json
import math
import statistics
N = 1
observations = []
def solve(p_base, mde, alpha, power, relative):
    nd = statistics.NormalDist()
    p1 = p_base
    p2 = p_base * (1 + mde) if relative else p_base + mde
    if not (0 < p1 < 1 and 0 < p2 < 1) or p1 == p2:
        return None
    za = nd.inv_cdf(1 - alpha / 2)
    zb = nd.inv_cdf(1 - power)
    n = (za + zb) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) / (p2 - p1) ** 2
    return math.ceil(n)
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('planning sample 1', [0.02, 0.1, 0.01, 0.8, False], 147),
  ('planning sample 2', [0.2, 0.2, 0.05, 0.9, True], 2249),
  ('planning sample 3', [0.05, 0.01, 0.1, 0.9, True], 3269669)],
 [('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 4', [0.1, 0.01, 0.1, 0.9, False], 16092),
  ('planning sample 6', [0.2, 0.1, 0.01, 0.9, True], 12336),
  ('planning sample 7', [0.5, 0.05, 0.1, 0.9, False], 1705)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 11', [0.02, 0.05, 0.1, 0.9, False], 291),
  ('planning sample 12', [0.5, 0.2, 0.01, 0.8, True], 573),
  ('planning sample 13', [0.5, 0.01, 0.05, 0.9, False], 52527)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 16', [0.2, 0.05, 0.1, 0.8, True], 20149),
  ('planning sample 17', [0.1, 0.02, 0.1, 0.8, False], 3024),
  ('planning sample 18', [0.2, 0.1, 0.1, 0.9, True], 7100)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 21', [0.02, 0.1, 0.01, 0.9, False], 187),
  ('planning sample 22', [0.2, 0.05, 0.1, 0.9, True], 27910),
  ('planning sample 25', [0.1, 0.2, 0.01, 0.9, True], 7277)]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
ten percent relative lift on five percent	`4977`	`31231`	Failed
one point absolute lift	`2351`	`14749`	Failed
higher power needs more users	`1500`	`34244`	Failed
stricter alpha	`3757`	`14588`	Failed
lift beyond one is invalid	`None`	`None`	Passed
planning sample 1	`38`	`147`	Failed
planning sample 2	`99`	`2249`	Failed
planning sample 3	`50394`	`3269669`	Failed

SHA-256 / 6789da4fbaf22066f4c597546647656ac83a244a0046bff6fabf9fe6950ba0d0

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json
import math
import statistics
N = 1
observations = []
def solve(p_base, mde, alpha, power, relative):
    nd = statistics.NormalDist()
    p1 = p_base
    p2 = p_base * (1 + mde) if relative else p_base + mde
    if not (0 < p1 < 1 and 0 < p2 < 1) or p1 == p2:
        return None
    za = nd.inv_cdf(1 - alpha / 2)
    zb = nd.inv_cdf(power / 2)
    n = (za + zb) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) / (p2 - p1) ** 2
    return math.ceil(n)
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('planning sample 1', [0.02, 0.1, 0.01, 0.8, False], 147),
  ('planning sample 2', [0.2, 0.2, 0.05, 0.9, True], 2249),
  ('planning sample 3', [0.05, 0.01, 0.1, 0.9, True], 3269669)],
 [('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 4', [0.1, 0.01, 0.1, 0.9, False], 16092),
  ('planning sample 6', [0.2, 0.1, 0.01, 0.9, True], 12336),
  ('planning sample 7', [0.5, 0.05, 0.1, 0.9, False], 1705)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 11', [0.02, 0.05, 0.1, 0.9, False], 291),
  ('planning sample 12', [0.5, 0.2, 0.01, 0.8, True], 573),
  ('planning sample 13', [0.5, 0.01, 0.05, 0.9, False], 52527)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 16', [0.2, 0.05, 0.1, 0.8, True], 20149),
  ('planning sample 17', [0.1, 0.02, 0.1, 0.8, False], 3024),
  ('planning sample 18', [0.2, 0.1, 0.1, 0.9, True], 7100)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 21', [0.02, 0.1, 0.01, 0.9, False], 187),
  ('planning sample 22', [0.2, 0.05, 0.1, 0.9, True], 27910),
  ('planning sample 25', [0.1, 0.2, 0.01, 0.9, True], 7277)]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
ten percent relative lift on five percent	`11590`	`31231`	Failed
one point absolute lift	`5473`	`14749`	Failed
higher power needs more users	`10966`	`34244`	Failed
stricter alpha	`6738`	`14588`	Failed
lift beyond one is invalid	`None`	`None`	Passed
planning sample 1	`68`	`147`	Failed
planning sample 2	`721`	`2249`	Failed
planning sample 3	`881172`	`3269669`	Failed

SHA-256 / 0d0d35beb694e4616dcd19e54e186cdfa591b8b977e8654d9a9fe210e8bfe937

3 / The verified repair

Exit 0

"""Failure Map reference implementation. Python standard library only."""
import json
import math
import statistics
N = 1
observations = []
def solve(p_base, mde, alpha, power, relative):
    nd = statistics.NormalDist()
    p1 = p_base
    p2 = p_base * (1 + mde) if relative else p_base + mde
    if not (0 < p1 < 1 and 0 < p2 < 1) or p1 == p2:
        return None
    za = nd.inv_cdf(1 - alpha / 2)
    zb = nd.inv_cdf(power)
    n = (za + zb) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) / (p2 - p1) ** 2
    return math.ceil(n)
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('planning sample 1', [0.02, 0.1, 0.01, 0.8, False], 147),
  ('planning sample 2', [0.2, 0.2, 0.05, 0.9, True], 2249),
  ('planning sample 3', [0.05, 0.01, 0.1, 0.9, True], 3269669)],
 [('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 4', [0.1, 0.01, 0.1, 0.9, False], 16092),
  ('planning sample 6', [0.2, 0.1, 0.01, 0.9, True], 12336),
  ('planning sample 7', [0.5, 0.05, 0.1, 0.9, False], 1705)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 11', [0.02, 0.05, 0.1, 0.9, False], 291),
  ('planning sample 12', [0.5, 0.2, 0.01, 0.8, True], 573),
  ('planning sample 13', [0.5, 0.01, 0.05, 0.9, False], 52527)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 16', [0.2, 0.05, 0.1, 0.8, True], 20149),
  ('planning sample 17', [0.1, 0.02, 0.1, 0.8, False], 3024),
  ('planning sample 18', [0.2, 0.1, 0.1, 0.9, True], 7100)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 21', [0.02, 0.1, 0.01, 0.9, False], 187),
  ('planning sample 22', [0.2, 0.05, 0.1, 0.9, True], 27910),
  ('planning sample 25', [0.1, 0.2, 0.01, 0.9, True], 7277)]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
ten percent relative lift on five percent	`31231`	`31231`	Passed
one point absolute lift	`14749`	`14749`	Passed
higher power needs more users	`34244`	`34244`	Passed
stricter alpha	`14588`	`14588`	Passed
lift beyond one is invalid	`None`	`None`	Passed
planning sample 1	`147`	`147`	Passed
planning sample 2	`2249`	`2249`	Passed
planning sample 3	`3269669`	`3269669`	Passed

SHA-256 / ef9860d0ebe995a1d59eafd0da5b8a020c7bb398b89a79afc810c6eb4b26ae11

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:57.538328+00:00.

Case digest / d3c8300f48b725badc5099e9921f9d83ffcd31604e0cc777e425650c7b2dd7c3