FAILURE MAP
← Case archive

FA-74506 / Experiment statistics / Open access

Per-arm sample size for conversion tests: Sample size is rounded to nearest · case 01

Planned samples fall just short of the requested power.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The final size is round(n) instead of ceil(n).

VERIFIED REPAIR

Round the per-arm sample size up.

Unsuccessful approach: Flooring guarantees the shortfall that rounding only sometimes causes.

Case contract

p2 = p_base (1 + mde) when relative else p_base + mde; both rates must lie in (0, 1) and differ (else None). n per arm = ceil((z_{1-alpha/2} + z_power)^2 (p1(1-p1) + p2(1-p2)) / (p2 - p1)^2) using statistics.NormalDist quantiles.

Why this case matters

Underpowered experiments waste traffic; overestimates delay launches.

1 / The failure

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json
import math
import statistics
N = 1
observations = []
def solve(p_base, mde, alpha, power, relative):
    nd = statistics.NormalDist()
    p1 = p_base
    p2 = p_base * (1 + mde) if relative else p_base + mde
    if not (0 < p1 < 1 and 0 < p2 < 1) or p1 == p2:
        return None
    za = nd.inv_cdf(1 - alpha / 2)
    zb = nd.inv_cdf(power)
    n = (za + zb) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) / (p2 - p1) ** 2
    return round(n)
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('planning sample 1', [0.02, 0.1, 0.01, 0.8, False], 147),
  ('planning sample 2', [0.2, 0.2, 0.05, 0.9, True], 2249),
  ('planning sample 3', [0.05, 0.01, 0.1, 0.9, True], 3269669)],
 [('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 4', [0.1, 0.01, 0.1, 0.9, False], 16092),
  ('planning sample 6', [0.2, 0.1, 0.01, 0.9, True], 12336),
  ('planning sample 7', [0.5, 0.05, 0.1, 0.9, False], 1705)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 11', [0.02, 0.05, 0.1, 0.9, False], 291),
  ('planning sample 12', [0.5, 0.2, 0.01, 0.8, True], 573),
  ('planning sample 17', [0.1, 0.02, 0.1, 0.8, False], 3024)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 16', [0.2, 0.05, 0.1, 0.8, True], 20149),
  ('planning sample 18', [0.2, 0.1, 0.1, 0.9, True], 7100),
  ('planning sample 33', [0.5, 0.05, 0.1, 0.8, False], 1231)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 21', [0.02, 0.1, 0.01, 0.9, False], 187),
  ('planning sample 25', [0.1, 0.2, 0.01, 0.9, True], 7277),
  ('planning sample 45', [0.1, 0.02, 0.01, 0.8, True], 530214)]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
ten percent relative lift on five percent3123131231Passed
one point absolute lift1474814749Failed
higher power needs more users3424434244Passed
stricter alpha1458714588Failed
lift beyond one is invalidNoneNonePassed
planning sample 1146147Failed
planning sample 222492249Passed
planning sample 332696683269669Failed

SHA-256 / 977ce1761934a9e3e6258c5f204b5458194184da830471c8a7bf372fdb8e9e7b

2 / The unsuccessful fix

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json
import math
import statistics
N = 1
observations = []
def solve(p_base, mde, alpha, power, relative):
    nd = statistics.NormalDist()
    p1 = p_base
    p2 = p_base * (1 + mde) if relative else p_base + mde
    if not (0 < p1 < 1 and 0 < p2 < 1) or p1 == p2:
        return None
    za = nd.inv_cdf(1 - alpha / 2)
    zb = nd.inv_cdf(power)
    n = (za + zb) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) / (p2 - p1) ** 2
    return math.floor(n)
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('planning sample 1', [0.02, 0.1, 0.01, 0.8, False], 147),
  ('planning sample 2', [0.2, 0.2, 0.05, 0.9, True], 2249),
  ('planning sample 3', [0.05, 0.01, 0.1, 0.9, True], 3269669)],
 [('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 4', [0.1, 0.01, 0.1, 0.9, False], 16092),
  ('planning sample 6', [0.2, 0.1, 0.01, 0.9, True], 12336),
  ('planning sample 7', [0.5, 0.05, 0.1, 0.9, False], 1705)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 11', [0.02, 0.05, 0.1, 0.9, False], 291),
  ('planning sample 12', [0.5, 0.2, 0.01, 0.8, True], 573),
  ('planning sample 17', [0.1, 0.02, 0.1, 0.8, False], 3024)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 16', [0.2, 0.05, 0.1, 0.8, True], 20149),
  ('planning sample 18', [0.2, 0.1, 0.1, 0.9, True], 7100),
  ('planning sample 33', [0.5, 0.05, 0.1, 0.8, False], 1231)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 21', [0.02, 0.1, 0.01, 0.9, False], 187),
  ('planning sample 25', [0.1, 0.2, 0.01, 0.9, True], 7277),
  ('planning sample 45', [0.1, 0.02, 0.01, 0.8, True], 530214)]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
ten percent relative lift on five percent3123031231Failed
one point absolute lift1474814749Failed
higher power needs more users3424334244Failed
stricter alpha1458714588Failed
lift beyond one is invalidNoneNonePassed
planning sample 1146147Failed
planning sample 222482249Failed
planning sample 332696683269669Failed

SHA-256 / edab9a98c2d4ca336789820023ba2fcced10f5215bcd731cf8b7c190e7392f59

3 / The verified repair

Exit 0
"""Failure Map reference implementation. Python standard library only."""
import json
import math
import statistics
N = 1
observations = []
def solve(p_base, mde, alpha, power, relative):
    nd = statistics.NormalDist()
    p1 = p_base
    p2 = p_base * (1 + mde) if relative else p_base + mde
    if not (0 < p1 < 1 and 0 < p2 < 1) or p1 == p2:
        return None
    za = nd.inv_cdf(1 - alpha / 2)
    zb = nd.inv_cdf(power)
    n = (za + zb) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) / (p2 - p1) ** 2
    return math.ceil(n)
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('planning sample 1', [0.02, 0.1, 0.01, 0.8, False], 147),
  ('planning sample 2', [0.2, 0.2, 0.05, 0.9, True], 2249),
  ('planning sample 3', [0.05, 0.01, 0.1, 0.9, True], 3269669)],
 [('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 4', [0.1, 0.01, 0.1, 0.9, False], 16092),
  ('planning sample 6', [0.2, 0.1, 0.01, 0.9, True], 12336),
  ('planning sample 7', [0.5, 0.05, 0.1, 0.9, False], 1705)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 11', [0.02, 0.05, 0.1, 0.9, False], 291),
  ('planning sample 12', [0.5, 0.2, 0.01, 0.8, True], 573),
  ('planning sample 17', [0.1, 0.02, 0.1, 0.8, False], 3024)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('stricter alpha', [0.5, 0.02, 0.01, 0.8, False], 14588),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 16', [0.2, 0.05, 0.1, 0.8, True], 20149),
  ('planning sample 18', [0.2, 0.1, 0.1, 0.9, True], 7100),
  ('planning sample 33', [0.5, 0.05, 0.1, 0.8, False], 1231)],
 [('ten percent relative lift on five percent', [0.05, 0.1, 0.05, 0.8, True], 31231),
  ('one point absolute lift', [0.1, 0.01, 0.05, 0.8, False], 14749),
  ('higher power needs more users', [0.2, 0.05, 0.05, 0.9, True], 34244),
  ('lift beyond one is invalid', [0.5, 1.0, 0.05, 0.8, True], None),
  ('zero effect is invalid', [0.1, 0.0, 0.05, 0.8, False], None),
  ('planning sample 21', [0.02, 0.1, 0.01, 0.9, False], 187),
  ('planning sample 25', [0.1, 0.2, 0.01, 0.9, True], 7277),
  ('planning sample 45', [0.1, 0.02, 0.01, 0.8, True], 530214)]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
ten percent relative lift on five percent3123131231Passed
one point absolute lift1474914749Passed
higher power needs more users3424434244Passed
stricter alpha1458814588Passed
lift beyond one is invalidNoneNonePassed
planning sample 1147147Passed
planning sample 222492249Passed
planning sample 332696693269669Passed

SHA-256 / c4cb85c63db96ecaebb97c2e3c18f4788b9f8aef59be45c9e0d65c26e3c7f97d

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:57.333647+00:00.

Case digest / 0ab332eb890309e5ba930089193642cbeeed13358a82fafaca0dbf2a57e67efa