FAILURE MAP
← Case archive

FA-74741 / Experiment statistics / Open access

O'Brien-Fleming type alpha spending: Repeated information fractions are accepted · case 01

A duplicated look slips through validation and reports zero alpha spent.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

Validation rejects only decreasing fractions.

VERIFIED REPAIR

Require strictly increasing fractions.

Unsuccessful approach: Rejecting only nonpositive fractions accepts decreasing schedules.

Case contract

Cumulative alpha spent at information fraction t is 2 - 2 Phi(z_{1-alpha/2} / sqrt(t)) (t capped at 1). Information fractions must strictly increase from 0, otherwise return None. Return the incremental alpha spent at each look rounded to 6.

Why this case matters

Spending functions let teams peek at experiments without inflating the false positive rate.

1 / The failure

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json
import math
import statistics
N = 1
observations = []
def solve(info_fractions, alpha):
    nd = statistics.NormalDist()
    z = nd.inv_cdf(1 - alpha / 2)
    out = []
    prev = 0.0
    last_t = 0.0
    for t in info_fractions:
        if t < last_t:
            return None
        t = min(t, 1.0)
        spent = 2 - 2 * nd.cdf(z / math.sqrt(t))
        out.append(round(spent - prev, 6))
        prev = spent
        last_t = t
    return out
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('four equally spaced looks', [[0.25, 0.5, 0.75, 1.0], 0.05], [8.9e-05, 0.005486, 0.018051, 0.026375]),
  ('single final look spends all alpha', [[1.0], 0.05], [0.05]),
  ('two looks at stricter alpha', [[0.5, 1.0], 0.01], [0.00027, 0.00973]),
  ('repeated fraction is invalid', [[0.5, 0.5], 0.05], None),
  ('decreasing fractions are invalid', [[0.6, 0.4], 0.05], None),
  ('look schedule sample 1', [[0.6, 0.8, 1.0], 0.01], [0.000883, 0.003095, 0.006022]),
  ('look schedule sample 2', [[0.2, 0.4, 0.5, 1.0], 0.1], [0.000235, 0.009067, 0.010707, 0.079991]),
  ('look schedule sample 3', [[0.25, 0.4, 0.6, 0.75], 0.1], [0.001003, 0.008299, 0.02441, 0.023811])],
 [('single final look spends all alpha', [[1.0], 0.05], [0.05]),
  ('two looks at stricter alpha', [[0.5, 1.0], 0.01], [0.00027, 0.00973]),
  ('repeated fraction is invalid', [[0.5, 0.5], 0.05], None),
  ('decreasing fractions are invalid', [[0.6, 0.4], 0.05], None),
  ('overrun beyond full information', [[0.8, 1.2], 0.05], [0.02843, 0.02157]),
  ('look schedule sample 6', [[0.2], 0.1], [0.000235]),
  ('look schedule sample 31', [[0.2, 0.4, 0.6, 0.6], 0.05], None),
  ('look schedule sample 35', [[0.2, 0.5, 0.75, 1.0, 1.0], 0.05], None)],
 [('four equally spaced looks', [[0.25, 0.5, 0.75, 1.0], 0.05], [8.9e-05, 0.005486, 0.018051, 0.026375]),
  ('two looks at stricter alpha', [[0.5, 1.0], 0.01], [0.00027, 0.00973]),
  ('repeated fraction is invalid', [[0.5, 0.5], 0.05], None),
  ('decreasing fractions are invalid', [[0.6, 0.4], 0.05], None),
  ('overrun beyond full information', [[0.8, 1.2], 0.05], [0.02843, 0.02157]),
  ('look schedule sample 11', [[0.2, 0.25, 0.8], 0.01], [0.0, 0.0, 0.003978]),
  ('look schedule sample 12', [[0.5, 0.6, 0.6], 0.01], None),
  ('look schedule sample 13', [[0.25, 0.5, 0.8, 1.0], 0.1], [0.001003, 0.019006, 0.045906, 0.034085])],
 [('four equally spaced looks', [[0.25, 0.5, 0.75, 1.0], 0.05], [8.9e-05, 0.005486, 0.018051, 0.026375]),
  ('single final look spends all alpha', [[1.0], 0.05], [0.05]),
  ('repeated fraction is invalid', [[0.5, 0.5], 0.05], None),
  ('decreasing fractions are invalid', [[0.6, 0.4], 0.05], None),
  ('overrun beyond full information', [[0.8, 1.2], 0.05], [0.02843, 0.02157]),
  ('look schedule sample 16', [[0.4, 0.5, 0.5], 0.1], None),
  ('look schedule sample 35', [[0.2, 0.5, 0.75, 1.0, 1.0], 0.05], None),
  ('look schedule sample 38', [[0.25, 0.8, 0.8], 0.05], None)],
 [('four equally spaced looks', [[0.25, 0.5, 0.75, 1.0], 0.05], [8.9e-05, 0.005486, 0.018051, 0.026375]),
  ('single final look spends all alpha', [[1.0], 0.05], [0.05]),
  ('two looks at stricter alpha', [[0.5, 1.0], 0.01], [0.00027, 0.00973]),
  ('decreasing fractions are invalid', [[0.6, 0.4], 0.05], None),
  ('overrun beyond full information', [[0.8, 1.2], 0.05], [0.02843, 0.02157]),
  ('look schedule sample 9', [[0.4, 0.6, 0.75, 1.0, 1.0], 0.1], None),
  ('look schedule sample 16', [[0.4, 0.5, 0.5], 0.1], None),
  ('look schedule sample 21', [[0.2, 0.4, 0.8], 0.01], [0.0, 4.6e-05, 0.003932])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
four equally spaced looks[8.9e-05, 0.005486, 0.018051, 0.026375][8.9e-05, 0.005486, 0.018051, 0.026375]Passed
single final look spends all alpha[0.05][0.05]Passed
two looks at stricter alpha[0.00027, 0.00973][0.00027, 0.00973]Passed
repeated fraction is invalid[0.005575, 0.0]NoneFailed
decreasing fractions are invalidNoneNonePassed
look schedule sample 1[0.000883, 0.003095, 0.006022][0.000883, 0.003095, 0.006022]Passed
look schedule sample 2[0.000235, 0.009067, 0.010707, 0.079991][0.000235, 0.009067, 0.010707, 0.079991]Passed
look schedule sample 3[0.001003, 0.008299, 0.02441, 0.023811][0.001003, 0.008299, 0.02441, 0.023811]Passed

SHA-256 / 909369fe8acfc06aca2957c49e129ee645f4b32283a34e347964cddc52ba26fc

2 / The unsuccessful fix

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json
import math
import statistics
N = 1
observations = []
def solve(info_fractions, alpha):
    nd = statistics.NormalDist()
    z = nd.inv_cdf(1 - alpha / 2)
    out = []
    prev = 0.0
    last_t = 0.0
    for t in info_fractions:
        if t <= 0:
            return None
        t = min(t, 1.0)
        spent = 2 - 2 * nd.cdf(z / math.sqrt(t))
        out.append(round(spent - prev, 6))
        prev = spent
        last_t = t
    return out
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('four equally spaced looks', [[0.25, 0.5, 0.75, 1.0], 0.05], [8.9e-05, 0.005486, 0.018051, 0.026375]),
  ('single final look spends all alpha', [[1.0], 0.05], [0.05]),
  ('two looks at stricter alpha', [[0.5, 1.0], 0.01], [0.00027, 0.00973]),
  ('repeated fraction is invalid', [[0.5, 0.5], 0.05], None),
  ('decreasing fractions are invalid', [[0.6, 0.4], 0.05], None),
  ('look schedule sample 1', [[0.6, 0.8, 1.0], 0.01], [0.000883, 0.003095, 0.006022]),
  ('look schedule sample 2', [[0.2, 0.4, 0.5, 1.0], 0.1], [0.000235, 0.009067, 0.010707, 0.079991]),
  ('look schedule sample 3', [[0.25, 0.4, 0.6, 0.75], 0.1], [0.001003, 0.008299, 0.02441, 0.023811])],
 [('single final look spends all alpha', [[1.0], 0.05], [0.05]),
  ('two looks at stricter alpha', [[0.5, 1.0], 0.01], [0.00027, 0.00973]),
  ('repeated fraction is invalid', [[0.5, 0.5], 0.05], None),
  ('decreasing fractions are invalid', [[0.6, 0.4], 0.05], None),
  ('overrun beyond full information', [[0.8, 1.2], 0.05], [0.02843, 0.02157]),
  ('look schedule sample 6', [[0.2], 0.1], [0.000235]),
  ('look schedule sample 31', [[0.2, 0.4, 0.6, 0.6], 0.05], None),
  ('look schedule sample 35', [[0.2, 0.5, 0.75, 1.0, 1.0], 0.05], None)],
 [('four equally spaced looks', [[0.25, 0.5, 0.75, 1.0], 0.05], [8.9e-05, 0.005486, 0.018051, 0.026375]),
  ('two looks at stricter alpha', [[0.5, 1.0], 0.01], [0.00027, 0.00973]),
  ('repeated fraction is invalid', [[0.5, 0.5], 0.05], None),
  ('decreasing fractions are invalid', [[0.6, 0.4], 0.05], None),
  ('overrun beyond full information', [[0.8, 1.2], 0.05], [0.02843, 0.02157]),
  ('look schedule sample 11', [[0.2, 0.25, 0.8], 0.01], [0.0, 0.0, 0.003978]),
  ('look schedule sample 12', [[0.5, 0.6, 0.6], 0.01], None),
  ('look schedule sample 13', [[0.25, 0.5, 0.8, 1.0], 0.1], [0.001003, 0.019006, 0.045906, 0.034085])],
 [('four equally spaced looks', [[0.25, 0.5, 0.75, 1.0], 0.05], [8.9e-05, 0.005486, 0.018051, 0.026375]),
  ('single final look spends all alpha', [[1.0], 0.05], [0.05]),
  ('repeated fraction is invalid', [[0.5, 0.5], 0.05], None),
  ('decreasing fractions are invalid', [[0.6, 0.4], 0.05], None),
  ('overrun beyond full information', [[0.8, 1.2], 0.05], [0.02843, 0.02157]),
  ('look schedule sample 16', [[0.4, 0.5, 0.5], 0.1], None),
  ('look schedule sample 35', [[0.2, 0.5, 0.75, 1.0, 1.0], 0.05], None),
  ('look schedule sample 38', [[0.25, 0.8, 0.8], 0.05], None)],
 [('four equally spaced looks', [[0.25, 0.5, 0.75, 1.0], 0.05], [8.9e-05, 0.005486, 0.018051, 0.026375]),
  ('single final look spends all alpha', [[1.0], 0.05], [0.05]),
  ('two looks at stricter alpha', [[0.5, 1.0], 0.01], [0.00027, 0.00973]),
  ('decreasing fractions are invalid', [[0.6, 0.4], 0.05], None),
  ('overrun beyond full information', [[0.8, 1.2], 0.05], [0.02843, 0.02157]),
  ('look schedule sample 9', [[0.4, 0.6, 0.75, 1.0, 1.0], 0.1], None),
  ('look schedule sample 16', [[0.4, 0.5, 0.5], 0.1], None),
  ('look schedule sample 21', [[0.2, 0.4, 0.8], 0.01], [0.0, 4.6e-05, 0.003932])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
four equally spaced looks[8.9e-05, 0.005486, 0.018051, 0.026375][8.9e-05, 0.005486, 0.018051, 0.026375]Passed
single final look spends all alpha[0.05][0.05]Passed
two looks at stricter alpha[0.00027, 0.00973][0.00027, 0.00973]Passed
repeated fraction is invalid[0.005575, 0.0]NoneFailed
decreasing fractions are invalid[0.011396, -0.009455]NoneFailed
look schedule sample 1[0.000883, 0.003095, 0.006022][0.000883, 0.003095, 0.006022]Passed
look schedule sample 2[0.000235, 0.009067, 0.010707, 0.079991][0.000235, 0.009067, 0.010707, 0.079991]Passed
look schedule sample 3[0.001003, 0.008299, 0.02441, 0.023811][0.001003, 0.008299, 0.02441, 0.023811]Passed

SHA-256 / 2cd4aee30d55480780e4c3d47b646c32f9507ac0604d9579d1d6b98db020af67

3 / The verified repair

Exit 0
"""Failure Map reference implementation. Python standard library only."""
import json
import math
import statistics
N = 1
observations = []
def solve(info_fractions, alpha):
    nd = statistics.NormalDist()
    z = nd.inv_cdf(1 - alpha / 2)
    out = []
    prev = 0.0
    last_t = 0.0
    for t in info_fractions:
        if t <= last_t:
            return None
        t = min(t, 1.0)
        spent = 2 - 2 * nd.cdf(z / math.sqrt(t))
        out.append(round(spent - prev, 6))
        prev = spent
        last_t = t
    return out
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('four equally spaced looks', [[0.25, 0.5, 0.75, 1.0], 0.05], [8.9e-05, 0.005486, 0.018051, 0.026375]),
  ('single final look spends all alpha', [[1.0], 0.05], [0.05]),
  ('two looks at stricter alpha', [[0.5, 1.0], 0.01], [0.00027, 0.00973]),
  ('repeated fraction is invalid', [[0.5, 0.5], 0.05], None),
  ('decreasing fractions are invalid', [[0.6, 0.4], 0.05], None),
  ('look schedule sample 1', [[0.6, 0.8, 1.0], 0.01], [0.000883, 0.003095, 0.006022]),
  ('look schedule sample 2', [[0.2, 0.4, 0.5, 1.0], 0.1], [0.000235, 0.009067, 0.010707, 0.079991]),
  ('look schedule sample 3', [[0.25, 0.4, 0.6, 0.75], 0.1], [0.001003, 0.008299, 0.02441, 0.023811])],
 [('single final look spends all alpha', [[1.0], 0.05], [0.05]),
  ('two looks at stricter alpha', [[0.5, 1.0], 0.01], [0.00027, 0.00973]),
  ('repeated fraction is invalid', [[0.5, 0.5], 0.05], None),
  ('decreasing fractions are invalid', [[0.6, 0.4], 0.05], None),
  ('overrun beyond full information', [[0.8, 1.2], 0.05], [0.02843, 0.02157]),
  ('look schedule sample 6', [[0.2], 0.1], [0.000235]),
  ('look schedule sample 31', [[0.2, 0.4, 0.6, 0.6], 0.05], None),
  ('look schedule sample 35', [[0.2, 0.5, 0.75, 1.0, 1.0], 0.05], None)],
 [('four equally spaced looks', [[0.25, 0.5, 0.75, 1.0], 0.05], [8.9e-05, 0.005486, 0.018051, 0.026375]),
  ('two looks at stricter alpha', [[0.5, 1.0], 0.01], [0.00027, 0.00973]),
  ('repeated fraction is invalid', [[0.5, 0.5], 0.05], None),
  ('decreasing fractions are invalid', [[0.6, 0.4], 0.05], None),
  ('overrun beyond full information', [[0.8, 1.2], 0.05], [0.02843, 0.02157]),
  ('look schedule sample 11', [[0.2, 0.25, 0.8], 0.01], [0.0, 0.0, 0.003978]),
  ('look schedule sample 12', [[0.5, 0.6, 0.6], 0.01], None),
  ('look schedule sample 13', [[0.25, 0.5, 0.8, 1.0], 0.1], [0.001003, 0.019006, 0.045906, 0.034085])],
 [('four equally spaced looks', [[0.25, 0.5, 0.75, 1.0], 0.05], [8.9e-05, 0.005486, 0.018051, 0.026375]),
  ('single final look spends all alpha', [[1.0], 0.05], [0.05]),
  ('repeated fraction is invalid', [[0.5, 0.5], 0.05], None),
  ('decreasing fractions are invalid', [[0.6, 0.4], 0.05], None),
  ('overrun beyond full information', [[0.8, 1.2], 0.05], [0.02843, 0.02157]),
  ('look schedule sample 16', [[0.4, 0.5, 0.5], 0.1], None),
  ('look schedule sample 35', [[0.2, 0.5, 0.75, 1.0, 1.0], 0.05], None),
  ('look schedule sample 38', [[0.25, 0.8, 0.8], 0.05], None)],
 [('four equally spaced looks', [[0.25, 0.5, 0.75, 1.0], 0.05], [8.9e-05, 0.005486, 0.018051, 0.026375]),
  ('single final look spends all alpha', [[1.0], 0.05], [0.05]),
  ('two looks at stricter alpha', [[0.5, 1.0], 0.01], [0.00027, 0.00973]),
  ('decreasing fractions are invalid', [[0.6, 0.4], 0.05], None),
  ('overrun beyond full information', [[0.8, 1.2], 0.05], [0.02843, 0.02157]),
  ('look schedule sample 9', [[0.4, 0.6, 0.75, 1.0, 1.0], 0.1], None),
  ('look schedule sample 16', [[0.4, 0.5, 0.5], 0.1], None),
  ('look schedule sample 21', [[0.2, 0.4, 0.8], 0.01], [0.0, 4.6e-05, 0.003932])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
four equally spaced looks[8.9e-05, 0.005486, 0.018051, 0.026375][8.9e-05, 0.005486, 0.018051, 0.026375]Passed
single final look spends all alpha[0.05][0.05]Passed
two looks at stricter alpha[0.00027, 0.00973][0.00027, 0.00973]Passed
repeated fraction is invalidNoneNonePassed
decreasing fractions are invalidNoneNonePassed
look schedule sample 1[0.000883, 0.003095, 0.006022][0.000883, 0.003095, 0.006022]Passed
look schedule sample 2[0.000235, 0.009067, 0.010707, 0.079991][0.000235, 0.009067, 0.010707, 0.079991]Passed
look schedule sample 3[0.001003, 0.008299, 0.02441, 0.023811][0.001003, 0.008299, 0.02441, 0.023811]Passed

SHA-256 / d875b3f64a52caa66ba4fb3989038665bcd6e90cc36c5af88bdc28390b0843c5

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:59.614197+00:00.

Case digest / 072cc6909c40bfd32ed59ef029984a3886801726ce0e10075ee9d1ff2003922a