FAILURE MAP
← Case archive

FA-74751 / Experiment statistics / Open access

Pre-period balance check: Balance uses population variances · case 01

Tiny pre-period samples exaggerate imbalance.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

Variances divide by n instead of n - 1.

VERIFIED REPAIR

Use the n - 1 divisor.

Unsuccessful approach: Dividing by n + 1 shrinks the variance further.

Case contract

Standardised mean difference = (mean_t - mean_c) / sqrt((var_c + var_t) / 2) with sample (n - 1) variances; imbalance iff |SMD| > threshold. Zero pooled variance -> [0.0, False] when means are equal else [None, True]. Fewer than two values in an arm -> None. Return [round(smd, 6), imbalance].

Why this case matters

Pre-period imbalance warns that randomisation or logging went wrong before any effect is read.

1 / The failure

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(pre_c, pre_t, threshold):
    def moments(xs):
        n = len(xs)
        m = sum(xs) / n
        return m, sum((x - m) ** 2 for x in xs) / n
    if len(pre_c) < 2 or len(pre_t) < 2:
        return None
    mc, vc = moments(pre_c)
    mt, vt = moments(pre_t)
    pooled = math.sqrt((vc + vt) / 2)
    if pooled == 0:
        return [0.0, False] if mt == mc else [None, True]
    smd = (mt - mc) / pooled
    return [round(smd, 6), abs(smd) > threshold]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('pre-period sample 1', [[1, 3, 6], [10, 3], 0.5], [0.806505, True]),
  ('pre-period sample 2', [[5, 9, 3, 4], [3, 7], 0.1], [-0.091542, False]),
  ('pre-period sample 3', [[6, 8, 3, 8, 9], [6, 5, 4, 10], 0.25], [-0.21898, False])],
 [('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('pre-period sample 4', [[5, 7], [2, 10, 7, 9, 9], 0.5], [0.564534, True]),
  ('pre-period sample 6', [[8, 0], [8, 5, 6, 2], 0.1], [0.285831, True]),
  ('pre-period sample 7', [[7, 0, 8, 6, 7], [9, 4, 5, 2, 2], 0.5], [-0.393496, False])],
 [('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 11', [[8, 3, 4], [7, 3, 4], 0.1], [-0.140028, True]),
  ('pre-period sample 12', [[4, 3, 3], [4, 0], 0.1], [-0.653197, True]),
  ('pre-period sample 13', [[3, 5], [5, 7, 10, 7, 2], 0.1], [0.951143, True])],
 [('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 16', [[3, 5, 5, 2], [4, 11], 0.1], [1.025379, True]),
  ('pre-period sample 17', [[6, 1, 1, 4, 1], [7, 10, 8, 4], 0.25], [1.934982, True]),
  ('pre-period sample 19', [[3, 1, 3, 7], [5, 4, 9], 0.1], [0.968246, True])],
 [('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 21', [[3, 2, 5, 0], [0, 6], 0.5], [0.149626, False]),
  ('pre-period sample 22', [[5, 8, 9, 5], [11, 8, 5, 6], 0.5], [0.316228, False]),
  ('pre-period sample 26', [[6, 7, 1, 4, 9], [3, 4, 11, 6, 10], 0.25], [0.422116, True])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
treatment arm more variable[0.840168, True][0.685994, True]Failed
SMD exactly at the tolerance passes[0.612372, True][0.5, False]Failed
negative imbalance is flagged[-4.898979, True][-4.0, True]Failed
small arms[1.264911, True][0.894427, True]Failed
constant equal arms[0.0, False][0.0, False]Passed
pre-period sample 1[1.103421, True][0.806505, True]Failed
pre-period sample 2[-0.116642, True][-0.091542, False]Failed
pre-period sample 3[-0.249133, False][-0.21898, False]Failed

SHA-256 / 8ca58197bb5d016cbee843672d8628828bcac9b48a742ccbf5f89b5f759bc628

2 / The unsuccessful fix

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(pre_c, pre_t, threshold):
    def moments(xs):
        n = len(xs)
        m = sum(xs) / n
        return m, sum((x - m) ** 2 for x in xs) / (n + 1)
    if len(pre_c) < 2 or len(pre_t) < 2:
        return None
    mc, vc = moments(pre_c)
    mt, vt = moments(pre_t)
    pooled = math.sqrt((vc + vt) / 2)
    if pooled == 0:
        return [0.0, False] if mt == mc else [None, True]
    smd = (mt - mc) / pooled
    return [round(smd, 6), abs(smd) > threshold]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('pre-period sample 1', [[1, 3, 6], [10, 3], 0.5], [0.806505, True]),
  ('pre-period sample 2', [[5, 9, 3, 4], [3, 7], 0.1], [-0.091542, False]),
  ('pre-period sample 3', [[6, 8, 3, 8, 9], [6, 5, 4, 10], 0.25], [-0.21898, False])],
 [('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('pre-period sample 4', [[5, 7], [2, 10, 7, 9, 9], 0.5], [0.564534, True]),
  ('pre-period sample 6', [[8, 0], [8, 5, 6, 2], 0.1], [0.285831, True]),
  ('pre-period sample 7', [[7, 0, 8, 6, 7], [9, 4, 5, 2, 2], 0.5], [-0.393496, False])],
 [('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 11', [[8, 3, 4], [7, 3, 4], 0.1], [-0.140028, True]),
  ('pre-period sample 12', [[4, 3, 3], [4, 0], 0.1], [-0.653197, True]),
  ('pre-period sample 13', [[3, 5], [5, 7, 10, 7, 2], 0.1], [0.951143, True])],
 [('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 16', [[3, 5, 5, 2], [4, 11], 0.1], [1.025379, True]),
  ('pre-period sample 17', [[6, 1, 1, 4, 1], [7, 10, 8, 4], 0.25], [1.934982, True]),
  ('pre-period sample 19', [[3, 1, 3, 7], [5, 4, 9], 0.1], [0.968246, True])],
 [('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 21', [[3, 2, 5, 0], [0, 6], 0.5], [0.149626, False]),
  ('pre-period sample 22', [[5, 8, 9, 5], [11, 8, 5, 6], 0.5], [0.316228, False]),
  ('pre-period sample 26', [[6, 7, 1, 4, 9], [3, 4, 11, 6, 10], 0.25], [0.422116, True])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
treatment arm more variable[0.970143, True][0.685994, True]Failed
SMD exactly at the tolerance passes[0.707107, True][0.5, False]Failed
negative imbalance is flagged[-5.656854, True][-4.0, True]Failed
small arms[1.549193, True][0.894427, True]Failed
constant equal arms[0.0, False][0.0, False]Passed
pre-period sample 1[1.330266, True][0.806505, True]Failed
pre-period sample 2[-0.135416, True][-0.091542, False]Failed
pre-period sample 3[-0.275863, True][-0.21898, False]Failed

SHA-256 / c5e9fc3fe5edd0f25dc2ce3f4f3dd6b22011c8ec8b99d356d47993e448f6d0e2

3 / The verified repair

Exit 0
"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(pre_c, pre_t, threshold):
    def moments(xs):
        n = len(xs)
        m = sum(xs) / n
        return m, sum((x - m) ** 2 for x in xs) / (n - 1)
    if len(pre_c) < 2 or len(pre_t) < 2:
        return None
    mc, vc = moments(pre_c)
    mt, vt = moments(pre_t)
    pooled = math.sqrt((vc + vt) / 2)
    if pooled == 0:
        return [0.0, False] if mt == mc else [None, True]
    smd = (mt - mc) / pooled
    return [round(smd, 6), abs(smd) > threshold]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('pre-period sample 1', [[1, 3, 6], [10, 3], 0.5], [0.806505, True]),
  ('pre-period sample 2', [[5, 9, 3, 4], [3, 7], 0.1], [-0.091542, False]),
  ('pre-period sample 3', [[6, 8, 3, 8, 9], [6, 5, 4, 10], 0.25], [-0.21898, False])],
 [('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('pre-period sample 4', [[5, 7], [2, 10, 7, 9, 9], 0.5], [0.564534, True]),
  ('pre-period sample 6', [[8, 0], [8, 5, 6, 2], 0.1], [0.285831, True]),
  ('pre-period sample 7', [[7, 0, 8, 6, 7], [9, 4, 5, 2, 2], 0.5], [-0.393496, False])],
 [('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 11', [[8, 3, 4], [7, 3, 4], 0.1], [-0.140028, True]),
  ('pre-period sample 12', [[4, 3, 3], [4, 0], 0.1], [-0.653197, True]),
  ('pre-period sample 13', [[3, 5], [5, 7, 10, 7, 2], 0.1], [0.951143, True])],
 [('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 16', [[3, 5, 5, 2], [4, 11], 0.1], [1.025379, True]),
  ('pre-period sample 17', [[6, 1, 1, 4, 1], [7, 10, 8, 4], 0.25], [1.934982, True]),
  ('pre-period sample 19', [[3, 1, 3, 7], [5, 4, 9], 0.1], [0.968246, True])],
 [('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 21', [[3, 2, 5, 0], [0, 6], 0.5], [0.149626, False]),
  ('pre-period sample 22', [[5, 8, 9, 5], [11, 8, 5, 6], 0.5], [0.316228, False]),
  ('pre-period sample 26', [[6, 7, 1, 4, 9], [3, 4, 11, 6, 10], 0.25], [0.422116, True])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
treatment arm more variable[0.685994, True][0.685994, True]Passed
SMD exactly at the tolerance passes[0.5, False][0.5, False]Passed
negative imbalance is flagged[-4.0, True][-4.0, True]Passed
small arms[0.894427, True][0.894427, True]Passed
constant equal arms[0.0, False][0.0, False]Passed
pre-period sample 1[0.806505, True][0.806505, True]Passed
pre-period sample 2[-0.091542, False][-0.091542, False]Passed
pre-period sample 3[-0.21898, False][-0.21898, False]Passed

SHA-256 / 9a029c9e6b460d1e3eaae0d68fdde861da68bd3cca336e29b35cd5c871121d92

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:59.671798+00:00.

Case digest / f78c4edafff634452e6936fa16933bff39ba8832006ce2038d4e6a9e533ebac8