FAILURE MAP
← Case archive

FA-74746 / Experiment statistics / Open access

Pre-period balance check: The difference is scaled by the control spread only · case 01

A treatment arm with extra variance changes the balance score asymmetrically.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The denominator is the control standard deviation.

VERIFIED REPAIR

Use the square root of the average of both variances.

Unsuccessful approach: Averaging the two standard deviations is not the pooled root-mean variance.

Case contract

Standardised mean difference = (mean_t - mean_c) / sqrt((var_c + var_t) / 2) with sample (n - 1) variances; imbalance iff |SMD| > threshold. Zero pooled variance -> [0.0, False] when means are equal else [None, True]. Fewer than two values in an arm -> None. Return [round(smd, 6), imbalance].

Why this case matters

Pre-period imbalance warns that randomisation or logging went wrong before any effect is read.

1 / The failure

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(pre_c, pre_t, threshold):
    def moments(xs):
        n = len(xs)
        m = sum(xs) / n
        return m, sum((x - m) ** 2 for x in xs) / (n - 1)
    if len(pre_c) < 2 or len(pre_t) < 2:
        return None
    mc, vc = moments(pre_c)
    mt, vt = moments(pre_t)
    pooled = math.sqrt(vc)
    if pooled == 0:
        return [0.0, False] if mt == mc else [None, True]
    smd = (mt - mc) / pooled
    return [round(smd, 6), abs(smd) > threshold]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('pre-period sample 1', [[1, 3, 6], [10, 3], 0.5], [0.806505, True]),
  ('pre-period sample 2', [[5, 9, 3, 4], [3, 7], 0.1], [-0.091542, False]),
  ('pre-period sample 3', [[6, 8, 3, 8, 9], [6, 5, 4, 10], 0.25], [-0.21898, False])],
 [('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('pre-period sample 6', [[8, 0], [8, 5, 6, 2], 0.1], [0.285831, True]),
  ('pre-period sample 7', [[7, 0, 8, 6, 7], [9, 4, 5, 2, 2], 0.5], [-0.393496, False]),
  ('pre-period sample 8', [[8, 7], [6, 1], 0.25], [-1.568929, True])],
 [('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 11', [[8, 3, 4], [7, 3, 4], 0.1], [-0.140028, True]),
  ('pre-period sample 12', [[4, 3, 3], [4, 0], 0.1], [-0.653197, True]),
  ('pre-period sample 14', [[6, 4, 1], [5, 9, 6, 9, 9], 0.5], [1.747429, True])],
 [('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 16', [[3, 5, 5, 2], [4, 11], 0.1], [1.025379, True]),
  ('pre-period sample 17', [[6, 1, 1, 4, 1], [7, 10, 8, 4], 0.25], [1.934982, True]),
  ('pre-period sample 21', [[3, 2, 5, 0], [0, 6], 0.5], [0.149626, False])],
 [('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 21', [[3, 2, 5, 0], [0, 6], 0.5], [0.149626, False]),
  ('pre-period sample 22', [[5, 8, 9, 5], [11, 8, 5, 6], 0.5], [0.316228, False]),
  ('pre-period sample 28', [[2, 6, 9, 6, 5], [6, 3, 8, 3, 7], 0.1], [-0.083045, False])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
treatment arm more variable[2.0, True][0.685994, True]Failed
SMD exactly at the tolerance passes[0.5, False][0.5, False]Passed
negative imbalance is flagged[-4.0, True][-4.0, True]Passed
small arms[1.414214, True][0.894427, True]Failed
constant equal arms[0.0, False][0.0, False]Passed
pre-period sample 1[1.258306, True][0.806505, True]Failed
pre-period sample 2[-0.095059, False][-0.091542, False]Failed
pre-period sample 3[-0.23037, False][-0.21898, False]Failed

SHA-256 / 7d324db9785e1314dcd3ee559e8bf57b039775b691c122e6c39c50a7d2430f31

2 / The unsuccessful fix

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(pre_c, pre_t, threshold):
    def moments(xs):
        n = len(xs)
        m = sum(xs) / n
        return m, sum((x - m) ** 2 for x in xs) / (n - 1)
    if len(pre_c) < 2 or len(pre_t) < 2:
        return None
    mc, vc = moments(pre_c)
    mt, vt = moments(pre_t)
    pooled = (math.sqrt(vc) + math.sqrt(vt)) / 2
    if pooled == 0:
        return [0.0, False] if mt == mc else [None, True]
    smd = (mt - mc) / pooled
    return [round(smd, 6), abs(smd) > threshold]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('pre-period sample 1', [[1, 3, 6], [10, 3], 0.5], [0.806505, True]),
  ('pre-period sample 2', [[5, 9, 3, 4], [3, 7], 0.1], [-0.091542, False]),
  ('pre-period sample 3', [[6, 8, 3, 8, 9], [6, 5, 4, 10], 0.25], [-0.21898, False])],
 [('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('pre-period sample 6', [[8, 0], [8, 5, 6, 2], 0.1], [0.285831, True]),
  ('pre-period sample 7', [[7, 0, 8, 6, 7], [9, 4, 5, 2, 2], 0.5], [-0.393496, False]),
  ('pre-period sample 8', [[8, 7], [6, 1], 0.25], [-1.568929, True])],
 [('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 11', [[8, 3, 4], [7, 3, 4], 0.1], [-0.140028, True]),
  ('pre-period sample 12', [[4, 3, 3], [4, 0], 0.1], [-0.653197, True]),
  ('pre-period sample 14', [[6, 4, 1], [5, 9, 6, 9, 9], 0.5], [1.747429, True])],
 [('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 16', [[3, 5, 5, 2], [4, 11], 0.1], [1.025379, True]),
  ('pre-period sample 17', [[6, 1, 1, 4, 1], [7, 10, 8, 4], 0.25], [1.934982, True]),
  ('pre-period sample 21', [[3, 2, 5, 0], [0, 6], 0.5], [0.149626, False])],
 [('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 21', [[3, 2, 5, 0], [0, 6], 0.5], [0.149626, False]),
  ('pre-period sample 22', [[5, 8, 9, 5], [11, 8, 5, 6], 0.5], [0.316228, False]),
  ('pre-period sample 28', [[2, 6, 9, 6, 5], [6, 3, 8, 3, 7], 0.1], [-0.083045, False])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
treatment arm more variable[0.8, True][0.685994, True]Failed
SMD exactly at the tolerance passes[0.5, False][0.5, False]Passed
negative imbalance is flagged[-4.0, True][-4.0, True]Passed
small arms[0.942809, True][0.894427, True]Failed
constant equal arms[0.0, False][0.0, False]Passed
pre-period sample 1[0.848249, True][0.806505, True]Failed
pre-period sample 2[-0.091602, False][-0.091542, False]Failed
pre-period sample 3[-0.219236, False][-0.21898, False]Failed

SHA-256 / ec7698fa9ea413eb0402218b101b916611728ffc53115d4527d995e4aceaf87c

3 / The verified repair

Exit 0
"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(pre_c, pre_t, threshold):
    def moments(xs):
        n = len(xs)
        m = sum(xs) / n
        return m, sum((x - m) ** 2 for x in xs) / (n - 1)
    if len(pre_c) < 2 or len(pre_t) < 2:
        return None
    mc, vc = moments(pre_c)
    mt, vt = moments(pre_t)
    pooled = math.sqrt((vc + vt) / 2)
    if pooled == 0:
        return [0.0, False] if mt == mc else [None, True]
    smd = (mt - mc) / pooled
    return [round(smd, 6), abs(smd) > threshold]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('pre-period sample 1', [[1, 3, 6], [10, 3], 0.5], [0.806505, True]),
  ('pre-period sample 2', [[5, 9, 3, 4], [3, 7], 0.1], [-0.091542, False]),
  ('pre-period sample 3', [[6, 8, 3, 8, 9], [6, 5, 4, 10], 0.25], [-0.21898, False])],
 [('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('pre-period sample 6', [[8, 0], [8, 5, 6, 2], 0.1], [0.285831, True]),
  ('pre-period sample 7', [[7, 0, 8, 6, 7], [9, 4, 5, 2, 2], 0.5], [-0.393496, False]),
  ('pre-period sample 8', [[8, 7], [6, 1], 0.25], [-1.568929, True])],
 [('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 11', [[8, 3, 4], [7, 3, 4], 0.1], [-0.140028, True]),
  ('pre-period sample 12', [[4, 3, 3], [4, 0], 0.1], [-0.653197, True]),
  ('pre-period sample 14', [[6, 4, 1], [5, 9, 6, 9, 9], 0.5], [1.747429, True])],
 [('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 16', [[3, 5, 5, 2], [4, 11], 0.1], [1.025379, True]),
  ('pre-period sample 17', [[6, 1, 1, 4, 1], [7, 10, 8, 4], 0.25], [1.934982, True]),
  ('pre-period sample 21', [[3, 2, 5, 0], [0, 6], 0.5], [0.149626, False])],
 [('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 21', [[3, 2, 5, 0], [0, 6], 0.5], [0.149626, False]),
  ('pre-period sample 22', [[5, 8, 9, 5], [11, 8, 5, 6], 0.5], [0.316228, False]),
  ('pre-period sample 28', [[2, 6, 9, 6, 5], [6, 3, 8, 3, 7], 0.1], [-0.083045, False])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
treatment arm more variable[0.685994, True][0.685994, True]Passed
SMD exactly at the tolerance passes[0.5, False][0.5, False]Passed
negative imbalance is flagged[-4.0, True][-4.0, True]Passed
small arms[0.894427, True][0.894427, True]Passed
constant equal arms[0.0, False][0.0, False]Passed
pre-period sample 1[0.806505, True][0.806505, True]Passed
pre-period sample 2[-0.091542, False][-0.091542, False]Passed
pre-period sample 3[-0.21898, False][-0.21898, False]Passed

SHA-256 / ce84e30d906b1c405a6b7f1fc09ebee70a4852aaefa0709c607cf664b90b7502

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:59.648879+00:00.

Case digest / 8176165557e1268cab20f5005cd82dbe297c82fa810adad6fb0707e4a40805b7