FAILURE MAP
← Case archive

FA-74761 / Experiment statistics / Open access

Pre-period balance check: Constant equal arms are reported as imbalanced · case 01

A binary pre-period flag that is identical in both arms blocks the readout.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

Zero pooled variance always returns [None, True].

VERIFIED REPAIR

Return [0.0, False] when both constant arms share the same mean.

Unsuccessful approach: Always reporting balance hides arms that differ by a constant.

Case contract

Standardised mean difference = (mean_t - mean_c) / sqrt((var_c + var_t) / 2) with sample (n - 1) variances; imbalance iff |SMD| > threshold. Zero pooled variance -> [0.0, False] when means are equal else [None, True]. Fewer than two values in an arm -> None. Return [round(smd, 6), imbalance].

Why this case matters

Pre-period imbalance warns that randomisation or logging went wrong before any effect is read.

1 / The failure

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(pre_c, pre_t, threshold):
    def moments(xs):
        n = len(xs)
        m = sum(xs) / n
        return m, sum((x - m) ** 2 for x in xs) / (n - 1)
    if len(pre_c) < 2 or len(pre_t) < 2:
        return None
    mc, vc = moments(pre_c)
    mt, vt = moments(pre_t)
    pooled = math.sqrt((vc + vt) / 2)
    if pooled == 0:
        return [None, True]
    smd = (mt - mc) / pooled
    return [round(smd, 6), abs(smd) > threshold]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('pre-period sample 1', [[1, 3, 6], [10, 3], 0.5], [0.806505, True]),
  ('pre-period sample 2', [[5, 9, 3, 4], [3, 7], 0.1], [-0.091542, False])],
 [('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('pre-period sample 6', [[8, 0], [8, 5, 6, 2], 0.1], [0.285831, True]),
  ('pre-period sample 7', [[7, 0, 8, 6, 7], [9, 4, 5, 2, 2], 0.5], [-0.393496, False]),
  ('pre-period sample 8', [[8, 7], [6, 1], 0.25], [-1.568929, True])],
 [('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 11', [[8, 3, 4], [7, 3, 4], 0.1], [-0.140028, True]),
  ('pre-period sample 12', [[4, 3, 3], [4, 0], 0.1], [-0.653197, True]),
  ('pre-period sample 13', [[3, 5], [5, 7, 10, 7, 2], 0.1], [0.951143, True])],
 [('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 16', [[3, 5, 5, 2], [4, 11], 0.1], [1.025379, True]),
  ('pre-period sample 17', [[6, 1, 1, 4, 1], [7, 10, 8, 4], 0.25], [1.934982, True]),
  ('pre-period sample 18', [[1, 3, 0], [11, 4, 2, 9, 6], 0.5], [1.812223, True])],
 [('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 21', [[3, 2, 5, 0], [0, 6], 0.5], [0.149626, False]),
  ('pre-period sample 22', [[5, 8, 9, 5], [11, 8, 5, 6], 0.5], [0.316228, False]),
  ('pre-period sample 23', [[9, 9], [2, 7, 8], 0.5], [-1.466471, True])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
treatment arm more variable[0.685994, True][0.685994, True]Passed
SMD exactly at the tolerance passes[0.5, False][0.5, False]Passed
negative imbalance is flagged[-4.0, True][-4.0, True]Passed
small arms[0.894427, True][0.894427, True]Passed
constant equal arms[None, True][0.0, False]Failed
constant different arms[None, True][None, True]Passed
pre-period sample 1[0.806505, True][0.806505, True]Passed
pre-period sample 2[-0.091542, False][-0.091542, False]Passed

SHA-256 / 502317737c4306f31f01fe80a07bd728a2bd6070fc33723f95af415c3d25ef97

2 / The unsuccessful fix

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(pre_c, pre_t, threshold):
    def moments(xs):
        n = len(xs)
        m = sum(xs) / n
        return m, sum((x - m) ** 2 for x in xs) / (n - 1)
    if len(pre_c) < 2 or len(pre_t) < 2:
        return None
    mc, vc = moments(pre_c)
    mt, vt = moments(pre_t)
    pooled = math.sqrt((vc + vt) / 2)
    if pooled == 0:
        return [0.0, False]
    smd = (mt - mc) / pooled
    return [round(smd, 6), abs(smd) > threshold]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('pre-period sample 1', [[1, 3, 6], [10, 3], 0.5], [0.806505, True]),
  ('pre-period sample 2', [[5, 9, 3, 4], [3, 7], 0.1], [-0.091542, False])],
 [('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('pre-period sample 6', [[8, 0], [8, 5, 6, 2], 0.1], [0.285831, True]),
  ('pre-period sample 7', [[7, 0, 8, 6, 7], [9, 4, 5, 2, 2], 0.5], [-0.393496, False]),
  ('pre-period sample 8', [[8, 7], [6, 1], 0.25], [-1.568929, True])],
 [('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 11', [[8, 3, 4], [7, 3, 4], 0.1], [-0.140028, True]),
  ('pre-period sample 12', [[4, 3, 3], [4, 0], 0.1], [-0.653197, True]),
  ('pre-period sample 13', [[3, 5], [5, 7, 10, 7, 2], 0.1], [0.951143, True])],
 [('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 16', [[3, 5, 5, 2], [4, 11], 0.1], [1.025379, True]),
  ('pre-period sample 17', [[6, 1, 1, 4, 1], [7, 10, 8, 4], 0.25], [1.934982, True]),
  ('pre-period sample 18', [[1, 3, 0], [11, 4, 2, 9, 6], 0.5], [1.812223, True])],
 [('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 21', [[3, 2, 5, 0], [0, 6], 0.5], [0.149626, False]),
  ('pre-period sample 22', [[5, 8, 9, 5], [11, 8, 5, 6], 0.5], [0.316228, False]),
  ('pre-period sample 23', [[9, 9], [2, 7, 8], 0.5], [-1.466471, True])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
treatment arm more variable[0.685994, True][0.685994, True]Passed
SMD exactly at the tolerance passes[0.5, False][0.5, False]Passed
negative imbalance is flagged[-4.0, True][-4.0, True]Passed
small arms[0.894427, True][0.894427, True]Passed
constant equal arms[0.0, False][0.0, False]Passed
constant different arms[0.0, False][None, True]Failed
pre-period sample 1[0.806505, True][0.806505, True]Passed
pre-period sample 2[-0.091542, False][-0.091542, False]Passed

SHA-256 / 9fd92bef642b499bf74da70d0f73f78372dfb97bbcfcba056df4e43827db9f0a

3 / The verified repair

Exit 0
"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(pre_c, pre_t, threshold):
    def moments(xs):
        n = len(xs)
        m = sum(xs) / n
        return m, sum((x - m) ** 2 for x in xs) / (n - 1)
    if len(pre_c) < 2 or len(pre_t) < 2:
        return None
    mc, vc = moments(pre_c)
    mt, vt = moments(pre_t)
    pooled = math.sqrt((vc + vt) / 2)
    if pooled == 0:
        return [0.0, False] if mt == mc else [None, True]
    smd = (mt - mc) / pooled
    return [round(smd, 6), abs(smd) > threshold]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('pre-period sample 1', [[1, 3, 6], [10, 3], 0.5], [0.806505, True]),
  ('pre-period sample 2', [[5, 9, 3, 4], [3, 7], 0.1], [-0.091542, False])],
 [('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('pre-period sample 6', [[8, 0], [8, 5, 6, 2], 0.1], [0.285831, True]),
  ('pre-period sample 7', [[7, 0, 8, 6, 7], [9, 4, 5, 2, 2], 0.5], [-0.393496, False]),
  ('pre-period sample 8', [[8, 7], [6, 1], 0.25], [-1.568929, True])],
 [('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 11', [[8, 3, 4], [7, 3, 4], 0.1], [-0.140028, True]),
  ('pre-period sample 12', [[4, 3, 3], [4, 0], 0.1], [-0.653197, True]),
  ('pre-period sample 13', [[3, 5], [5, 7, 10, 7, 2], 0.1], [0.951143, True])],
 [('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 16', [[3, 5, 5, 2], [4, 11], 0.1], [1.025379, True]),
  ('pre-period sample 17', [[6, 1, 1, 4, 1], [7, 10, 8, 4], 0.25], [1.934982, True]),
  ('pre-period sample 18', [[1, 3, 0], [11, 4, 2, 9, 6], 0.5], [1.812223, True])],
 [('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 21', [[3, 2, 5, 0], [0, 6], 0.5], [0.149626, False]),
  ('pre-period sample 22', [[5, 8, 9, 5], [11, 8, 5, 6], 0.5], [0.316228, False]),
  ('pre-period sample 23', [[9, 9], [2, 7, 8], 0.5], [-1.466471, True])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
treatment arm more variable[0.685994, True][0.685994, True]Passed
SMD exactly at the tolerance passes[0.5, False][0.5, False]Passed
negative imbalance is flagged[-4.0, True][-4.0, True]Passed
small arms[0.894427, True][0.894427, True]Passed
constant equal arms[0.0, False][0.0, False]Passed
constant different arms[None, True][None, True]Passed
pre-period sample 1[0.806505, True][0.806505, True]Passed
pre-period sample 2[-0.091542, False][-0.091542, False]Passed

SHA-256 / 19f33ccfc064850f3661b5f3ea1b0e465a4a82c777649413c23af70bf87c049a

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:59.712831+00:00.

Case digest / 1cec47b8709bf487d9922af876cd2a11bf626d44f311df72de68b03f77a7cc06