FAILURE MAP
← Case archive

FA-74756 / Experiment statistics / Open access

Pre-period balance check: Only positive imbalance is flagged · case 01

Treatment arms that start lower than control pass the balance check.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The flag tests smd > threshold without the absolute value.

VERIFIED REPAIR

Flag when |SMD| exceeds the threshold.

Unsuccessful approach: Making the comparison inclusive flags SMDs exactly at the tolerance.

Case contract

Standardised mean difference = (mean_t - mean_c) / sqrt((var_c + var_t) / 2) with sample (n - 1) variances; imbalance iff |SMD| > threshold. Zero pooled variance -> [0.0, False] when means are equal else [None, True]. Fewer than two values in an arm -> None. Return [round(smd, 6), imbalance].

Why this case matters

Pre-period imbalance warns that randomisation or logging went wrong before any effect is read.

1 / The failure

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(pre_c, pre_t, threshold):
    def moments(xs):
        n = len(xs)
        m = sum(xs) / n
        return m, sum((x - m) ** 2 for x in xs) / (n - 1)
    if len(pre_c) < 2 or len(pre_t) < 2:
        return None
    mc, vc = moments(pre_c)
    mt, vt = moments(pre_t)
    pooled = math.sqrt((vc + vt) / 2)
    if pooled == 0:
        return [0.0, False] if mt == mc else [None, True]
    smd = (mt - mc) / pooled
    return [round(smd, 6), smd > threshold]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('pre-period sample 1', [[1, 3, 6], [10, 3], 0.5], [0.806505, True]),
  ('pre-period sample 2', [[5, 9, 3, 4], [3, 7], 0.1], [-0.091542, False]),
  ('pre-period sample 3', [[6, 8, 3, 8, 9], [6, 5, 4, 10], 0.25], [-0.21898, False])],
 [('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('pre-period sample 6', [[8, 0], [8, 5, 6, 2], 0.1], [0.285831, True]),
  ('pre-period sample 7', [[7, 0, 8, 6, 7], [9, 4, 5, 2, 2], 0.5], [-0.393496, False]),
  ('pre-period sample 24', [[9, 7], [2, 9], 0.25], [-0.686803, True])],
 [('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 5', [[6, 6, 8], [5, 4, 8, 1, 9], 0.1], [-0.525201, True]),
  ('pre-period sample 11', [[8, 3, 4], [7, 3, 4], 0.1], [-0.140028, True])],
 [('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 16', [[3, 5, 5, 2], [4, 11], 0.1], [1.025379, True]),
  ('pre-period sample 25', [[6, 8, 2], [3, 3, 7, 0, 7], 0.25], [-0.440386, True])],
 [('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 8', [[8, 7], [6, 1], 0.25], [-1.568929, True]),
  ('pre-period sample 21', [[3, 2, 5, 0], [0, 6], 0.5], [0.149626, False]),
  ('pre-period sample 22', [[5, 8, 9, 5], [11, 8, 5, 6], 0.5], [0.316228, False])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
treatment arm more variable[0.685994, True][0.685994, True]Passed
SMD exactly at the tolerance passes[0.5, False][0.5, False]Passed
negative imbalance is flagged[-4.0, False][-4.0, True]Failed
small arms[0.894427, True][0.894427, True]Passed
constant equal arms[0.0, False][0.0, False]Passed
pre-period sample 1[0.806505, True][0.806505, True]Passed
pre-period sample 2[-0.091542, False][-0.091542, False]Passed
pre-period sample 3[-0.21898, False][-0.21898, False]Passed

SHA-256 / c065a52269c43b2091b7caab54ec3d889c0759885d5f44b3c0b73160bcc1e6c9

2 / The unsuccessful fix

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(pre_c, pre_t, threshold):
    def moments(xs):
        n = len(xs)
        m = sum(xs) / n
        return m, sum((x - m) ** 2 for x in xs) / (n - 1)
    if len(pre_c) < 2 or len(pre_t) < 2:
        return None
    mc, vc = moments(pre_c)
    mt, vt = moments(pre_t)
    pooled = math.sqrt((vc + vt) / 2)
    if pooled == 0:
        return [0.0, False] if mt == mc else [None, True]
    smd = (mt - mc) / pooled
    return [round(smd, 6), abs(smd) >= threshold]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('pre-period sample 1', [[1, 3, 6], [10, 3], 0.5], [0.806505, True]),
  ('pre-period sample 2', [[5, 9, 3, 4], [3, 7], 0.1], [-0.091542, False]),
  ('pre-period sample 3', [[6, 8, 3, 8, 9], [6, 5, 4, 10], 0.25], [-0.21898, False])],
 [('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('pre-period sample 6', [[8, 0], [8, 5, 6, 2], 0.1], [0.285831, True]),
  ('pre-period sample 7', [[7, 0, 8, 6, 7], [9, 4, 5, 2, 2], 0.5], [-0.393496, False]),
  ('pre-period sample 24', [[9, 7], [2, 9], 0.25], [-0.686803, True])],
 [('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 5', [[6, 6, 8], [5, 4, 8, 1, 9], 0.1], [-0.525201, True]),
  ('pre-period sample 11', [[8, 3, 4], [7, 3, 4], 0.1], [-0.140028, True])],
 [('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 16', [[3, 5, 5, 2], [4, 11], 0.1], [1.025379, True]),
  ('pre-period sample 25', [[6, 8, 2], [3, 3, 7, 0, 7], 0.25], [-0.440386, True])],
 [('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 8', [[8, 7], [6, 1], 0.25], [-1.568929, True]),
  ('pre-period sample 21', [[3, 2, 5, 0], [0, 6], 0.5], [0.149626, False]),
  ('pre-period sample 22', [[5, 8, 9, 5], [11, 8, 5, 6], 0.5], [0.316228, False])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
treatment arm more variable[0.685994, True][0.685994, True]Passed
SMD exactly at the tolerance passes[0.5, True][0.5, False]Failed
negative imbalance is flagged[-4.0, True][-4.0, True]Passed
small arms[0.894427, True][0.894427, True]Passed
constant equal arms[0.0, False][0.0, False]Passed
pre-period sample 1[0.806505, True][0.806505, True]Passed
pre-period sample 2[-0.091542, False][-0.091542, False]Passed
pre-period sample 3[-0.21898, False][-0.21898, False]Passed

SHA-256 / fc00bfa5050a54396a609d1071195db24cdc634df5603752712f51d2852b2813

3 / The verified repair

Exit 0
"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(pre_c, pre_t, threshold):
    def moments(xs):
        n = len(xs)
        m = sum(xs) / n
        return m, sum((x - m) ** 2 for x in xs) / (n - 1)
    if len(pre_c) < 2 or len(pre_t) < 2:
        return None
    mc, vc = moments(pre_c)
    mt, vt = moments(pre_t)
    pooled = math.sqrt((vc + vt) / 2)
    if pooled == 0:
        return [0.0, False] if mt == mc else [None, True]
    smd = (mt - mc) / pooled
    return [round(smd, 6), abs(smd) > threshold]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('pre-period sample 1', [[1, 3, 6], [10, 3], 0.5], [0.806505, True]),
  ('pre-period sample 2', [[5, 9, 3, 4], [3, 7], 0.1], [-0.091542, False]),
  ('pre-period sample 3', [[6, 8, 3, 8, 9], [6, 5, 4, 10], 0.25], [-0.21898, False])],
 [('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('pre-period sample 6', [[8, 0], [8, 5, 6, 2], 0.1], [0.285831, True]),
  ('pre-period sample 7', [[7, 0, 8, 6, 7], [9, 4, 5, 2, 2], 0.5], [-0.393496, False]),
  ('pre-period sample 24', [[9, 7], [2, 9], 0.25], [-0.686803, True])],
 [('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('negative imbalance is flagged', [[5, 6, 7], [1, 2, 3], 0.25], [-4.0, True]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 5', [[6, 6, 8], [5, 4, 8, 1, 9], 0.1], [-0.525201, True]),
  ('pre-period sample 11', [[8, 3, 4], [7, 3, 4], 0.1], [-0.140028, True])],
 [('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('small arms', [[1, 3], [2, 6], 0.1], [0.894427, True]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 16', [[3, 5, 5, 2], [4, 11], 0.1], [1.025379, True]),
  ('pre-period sample 25', [[6, 8, 2], [3, 3, 7, 0, 7], 0.25], [-0.440386, True])],
 [('treatment arm more variable', [[1, 2, 3], [0, 4, 8], 0.25], [0.685994, True]),
  ('SMD exactly at the tolerance passes', [[0, 2, 4], [1, 3, 5], 0.5], [0.5, False]),
  ('constant equal arms', [[4, 4], [4, 4], 0.1], [0.0, False]),
  ('constant different arms', [[4, 4], [5, 5], 0.1], [None, True]),
  ('single observation arm', [[1], [1, 2], 0.1], None),
  ('pre-period sample 8', [[8, 7], [6, 1], 0.25], [-1.568929, True]),
  ('pre-period sample 21', [[3, 2, 5, 0], [0, 6], 0.5], [0.149626, False]),
  ('pre-period sample 22', [[5, 8, 9, 5], [11, 8, 5, 6], 0.5], [0.316228, False])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
treatment arm more variable[0.685994, True][0.685994, True]Passed
SMD exactly at the tolerance passes[0.5, False][0.5, False]Passed
negative imbalance is flagged[-4.0, True][-4.0, True]Passed
small arms[0.894427, True][0.894427, True]Passed
constant equal arms[0.0, False][0.0, False]Passed
pre-period sample 1[0.806505, True][0.806505, True]Passed
pre-period sample 2[-0.091542, False][-0.091542, False]Passed
pre-period sample 3[-0.21898, False][-0.21898, False]Passed

SHA-256 / af6e1f0e8a3b0f2470f3eb772ea3227c912c3bbce10e827eac129660aa30f161

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:59.685829+00:00.

Case digest / 111dec5caa01b7225397fd055de6bea8d5e69a4401428b6e586eaeaf4080d5c0