FA-74431 / Experiment statistics / Open access

CUPED covariate adjustment: Theta is normalised by the outcome variance · case 01

The adjustment coefficient is wrong whenever outcome and covariate scales differ.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

theta divides cov(y, x) by var(y) instead of var(x).

VERIFIED REPAIR

Divide the covariance by the covariate variance.

Unsuccessful approach: Inverting to var(x) / cov(y, x) gives the reciprocal regression slope.

Case contract

theta = cov(y, x) / var(x) over the pooled data of both arms (theta = 0 when x is constant). Each arm's adjusted mean is mean(y_arm) - theta * (mean(x_arm) - pooled mean(x)). Return [theta, adjusted control, adjusted treatment, difference] rounded to 6 places.

Why this case matters

CUPED uses pre-period data to cut variance; mistakes either waste it or bias the estimate.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(y_c, x_c, y_t, x_t):
    ys = y_c + y_t
    xs = x_c + x_t
    n = len(xs)
    mx, my = sum(xs) / n, sum(ys) / n
    vx = sum((x - mx) ** 2 for x in xs)
    cxy = sum((x - mx) * (y - my) for x, y in zip(xs, ys))
    theta = cxy / sum((y - my) ** 2 for y in ys) if vx else 0.0
    def adj(ys_, xs_):
        mean_x = sum(xs_) / len(xs_)
        return sum(ys_) / len(ys_) - theta * (mean_x - mx)
    ac, at = adj(y_c, x_c), adj(y_t, x_t)
    return [round(theta, 6), round(ac, 6), round(at, 6), round(at - ac, 6)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 1',
   [[8, 8, 10, 16], [3, 5, 7, 8], [6, 10], [3, 7]],
   [1.319149, 10.170213, 8.659574, -1.510638]),
  ('pre/post sample 2',
   [[9, 8, 18], [8, 8, 8], [0, 3, 11, 1, 2], [0, 2, 8, 0, 1]],
   [1.376593, 6.676516, 6.39409, -0.282426]),
  ('pre/post sample 3',
   [[4, 12, 1], [1, 5, 1], [6, 15, 20, 16], [0, 9, 8, 6]],
   [1.748201, 9.079822, 11.690134, 2.610312])],
 [('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 4',
   [[9, 8, 8, 3, 8], [6, 4, 7, 1, 5], [10, 20, 12, 6, 6], [3, 8, 4, 0, 0]],
   [1.131285, 6.294972, 11.705028, 5.410056]),
  ('pre/post sample 5',
   [[14, 4, 11, 5], [6, 4, 9, 4], [9, 1, 6, 8, 6], [7, 0, 3, 7, 3]],
   [1.145522, 7.386298, 6.890962, -0.495336]),
  ('pre/post sample 6',
   [[9, 12, 4, 8, 10], [7, 6, 3, 7, 4], [10, 12], [4, 9]],
   [0.713542, 8.824256, 10.43936, 1.615104])],
 [('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 11', [[10, 16], [8, 7], [7, 19], [6, 9]], [3.0, 13.0, 13.0, 0.0]),
  ('pre/post sample 12', [[10, 10], [4, 7], [10, 9], [6, 3]], [0.2, 9.9, 9.6, -0.3]),
  ('pre/post sample 13', [[8, 5, 9, 7], [6, 5, 7, 5], [16, 11], [5, 8]], [0.375, 7.34375, 13.3125, 5.96875])],
 [('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 16',
   [[18, 4, 1, 10], [8, 1, 0, 5], [16, 13, 15, 3, 7], [5, 8, 5, 0, 2]],
   [1.789106, 8.746974, 10.402421, 1.655447]),
  ('pre/post sample 18',
   [[8, 21], [7, 9], [9, 11, 4], [6, 7, 0]],
   [1.465812, 11.275214, 10.149858, -1.125356]),
  ('pre/post sample 19',
   [[10, 20, 2, 11, 4], [9, 9, 1, 5, 3], [12, 18, 10], [9, 7, 4]],
   [1.523364, 10.123598, 12.127336, 2.003738])],
 [('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 21', [[3, 12], [2, 5], [21, 16], [9, 6]], [2.6, 12.7, 13.3, 0.6]),
  ('pre/post sample 25', [[2, 7], [1, 3], [19, 5], [7, 2]], [2.831325, 8.039157, 8.460843, 0.421687]),
  ('pre/post sample 26',
   [[11, 8], [5, 7], [7, 15, 12, 10], [2, 9, 4, 7]],
   [0.670213, 9.276596, 11.111702, 1.835106])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
pre-period imbalance is corrected	`[0.5, 12.5, 15.5, 3.0]`	`[2.0, 14.0, 14.0, 0.0]`	Failed
perfectly correlated covariate	`[0.5, 4.25, 5.75, 1.5]`	`[2.0, 5.0, 5.0, 0.0]`	Failed
constant covariate leaves means unadjusted	`[0.0, 2.0, 3.0, 1.0]`	`[0.0, 2.0, 3.0, 1.0]`	Passed
arms of different size	`[0.528169, 4.528169, 7.235915, 2.707746]`	`[1.851852, 5.851852, 6.574074, 0.722222]`	Failed
covariate balanced across arms	`[0.8, 2.0, 3.0, 1.0]`	`[1.0, 2.0, 3.0, 1.0]`	Failed
pre/post sample 1	`[0.522472, 10.369382, 8.261236, -2.108146]`	`[1.319149, 10.170213, 8.659574, -1.510638]`	Failed
pre/post sample 2	`[0.558271, 9.642935, 4.614239, -5.028697]`	`[1.376593, 6.676516, 6.39409, -0.282426]`	Failed
pre/post sample 3	`[0.469565, 6.583437, 13.562422, 6.978986]`	`[1.748201, 9.079822, 11.690134, 2.610312]`	Failed

SHA-256 / 947c3919243ea8a3e71c0d2db1997294b34f19d5c8bb2f7c7756910d99253e10

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(y_c, x_c, y_t, x_t):
    ys = y_c + y_t
    xs = x_c + x_t
    n = len(xs)
    mx, my = sum(xs) / n, sum(ys) / n
    vx = sum((x - mx) ** 2 for x in xs)
    cxy = sum((x - mx) * (y - my) for x, y in zip(xs, ys))
    theta = vx / cxy if cxy else 0.0
    def adj(ys_, xs_):
        mean_x = sum(xs_) / len(xs_)
        return sum(ys_) / len(ys_) - theta * (mean_x - mx)
    ac, at = adj(y_c, x_c), adj(y_t, x_t)
    return [round(theta, 6), round(ac, 6), round(at, 6), round(at - ac, 6)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 1',
   [[8, 8, 10, 16], [3, 5, 7, 8], [6, 10], [3, 7]],
   [1.319149, 10.170213, 8.659574, -1.510638]),
  ('pre/post sample 2',
   [[9, 8, 18], [8, 8, 8], [0, 3, 11, 1, 2], [0, 2, 8, 0, 1]],
   [1.376593, 6.676516, 6.39409, -0.282426]),
  ('pre/post sample 3',
   [[4, 12, 1], [1, 5, 1], [6, 15, 20, 16], [0, 9, 8, 6]],
   [1.748201, 9.079822, 11.690134, 2.610312])],
 [('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 4',
   [[9, 8, 8, 3, 8], [6, 4, 7, 1, 5], [10, 20, 12, 6, 6], [3, 8, 4, 0, 0]],
   [1.131285, 6.294972, 11.705028, 5.410056]),
  ('pre/post sample 5',
   [[14, 4, 11, 5], [6, 4, 9, 4], [9, 1, 6, 8, 6], [7, 0, 3, 7, 3]],
   [1.145522, 7.386298, 6.890962, -0.495336]),
  ('pre/post sample 6',
   [[9, 12, 4, 8, 10], [7, 6, 3, 7, 4], [10, 12], [4, 9]],
   [0.713542, 8.824256, 10.43936, 1.615104])],
 [('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 11', [[10, 16], [8, 7], [7, 19], [6, 9]], [3.0, 13.0, 13.0, 0.0]),
  ('pre/post sample 12', [[10, 10], [4, 7], [10, 9], [6, 3]], [0.2, 9.9, 9.6, -0.3]),
  ('pre/post sample 13', [[8, 5, 9, 7], [6, 5, 7, 5], [16, 11], [5, 8]], [0.375, 7.34375, 13.3125, 5.96875])],
 [('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 16',
   [[18, 4, 1, 10], [8, 1, 0, 5], [16, 13, 15, 3, 7], [5, 8, 5, 0, 2]],
   [1.789106, 8.746974, 10.402421, 1.655447]),
  ('pre/post sample 18',
   [[8, 21], [7, 9], [9, 11, 4], [6, 7, 0]],
   [1.465812, 11.275214, 10.149858, -1.125356]),
  ('pre/post sample 19',
   [[10, 20, 2, 11, 4], [9, 9, 1, 5, 3], [12, 18, 10], [9, 7, 4]],
   [1.523364, 10.123598, 12.127336, 2.003738])],
 [('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 21', [[3, 12], [2, 5], [21, 16], [9, 6]], [2.6, 12.7, 13.3, 0.6]),
  ('pre/post sample 25', [[2, 7], [1, 3], [19, 5], [7, 2]], [2.831325, 8.039157, 8.460843, 0.421687]),
  ('pre/post sample 26',
   [[11, 8], [5, 7], [7, 15, 12, 10], [2, 9, 4, 7]],
   [0.670213, 9.276596, 11.111702, 1.835106])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
pre-period imbalance is corrected	`[0.5, 12.5, 15.5, 3.0]`	`[2.0, 14.0, 14.0, 0.0]`	Failed
perfectly correlated covariate	`[0.5, 4.25, 5.75, 1.5]`	`[2.0, 5.0, 5.0, 0.0]`	Failed
constant covariate leaves means unadjusted	`[0.0, 2.0, 3.0, 1.0]`	`[0.0, 2.0, 3.0, 1.0]`	Passed
arms of different size	`[0.54, 4.54, 7.23, 2.69]`	`[1.851852, 5.851852, 6.574074, 0.722222]`	Failed
covariate balanced across arms	`[1.0, 2.0, 3.0, 1.0]`	`[1.0, 2.0, 3.0, 1.0]`	Passed
pre/post sample 1	`[0.758065, 10.310484, 8.379032, -1.931452]`	`[1.319149, 10.170213, 8.659574, -1.510638]`	Failed
pre/post sample 2	`[0.726431, 9.033354, 4.979987, -4.053367]`	`[1.376593, 6.676516, 6.39409, -0.282426]`	Failed
pre/post sample 3	`[0.572016, 6.783461, 13.412404, 6.628944]`	`[1.748201, 9.079822, 11.690134, 2.610312]`	Failed

SHA-256 / 87a6fc0af6c211c4d26a30393c6789bf786e1151957c68cb5757b2aa1d3d93d0

3 / The verified repair

Exit 0

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(y_c, x_c, y_t, x_t):
    ys = y_c + y_t
    xs = x_c + x_t
    n = len(xs)
    mx, my = sum(xs) / n, sum(ys) / n
    vx = sum((x - mx) ** 2 for x in xs)
    cxy = sum((x - mx) * (y - my) for x, y in zip(xs, ys))
    theta = cxy / vx if vx else 0.0
    def adj(ys_, xs_):
        mean_x = sum(xs_) / len(xs_)
        return sum(ys_) / len(ys_) - theta * (mean_x - mx)
    ac, at = adj(y_c, x_c), adj(y_t, x_t)
    return [round(theta, 6), round(ac, 6), round(at, 6), round(at - ac, 6)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 1',
   [[8, 8, 10, 16], [3, 5, 7, 8], [6, 10], [3, 7]],
   [1.319149, 10.170213, 8.659574, -1.510638]),
  ('pre/post sample 2',
   [[9, 8, 18], [8, 8, 8], [0, 3, 11, 1, 2], [0, 2, 8, 0, 1]],
   [1.376593, 6.676516, 6.39409, -0.282426]),
  ('pre/post sample 3',
   [[4, 12, 1], [1, 5, 1], [6, 15, 20, 16], [0, 9, 8, 6]],
   [1.748201, 9.079822, 11.690134, 2.610312])],
 [('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 4',
   [[9, 8, 8, 3, 8], [6, 4, 7, 1, 5], [10, 20, 12, 6, 6], [3, 8, 4, 0, 0]],
   [1.131285, 6.294972, 11.705028, 5.410056]),
  ('pre/post sample 5',
   [[14, 4, 11, 5], [6, 4, 9, 4], [9, 1, 6, 8, 6], [7, 0, 3, 7, 3]],
   [1.145522, 7.386298, 6.890962, -0.495336]),
  ('pre/post sample 6',
   [[9, 12, 4, 8, 10], [7, 6, 3, 7, 4], [10, 12], [4, 9]],
   [0.713542, 8.824256, 10.43936, 1.615104])],
 [('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 11', [[10, 16], [8, 7], [7, 19], [6, 9]], [3.0, 13.0, 13.0, 0.0]),
  ('pre/post sample 12', [[10, 10], [4, 7], [10, 9], [6, 3]], [0.2, 9.9, 9.6, -0.3]),
  ('pre/post sample 13', [[8, 5, 9, 7], [6, 5, 7, 5], [16, 11], [5, 8]], [0.375, 7.34375, 13.3125, 5.96875])],
 [('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 16',
   [[18, 4, 1, 10], [8, 1, 0, 5], [16, 13, 15, 3, 7], [5, 8, 5, 0, 2]],
   [1.789106, 8.746974, 10.402421, 1.655447]),
  ('pre/post sample 18',
   [[8, 21], [7, 9], [9, 11, 4], [6, 7, 0]],
   [1.465812, 11.275214, 10.149858, -1.125356]),
  ('pre/post sample 19',
   [[10, 20, 2, 11, 4], [9, 9, 1, 5, 3], [12, 18, 10], [9, 7, 4]],
   [1.523364, 10.123598, 12.127336, 2.003738])],
 [('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 21', [[3, 12], [2, 5], [21, 16], [9, 6]], [2.6, 12.7, 13.3, 0.6]),
  ('pre/post sample 25', [[2, 7], [1, 3], [19, 5], [7, 2]], [2.831325, 8.039157, 8.460843, 0.421687]),
  ('pre/post sample 26',
   [[11, 8], [5, 7], [7, 15, 12, 10], [2, 9, 4, 7]],
   [0.670213, 9.276596, 11.111702, 1.835106])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
pre-period imbalance is corrected	`[2.0, 14.0, 14.0, 0.0]`	`[2.0, 14.0, 14.0, 0.0]`	Passed
perfectly correlated covariate	`[2.0, 5.0, 5.0, 0.0]`	`[2.0, 5.0, 5.0, 0.0]`	Passed
constant covariate leaves means unadjusted	`[0.0, 2.0, 3.0, 1.0]`	`[0.0, 2.0, 3.0, 1.0]`	Passed
arms of different size	`[1.851852, 5.851852, 6.574074, 0.722222]`	`[1.851852, 5.851852, 6.574074, 0.722222]`	Passed
covariate balanced across arms	`[1.0, 2.0, 3.0, 1.0]`	`[1.0, 2.0, 3.0, 1.0]`	Passed
pre/post sample 1	`[1.319149, 10.170213, 8.659574, -1.510638]`	`[1.319149, 10.170213, 8.659574, -1.510638]`	Passed
pre/post sample 2	`[1.376593, 6.676516, 6.39409, -0.282426]`	`[1.376593, 6.676516, 6.39409, -0.282426]`	Passed
pre/post sample 3	`[1.748201, 9.079822, 11.690134, 2.610312]`	`[1.748201, 9.079822, 11.690134, 2.610312]`	Passed

SHA-256 / eb44c1d73046ba31416082934f003c1637797598798fdbe47a28dfbdeb07459a

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:56.673213+00:00.

Case digest / 570003e7e53e282129465b779cf6bc8f0f2085cf793538c7d97da655270c7e63