FA-74441 / Experiment statistics / Open access
CUPED covariate adjustment: Theta is fitted on the control arm only · case 01
The adjustment ignores treatment data and centres on the control covariate mean.
ROOT CAUSE
The pooled lists are built from the control arm alone.
VERIFIED REPAIR
Fit theta and the centring mean on the pooled data from both arms.
Unsuccessful approach: Fitting on the treatment arm alone is equally one-sided.
Case contract
theta = cov(y, x) / var(x) over the pooled data of both arms (theta = 0 when x is constant). Each arm's adjusted mean is mean(y_arm) - theta * (mean(x_arm) - pooled mean(x)). Return [theta, adjusted control, adjusted treatment, difference] rounded to 6 places.
Why this case matters
CUPED uses pre-period data to cut variance; mistakes either waste it or bias the estimate.
1 / The failure
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(y_c, x_c, y_t, x_t):
ys = y_c
xs = x_c
n = len(xs)
mx, my = sum(xs) / n, sum(ys) / n
vx = sum((x - mx) ** 2 for x in xs)
cxy = sum((x - mx) * (y - my) for x, y in zip(xs, ys))
theta = cxy / vx if vx else 0.0
def adj(ys_, xs_):
mean_x = sum(xs_) / len(xs_)
return sum(ys_) / len(ys_) - theta * (mean_x - mx)
ac, at = adj(y_c, x_c), adj(y_t, x_t)
return [round(theta, 6), round(ac, 6), round(at, 6), round(at - ac, 6)]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('pre-period imbalance is corrected',
[[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
[2.0, 14.0, 14.0, 0.0]),
('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
('constant covariate leaves means unadjusted',
[[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
[0.0, 2.0, 3.0, 1.0]),
('arms of different size',
[[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
[1.851852, 5.851852, 6.574074, 0.722222]),
('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
('pre/post sample 1',
[[8, 8, 10, 16], [3, 5, 7, 8], [6, 10], [3, 7]],
[1.319149, 10.170213, 8.659574, -1.510638]),
('pre/post sample 2',
[[9, 8, 18], [8, 8, 8], [0, 3, 11, 1, 2], [0, 2, 8, 0, 1]],
[1.376593, 6.676516, 6.39409, -0.282426]),
('pre/post sample 3',
[[4, 12, 1], [1, 5, 1], [6, 15, 20, 16], [0, 9, 8, 6]],
[1.748201, 9.079822, 11.690134, 2.610312])],
[('pre-period imbalance is corrected',
[[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
[2.0, 14.0, 14.0, 0.0]),
('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
('constant covariate leaves means unadjusted',
[[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
[0.0, 2.0, 3.0, 1.0]),
('arms of different size',
[[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
[1.851852, 5.851852, 6.574074, 0.722222]),
('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
('pre/post sample 5',
[[14, 4, 11, 5], [6, 4, 9, 4], [9, 1, 6, 8, 6], [7, 0, 3, 7, 3]],
[1.145522, 7.386298, 6.890962, -0.495336]),
('pre/post sample 6',
[[9, 12, 4, 8, 10], [7, 6, 3, 7, 4], [10, 12], [4, 9]],
[0.713542, 8.824256, 10.43936, 1.615104]),
('pre/post sample 7',
[[10, 6, 11, 10], [7, 3, 5, 4], [12, 6, 12, 8], [9, 2, 4, 6]],
[0.666667, 9.416667, 9.333333, -0.083333])],
[('pre-period imbalance is corrected',
[[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
[2.0, 14.0, 14.0, 0.0]),
('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
('constant covariate leaves means unadjusted',
[[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
[0.0, 2.0, 3.0, 1.0]),
('arms of different size',
[[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
[1.851852, 5.851852, 6.574074, 0.722222]),
('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
('pre/post sample 11', [[10, 16], [8, 7], [7, 19], [6, 9]], [3.0, 13.0, 13.0, 0.0]),
('pre/post sample 12', [[10, 10], [4, 7], [10, 9], [6, 3]], [0.2, 9.9, 9.6, -0.3]),
('pre/post sample 13', [[8, 5, 9, 7], [6, 5, 7, 5], [16, 11], [5, 8]], [0.375, 7.34375, 13.3125, 5.96875])],
[('pre-period imbalance is corrected',
[[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
[2.0, 14.0, 14.0, 0.0]),
('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
('constant covariate leaves means unadjusted',
[[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
[0.0, 2.0, 3.0, 1.0]),
('arms of different size',
[[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
[1.851852, 5.851852, 6.574074, 0.722222]),
('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
('pre/post sample 16',
[[18, 4, 1, 10], [8, 1, 0, 5], [16, 13, 15, 3, 7], [5, 8, 5, 0, 2]],
[1.789106, 8.746974, 10.402421, 1.655447]),
('pre/post sample 17',
[[10, 10, 3], [7, 9, 3], [5, 12, 7], [3, 9, 3]],
[1.029412, 6.980392, 8.686275, 1.705882]),
('pre/post sample 19',
[[10, 20, 2, 11, 4], [9, 9, 1, 5, 3], [12, 18, 10], [9, 7, 4]],
[1.523364, 10.123598, 12.127336, 2.003738])],
[('pre-period imbalance is corrected',
[[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
[2.0, 14.0, 14.0, 0.0]),
('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
('constant covariate leaves means unadjusted',
[[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
[0.0, 2.0, 3.0, 1.0]),
('arms of different size',
[[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
[1.851852, 5.851852, 6.574074, 0.722222]),
('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
('pre/post sample 21', [[3, 12], [2, 5], [21, 16], [9, 6]], [2.6, 12.7, 13.3, 0.6]),
('pre/post sample 22',
[[12, 5, 5, 2], [9, 2, 4, 1], [9, 8, 10, 9], [3, 5, 8, 4]],
[0.981481, 6.490741, 8.509259, 2.018519]),
('pre/post sample 26',
[[11, 8], [5, 7], [7, 15, 12, 10], [2, 9, 4, 7]],
[0.670213, 9.276596, 11.111702, 1.835106])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| pre-period imbalance is corrected | [2.0, 12.0, 12.0, 0.0] | [2.0, 14.0, 14.0, 0.0] | Failed |
| perfectly correlated covariate | [2.0, 4.0, 4.0, 0.0] | [2.0, 5.0, 5.0, 0.0] | Failed |
| constant covariate leaves means unadjusted | [0.0, 2.0, 3.0, 1.0] | [0.0, 2.0, 3.0, 1.0] | Passed |
| arms of different size | [2.0, 4.0, 4.5, 0.5] | [1.851852, 5.851852, 6.574074, 0.722222] | Failed |
| covariate balanced across arms | [1.0, 2.0, 3.0, 1.0] | [1.0, 2.0, 3.0, 1.0] | Passed |
| pre/post sample 1 | [1.389831, 10.5, 9.042373, -1.457627] | [1.319149, 10.170213, 8.659574, -1.510638] | Failed |
| pre/post sample 2 | [0.0, 11.666667, 3.4, -8.266667] | [1.376593, 6.676516, 6.39409, -0.282426] | Failed |
| pre/post sample 3 | [2.375, 5.666667, 6.135417, 0.46875] | [1.748201, 9.079822, 11.690134, 2.610312] | Failed |
SHA-256 / 91d7216ef0582cf409f70034384fbfae851171ee7fd0d490bdf9ecc2d134d504
2 / The unsuccessful fix
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(y_c, x_c, y_t, x_t):
ys = y_t
xs = x_t
n = len(xs)
mx, my = sum(xs) / n, sum(ys) / n
vx = sum((x - mx) ** 2 for x in xs)
cxy = sum((x - mx) * (y - my) for x, y in zip(xs, ys))
theta = cxy / vx if vx else 0.0
def adj(ys_, xs_):
mean_x = sum(xs_) / len(xs_)
return sum(ys_) / len(ys_) - theta * (mean_x - mx)
ac, at = adj(y_c, x_c), adj(y_t, x_t)
return [round(theta, 6), round(ac, 6), round(at, 6), round(at - ac, 6)]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('pre-period imbalance is corrected',
[[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
[2.0, 14.0, 14.0, 0.0]),
('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
('constant covariate leaves means unadjusted',
[[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
[0.0, 2.0, 3.0, 1.0]),
('arms of different size',
[[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
[1.851852, 5.851852, 6.574074, 0.722222]),
('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
('pre/post sample 1',
[[8, 8, 10, 16], [3, 5, 7, 8], [6, 10], [3, 7]],
[1.319149, 10.170213, 8.659574, -1.510638]),
('pre/post sample 2',
[[9, 8, 18], [8, 8, 8], [0, 3, 11, 1, 2], [0, 2, 8, 0, 1]],
[1.376593, 6.676516, 6.39409, -0.282426]),
('pre/post sample 3',
[[4, 12, 1], [1, 5, 1], [6, 15, 20, 16], [0, 9, 8, 6]],
[1.748201, 9.079822, 11.690134, 2.610312])],
[('pre-period imbalance is corrected',
[[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
[2.0, 14.0, 14.0, 0.0]),
('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
('constant covariate leaves means unadjusted',
[[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
[0.0, 2.0, 3.0, 1.0]),
('arms of different size',
[[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
[1.851852, 5.851852, 6.574074, 0.722222]),
('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
('pre/post sample 5',
[[14, 4, 11, 5], [6, 4, 9, 4], [9, 1, 6, 8, 6], [7, 0, 3, 7, 3]],
[1.145522, 7.386298, 6.890962, -0.495336]),
('pre/post sample 6',
[[9, 12, 4, 8, 10], [7, 6, 3, 7, 4], [10, 12], [4, 9]],
[0.713542, 8.824256, 10.43936, 1.615104]),
('pre/post sample 7',
[[10, 6, 11, 10], [7, 3, 5, 4], [12, 6, 12, 8], [9, 2, 4, 6]],
[0.666667, 9.416667, 9.333333, -0.083333])],
[('pre-period imbalance is corrected',
[[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
[2.0, 14.0, 14.0, 0.0]),
('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
('constant covariate leaves means unadjusted',
[[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
[0.0, 2.0, 3.0, 1.0]),
('arms of different size',
[[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
[1.851852, 5.851852, 6.574074, 0.722222]),
('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
('pre/post sample 11', [[10, 16], [8, 7], [7, 19], [6, 9]], [3.0, 13.0, 13.0, 0.0]),
('pre/post sample 12', [[10, 10], [4, 7], [10, 9], [6, 3]], [0.2, 9.9, 9.6, -0.3]),
('pre/post sample 13', [[8, 5, 9, 7], [6, 5, 7, 5], [16, 11], [5, 8]], [0.375, 7.34375, 13.3125, 5.96875])],
[('pre-period imbalance is corrected',
[[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
[2.0, 14.0, 14.0, 0.0]),
('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
('constant covariate leaves means unadjusted',
[[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
[0.0, 2.0, 3.0, 1.0]),
('arms of different size',
[[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
[1.851852, 5.851852, 6.574074, 0.722222]),
('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
('pre/post sample 16',
[[18, 4, 1, 10], [8, 1, 0, 5], [16, 13, 15, 3, 7], [5, 8, 5, 0, 2]],
[1.789106, 8.746974, 10.402421, 1.655447]),
('pre/post sample 17',
[[10, 10, 3], [7, 9, 3], [5, 12, 7], [3, 9, 3]],
[1.029412, 6.980392, 8.686275, 1.705882]),
('pre/post sample 19',
[[10, 20, 2, 11, 4], [9, 9, 1, 5, 3], [12, 18, 10], [9, 7, 4]],
[1.523364, 10.123598, 12.127336, 2.003738])],
[('pre-period imbalance is corrected',
[[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
[2.0, 14.0, 14.0, 0.0]),
('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
('constant covariate leaves means unadjusted',
[[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
[0.0, 2.0, 3.0, 1.0]),
('arms of different size',
[[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
[1.851852, 5.851852, 6.574074, 0.722222]),
('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
('pre/post sample 21', [[3, 12], [2, 5], [21, 16], [9, 6]], [2.6, 12.7, 13.3, 0.6]),
('pre/post sample 22',
[[12, 5, 5, 2], [9, 2, 4, 1], [9, 8, 10, 9], [3, 5, 8, 4]],
[0.981481, 6.490741, 8.509259, 2.018519]),
('pre/post sample 26',
[[11, 8], [5, 7], [7, 15, 12, 10], [2, 9, 4, 7]],
[0.670213, 9.276596, 11.111702, 1.835106])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| pre-period imbalance is corrected | [2.0, 16.0, 16.0, 0.0] | [2.0, 14.0, 14.0, 0.0] | Failed |
| perfectly correlated covariate | [2.0, 6.0, 6.0, 0.0] | [2.0, 5.0, 5.0, 0.0] | Failed |
| constant covariate leaves means unadjusted | [0.0, 2.0, 3.0, 1.0] | [0.0, 2.0, 3.0, 1.0] | Passed |
| arms of different size | [1.7, 6.55, 7.5, 0.95] | [1.851852, 5.851852, 6.574074, 0.722222] | Failed |
| covariate balanced across arms | [1.0, 2.0, 3.0, 1.0] | [1.0, 2.0, 3.0, 1.0] | Passed |
| pre/post sample 1 | [1.0, 9.75, 8.0, -1.75] | [1.319149, 10.170213, 8.659574, -1.510638] | Failed |
| pre/post sample 2 | [1.308036, 4.08006, 3.4, -0.68006] | [1.376593, 6.676516, 6.39409, -0.282426] | Failed |
| pre/post sample 3 | [1.297436, 10.099573, 14.25, 4.150427] | [1.748201, 9.079822, 11.690134, 2.610312] | Failed |
SHA-256 / 7cdf08528a25ba38eee645b8a992b29d29d1b2a762f212dc6c152246269852b4
3 / The verified repair
Exit 0"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(y_c, x_c, y_t, x_t):
ys = y_c + y_t
xs = x_c + x_t
n = len(xs)
mx, my = sum(xs) / n, sum(ys) / n
vx = sum((x - mx) ** 2 for x in xs)
cxy = sum((x - mx) * (y - my) for x, y in zip(xs, ys))
theta = cxy / vx if vx else 0.0
def adj(ys_, xs_):
mean_x = sum(xs_) / len(xs_)
return sum(ys_) / len(ys_) - theta * (mean_x - mx)
ac, at = adj(y_c, x_c), adj(y_t, x_t)
return [round(theta, 6), round(ac, 6), round(at, 6), round(at - ac, 6)]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('pre-period imbalance is corrected',
[[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
[2.0, 14.0, 14.0, 0.0]),
('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
('constant covariate leaves means unadjusted',
[[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
[0.0, 2.0, 3.0, 1.0]),
('arms of different size',
[[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
[1.851852, 5.851852, 6.574074, 0.722222]),
('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
('pre/post sample 1',
[[8, 8, 10, 16], [3, 5, 7, 8], [6, 10], [3, 7]],
[1.319149, 10.170213, 8.659574, -1.510638]),
('pre/post sample 2',
[[9, 8, 18], [8, 8, 8], [0, 3, 11, 1, 2], [0, 2, 8, 0, 1]],
[1.376593, 6.676516, 6.39409, -0.282426]),
('pre/post sample 3',
[[4, 12, 1], [1, 5, 1], [6, 15, 20, 16], [0, 9, 8, 6]],
[1.748201, 9.079822, 11.690134, 2.610312])],
[('pre-period imbalance is corrected',
[[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
[2.0, 14.0, 14.0, 0.0]),
('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
('constant covariate leaves means unadjusted',
[[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
[0.0, 2.0, 3.0, 1.0]),
('arms of different size',
[[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
[1.851852, 5.851852, 6.574074, 0.722222]),
('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
('pre/post sample 5',
[[14, 4, 11, 5], [6, 4, 9, 4], [9, 1, 6, 8, 6], [7, 0, 3, 7, 3]],
[1.145522, 7.386298, 6.890962, -0.495336]),
('pre/post sample 6',
[[9, 12, 4, 8, 10], [7, 6, 3, 7, 4], [10, 12], [4, 9]],
[0.713542, 8.824256, 10.43936, 1.615104]),
('pre/post sample 7',
[[10, 6, 11, 10], [7, 3, 5, 4], [12, 6, 12, 8], [9, 2, 4, 6]],
[0.666667, 9.416667, 9.333333, -0.083333])],
[('pre-period imbalance is corrected',
[[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
[2.0, 14.0, 14.0, 0.0]),
('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
('constant covariate leaves means unadjusted',
[[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
[0.0, 2.0, 3.0, 1.0]),
('arms of different size',
[[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
[1.851852, 5.851852, 6.574074, 0.722222]),
('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
('pre/post sample 11', [[10, 16], [8, 7], [7, 19], [6, 9]], [3.0, 13.0, 13.0, 0.0]),
('pre/post sample 12', [[10, 10], [4, 7], [10, 9], [6, 3]], [0.2, 9.9, 9.6, -0.3]),
('pre/post sample 13', [[8, 5, 9, 7], [6, 5, 7, 5], [16, 11], [5, 8]], [0.375, 7.34375, 13.3125, 5.96875])],
[('pre-period imbalance is corrected',
[[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
[2.0, 14.0, 14.0, 0.0]),
('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
('constant covariate leaves means unadjusted',
[[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
[0.0, 2.0, 3.0, 1.0]),
('arms of different size',
[[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
[1.851852, 5.851852, 6.574074, 0.722222]),
('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
('pre/post sample 16',
[[18, 4, 1, 10], [8, 1, 0, 5], [16, 13, 15, 3, 7], [5, 8, 5, 0, 2]],
[1.789106, 8.746974, 10.402421, 1.655447]),
('pre/post sample 17',
[[10, 10, 3], [7, 9, 3], [5, 12, 7], [3, 9, 3]],
[1.029412, 6.980392, 8.686275, 1.705882]),
('pre/post sample 19',
[[10, 20, 2, 11, 4], [9, 9, 1, 5, 3], [12, 18, 10], [9, 7, 4]],
[1.523364, 10.123598, 12.127336, 2.003738])],
[('pre-period imbalance is corrected',
[[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
[2.0, 14.0, 14.0, 0.0]),
('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
('constant covariate leaves means unadjusted',
[[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
[0.0, 2.0, 3.0, 1.0]),
('arms of different size',
[[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
[1.851852, 5.851852, 6.574074, 0.722222]),
('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
('pre/post sample 21', [[3, 12], [2, 5], [21, 16], [9, 6]], [2.6, 12.7, 13.3, 0.6]),
('pre/post sample 22',
[[12, 5, 5, 2], [9, 2, 4, 1], [9, 8, 10, 9], [3, 5, 8, 4]],
[0.981481, 6.490741, 8.509259, 2.018519]),
('pre/post sample 26',
[[11, 8], [5, 7], [7, 15, 12, 10], [2, 9, 4, 7]],
[0.670213, 9.276596, 11.111702, 1.835106])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| pre-period imbalance is corrected | [2.0, 14.0, 14.0, 0.0] | [2.0, 14.0, 14.0, 0.0] | Passed |
| perfectly correlated covariate | [2.0, 5.0, 5.0, 0.0] | [2.0, 5.0, 5.0, 0.0] | Passed |
| constant covariate leaves means unadjusted | [0.0, 2.0, 3.0, 1.0] | [0.0, 2.0, 3.0, 1.0] | Passed |
| arms of different size | [1.851852, 5.851852, 6.574074, 0.722222] | [1.851852, 5.851852, 6.574074, 0.722222] | Passed |
| covariate balanced across arms | [1.0, 2.0, 3.0, 1.0] | [1.0, 2.0, 3.0, 1.0] | Passed |
| pre/post sample 1 | [1.319149, 10.170213, 8.659574, -1.510638] | [1.319149, 10.170213, 8.659574, -1.510638] | Passed |
| pre/post sample 2 | [1.376593, 6.676516, 6.39409, -0.282426] | [1.376593, 6.676516, 6.39409, -0.282426] | Passed |
| pre/post sample 3 | [1.748201, 9.079822, 11.690134, 2.610312] | [1.748201, 9.079822, 11.690134, 2.610312] | Passed |
SHA-256 / d84d8a269d558f4adbf0437ed46bab097d72b0ac19651d872ae66af6a94470f0
Verification & scope
A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.
Observations recorded using Python 3.12.14 at 2026-09-29T14:48:56.840758+00:00.
Case digest / b3f937ac868ecd4542a991f32c07338cebd7fc40b6e37a5c8493a5865a2d3265