FAILURE MAP
← Case archive

FA-74436 / Experiment statistics / Open access

CUPED covariate adjustment: Covariates are centred on their own arm mean · case 01

The adjustment cancels out and CUPED leaves the estimate unchanged.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

Each unit is adjusted by x - mean(x_arm), whose average is zero within the arm.

THE FAILURE

Each unit is adjusted by x - mean(x_arm), whose average is zero within the arm.

Unsuccessful approach: Centring on the control mean biases the adjusted control arm away from zero adjustment.

Case contract

theta = cov(y, x) / var(x) over the pooled data of both arms (theta = 0 when x is constant). Each arm's adjusted mean is mean(y_arm) - theta * (mean(x_arm) - pooled mean(x)). Return [theta, adjusted control, adjusted treatment, difference] rounded to 6 places.

Why this case matters

CUPED uses pre-period data to cut variance; mistakes either waste it or bias the estimate.

1 / The failure

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(y_c, x_c, y_t, x_t):
    ys = y_c + y_t
    xs = x_c + x_t
    n = len(xs)
    mx, my = sum(xs) / n, sum(ys) / n
    vx = sum((x - mx) ** 2 for x in xs)
    cxy = sum((x - mx) * (y - my) for x, y in zip(xs, ys))
    theta = cxy / vx if vx else 0.0
    def adj(ys_, xs_):
        mean_x = sum(xs_) / len(xs_)
        return sum(y - theta * (x - mean_x) for x, y in zip(xs_, ys_)) / len(ys_)
    ac, at = adj(y_c, x_c), adj(y_t, x_t)
    return [round(theta, 6), round(ac, 6), round(at, 6), round(at - ac, 6)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 1',
   [[8, 8, 10, 16], [3, 5, 7, 8], [6, 10], [3, 7]],
   [1.319149, 10.170213, 8.659574, -1.510638]),
  ('pre/post sample 2',
   [[9, 8, 18], [8, 8, 8], [0, 3, 11, 1, 2], [0, 2, 8, 0, 1]],
   [1.376593, 6.676516, 6.39409, -0.282426]),
  ('pre/post sample 3',
   [[4, 12, 1], [1, 5, 1], [6, 15, 20, 16], [0, 9, 8, 6]],
   [1.748201, 9.079822, 11.690134, 2.610312])],
 [('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 5',
   [[14, 4, 11, 5], [6, 4, 9, 4], [9, 1, 6, 8, 6], [7, 0, 3, 7, 3]],
   [1.145522, 7.386298, 6.890962, -0.495336]),
  ('pre/post sample 6',
   [[9, 12, 4, 8, 10], [7, 6, 3, 7, 4], [10, 12], [4, 9]],
   [0.713542, 8.824256, 10.43936, 1.615104]),
  ('pre/post sample 7',
   [[10, 6, 11, 10], [7, 3, 5, 4], [12, 6, 12, 8], [9, 2, 4, 6]],
   [0.666667, 9.416667, 9.333333, -0.083333])],
 [('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 11', [[10, 16], [8, 7], [7, 19], [6, 9]], [3.0, 13.0, 13.0, 0.0]),
  ('pre/post sample 12', [[10, 10], [4, 7], [10, 9], [6, 3]], [0.2, 9.9, 9.6, -0.3]),
  ('pre/post sample 13', [[8, 5, 9, 7], [6, 5, 7, 5], [16, 11], [5, 8]], [0.375, 7.34375, 13.3125, 5.96875])],
 [('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 16',
   [[18, 4, 1, 10], [8, 1, 0, 5], [16, 13, 15, 3, 7], [5, 8, 5, 0, 2]],
   [1.789106, 8.746974, 10.402421, 1.655447]),
  ('pre/post sample 17',
   [[10, 10, 3], [7, 9, 3], [5, 12, 7], [3, 9, 3]],
   [1.029412, 6.980392, 8.686275, 1.705882]),
  ('pre/post sample 21', [[3, 12], [2, 5], [21, 16], [9, 6]], [2.6, 12.7, 13.3, 0.6])],
 [('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 21', [[3, 12], [2, 5], [21, 16], [9, 6]], [2.6, 12.7, 13.3, 0.6]),
  ('pre/post sample 22',
   [[12, 5, 5, 2], [9, 2, 4, 1], [9, 8, 10, 9], [3, 5, 8, 4]],
   [0.981481, 6.490741, 8.509259, 2.018519]),
  ('pre/post sample 28',
   [[18, 6, 7, 19, 12], [9, 5, 3, 8, 6], [6, 24, 10, 4], [0, 9, 3, 0]],
   [1.773437, 9.877778, 14.152778, 4.275])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
pre-period imbalance is corrected[2.0, 12.0, 16.0, 4.0][2.0, 14.0, 14.0, 0.0]Failed
perfectly correlated covariate[2.0, 4.0, 6.0, 2.0][2.0, 5.0, 5.0, 0.0]Failed
constant covariate leaves means unadjusted[0.0, 2.0, 3.0, 1.0][0.0, 2.0, 3.0, 1.0]Passed
arms of different size[1.851852, 4.0, 7.5, 3.5][1.851852, 5.851852, 6.574074, 0.722222]Failed
covariate balanced across arms[1.0, 2.0, 3.0, 1.0][1.0, 2.0, 3.0, 1.0]Passed
pre/post sample 1[1.319149, 10.5, 8.0, -2.5][1.319149, 10.170213, 8.659574, -1.510638]Failed
pre/post sample 2[1.376593, 11.666667, 3.4, -8.266667][1.376593, 6.676516, 6.39409, -0.282426]Failed
pre/post sample 3[1.748201, 5.666667, 14.25, 8.583333][1.748201, 9.079822, 11.690134, 2.610312]Failed

SHA-256 / eef028d7e8500996f11891692a0bee70e38e35ceffb5981bfe0670e6189d7b83

2 / The unsuccessful fix

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(y_c, x_c, y_t, x_t):
    ys = y_c + y_t
    xs = x_c + x_t
    n = len(xs)
    mx, my = sum(xs) / n, sum(ys) / n
    vx = sum((x - mx) ** 2 for x in xs)
    cxy = sum((x - mx) * (y - my) for x, y in zip(xs, ys))
    theta = cxy / vx if vx else 0.0
    def adj(ys_, xs_):
        mean_x = sum(xs_) / len(xs_)
        return sum(ys_) / len(ys_) - theta * (mean_x - sum(x_c) / len(x_c))
    ac, at = adj(y_c, x_c), adj(y_t, x_t)
    return [round(theta, 6), round(ac, 6), round(at, 6), round(at - ac, 6)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 1',
   [[8, 8, 10, 16], [3, 5, 7, 8], [6, 10], [3, 7]],
   [1.319149, 10.170213, 8.659574, -1.510638]),
  ('pre/post sample 2',
   [[9, 8, 18], [8, 8, 8], [0, 3, 11, 1, 2], [0, 2, 8, 0, 1]],
   [1.376593, 6.676516, 6.39409, -0.282426]),
  ('pre/post sample 3',
   [[4, 12, 1], [1, 5, 1], [6, 15, 20, 16], [0, 9, 8, 6]],
   [1.748201, 9.079822, 11.690134, 2.610312])],
 [('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 5',
   [[14, 4, 11, 5], [6, 4, 9, 4], [9, 1, 6, 8, 6], [7, 0, 3, 7, 3]],
   [1.145522, 7.386298, 6.890962, -0.495336]),
  ('pre/post sample 6',
   [[9, 12, 4, 8, 10], [7, 6, 3, 7, 4], [10, 12], [4, 9]],
   [0.713542, 8.824256, 10.43936, 1.615104]),
  ('pre/post sample 7',
   [[10, 6, 11, 10], [7, 3, 5, 4], [12, 6, 12, 8], [9, 2, 4, 6]],
   [0.666667, 9.416667, 9.333333, -0.083333])],
 [('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 11', [[10, 16], [8, 7], [7, 19], [6, 9]], [3.0, 13.0, 13.0, 0.0]),
  ('pre/post sample 12', [[10, 10], [4, 7], [10, 9], [6, 3]], [0.2, 9.9, 9.6, -0.3]),
  ('pre/post sample 13', [[8, 5, 9, 7], [6, 5, 7, 5], [16, 11], [5, 8]], [0.375, 7.34375, 13.3125, 5.96875])],
 [('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 16',
   [[18, 4, 1, 10], [8, 1, 0, 5], [16, 13, 15, 3, 7], [5, 8, 5, 0, 2]],
   [1.789106, 8.746974, 10.402421, 1.655447]),
  ('pre/post sample 17',
   [[10, 10, 3], [7, 9, 3], [5, 12, 7], [3, 9, 3]],
   [1.029412, 6.980392, 8.686275, 1.705882]),
  ('pre/post sample 21', [[3, 12], [2, 5], [21, 16], [9, 6]], [2.6, 12.7, 13.3, 0.6])],
 [('pre-period imbalance is corrected',
   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],
   [2.0, 14.0, 14.0, 0.0]),
  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),
  ('constant covariate leaves means unadjusted',
   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],
   [0.0, 2.0, 3.0, 1.0]),
  ('arms of different size',
   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],
   [1.851852, 5.851852, 6.574074, 0.722222]),
  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),
  ('pre/post sample 21', [[3, 12], [2, 5], [21, 16], [9, 6]], [2.6, 12.7, 13.3, 0.6]),
  ('pre/post sample 22',
   [[12, 5, 5, 2], [9, 2, 4, 1], [9, 8, 10, 9], [3, 5, 8, 4]],
   [0.981481, 6.490741, 8.509259, 2.018519]),
  ('pre/post sample 28',
   [[18, 6, 7, 19, 12], [9, 5, 3, 8, 6], [6, 24, 10, 4], [0, 9, 3, 0]],
   [1.773437, 9.877778, 14.152778, 4.275])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
pre-period imbalance is corrected[2.0, 12.0, 12.0, 0.0][2.0, 14.0, 14.0, 0.0]Failed
perfectly correlated covariate[2.0, 4.0, 4.0, 0.0][2.0, 5.0, 5.0, 0.0]Failed
constant covariate leaves means unadjusted[0.0, 2.0, 3.0, 1.0][0.0, 2.0, 3.0, 1.0]Passed
arms of different size[1.851852, 4.0, 4.722222, 0.722222][1.851852, 5.851852, 6.574074, 0.722222]Failed
covariate balanced across arms[1.0, 2.0, 3.0, 1.0][1.0, 2.0, 3.0, 1.0]Passed
pre/post sample 1[1.319149, 10.5, 8.989362, -1.510638][1.319149, 10.170213, 8.659574, -1.510638]Failed
pre/post sample 2[1.376593, 11.666667, 11.384241, -0.282426][1.376593, 6.676516, 6.39409, -0.282426]Failed
pre/post sample 3[1.748201, 5.666667, 8.276978, 2.610312][1.748201, 9.079822, 11.690134, 2.610312]Failed

SHA-256 / 4794f5872bd1fda371b266d4dbb2f2ce8383f7a754f7571b7d29725fafcbe0d6

HELD IN THE MEMBER ARCHIVE

The verified repair and its recorded checks are member-only.

This mechanism has 8 recorded checks per implementation. The open-access tier publishes the failure and the unsuccessful fix; the repaired source that passes every check, and the observations that prove it, are available to members.

Every case sharing this mechanism uses the same contract and the same repair, so this one record is held back for all of them.

Member access is invitation-based. Sign in with your invited account to inspect the repair.

Sign in to the archive ↗

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:56.686652+00:00.

Case digest / ef4d0d7f0d33d3a77e40250fdad83fefe7bba12146e30b0bd1155987e856fb60