{"abstract":"The adjustment ignores treatment data and centres on the control covariate mean.","category":"Experiment statistics","checks":8,"contract":"theta = cov(y, x) / var(x) over the pooled data of both arms (theta = 0 when x is constant). Each arm's adjusted mean is mean(y_arm) - theta * (mean(x_arm) - pooled mean(x)). Return [theta, adjusted control, adjusted treatment, difference] rounded to 6 places.","evaluation_group":"w2-experiment-statistics-cuped-adjustment","failed_approach":"Fitting on the treatment arm alone is equally one-sided.","family":"w2-experiment-statistics-cuped-adjustment-estimation-sample","id":"FA-74441","implementations":{"attempt":{"sha256":"7cdf08528a25ba38eee645b8a992b29d29d1b2a762f212dc6c152246269852b4","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(y_c, x_c, y_t, x_t):\n    ys = y_t\n    xs = x_t\n    n = len(xs)\n    mx, my = sum(xs) / n, sum(ys) / n\n    vx = sum((x - mx) ** 2 for x in xs)\n    cxy = sum((x - mx) * (y - my) for x, y in zip(xs, ys))\n    theta = cxy / vx if vx else 0.0\n    def adj(ys_, xs_):\n        mean_x = sum(xs_) / len(xs_)\n        return sum(ys_) / len(ys_) - theta * (mean_x - mx)\n    ac, at = adj(y_c, x_c), adj(y_t, x_t)\n    return [round(theta, 6), round(ac, 6), round(at, 6), round(at - ac, 6)]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('pre-period imbalance is corrected',\n   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],\n   [2.0, 14.0, 14.0, 0.0]),\n  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),\n  ('constant covariate leaves means unadjusted',\n   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],\n   [0.0, 2.0, 3.0, 1.0]),\n  ('arms of different size',\n   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],\n   [1.851852, 5.851852, 6.574074, 0.722222]),\n  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),\n  ('pre/post sample 1',\n   [[8, 8, 10, 16], [3, 5, 7, 8], [6, 10], [3, 7]],\n   [1.319149, 10.170213, 8.659574, -1.510638]),\n  ('pre/post sample 2',\n   [[9, 8, 18], [8, 8, 8], [0, 3, 11, 1, 2], [0, 2, 8, 0, 1]],\n   [1.376593, 6.676516, 6.39409, -0.282426]),\n  ('pre/post sample 3',\n   [[4, 12, 1], [1, 5, 1], [6, 15, 20, 16], [0, 9, 8, 6]],\n   [1.748201, 9.079822, 11.690134, 2.610312])],\n [('pre-period imbalance is corrected',\n   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],\n   [2.0, 14.0, 14.0, 0.0]),\n  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),\n  ('constant covariate leaves means unadjusted',\n   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],\n   [0.0, 2.0, 3.0, 1.0]),\n  ('arms of different size',\n   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],\n   [1.851852, 5.851852, 6.574074, 0.722222]),\n  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),\n  ('pre/post sample 5',\n   [[14, 4, 11, 5], [6, 4, 9, 4], [9, 1, 6, 8, 6], [7, 0, 3, 7, 3]],\n   [1.145522, 7.386298, 6.890962, -0.495336]),\n  ('pre/post sample 6',\n   [[9, 12, 4, 8, 10], [7, 6, 3, 7, 4], [10, 12], [4, 9]],\n   [0.713542, 8.824256, 10.43936, 1.615104]),\n  ('pre/post sample 7',\n   [[10, 6, 11, 10], [7, 3, 5, 4], [12, 6, 12, 8], [9, 2, 4, 6]],\n   [0.666667, 9.416667, 9.333333, -0.083333])],\n [('pre-period imbalance is corrected',\n   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],\n   [2.0, 14.0, 14.0, 0.0]),\n  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),\n  ('constant covariate leaves means unadjusted',\n   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],\n   [0.0, 2.0, 3.0, 1.0]),\n  ('arms of different size',\n   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],\n   [1.851852, 5.851852, 6.574074, 0.722222]),\n  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),\n  ('pre/post sample 11', [[10, 16], [8, 7], [7, 19], [6, 9]], [3.0, 13.0, 13.0, 0.0]),\n  ('pre/post sample 12', [[10, 10], [4, 7], [10, 9], [6, 3]], [0.2, 9.9, 9.6, -0.3]),\n  ('pre/post sample 13', [[8, 5, 9, 7], [6, 5, 7, 5], [16, 11], [5, 8]], [0.375, 7.34375, 13.3125, 5.96875])],\n [('pre-period imbalance is corrected',\n   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],\n   [2.0, 14.0, 14.0, 0.0]),\n  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),\n  ('constant covariate leaves means unadjusted',\n   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],\n   [0.0, 2.0, 3.0, 1.0]),\n  ('arms of different size',\n   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],\n   [1.851852, 5.851852, 6.574074, 0.722222]),\n  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),\n  ('pre/post sample 16',\n   [[18, 4, 1, 10], [8, 1, 0, 5], [16, 13, 15, 3, 7], [5, 8, 5, 0, 2]],\n   [1.789106, 8.746974, 10.402421, 1.655447]),\n  ('pre/post sample 17',\n   [[10, 10, 3], [7, 9, 3], [5, 12, 7], [3, 9, 3]],\n   [1.029412, 6.980392, 8.686275, 1.705882]),\n  ('pre/post sample 19',\n   [[10, 20, 2, 11, 4], [9, 9, 1, 5, 3], [12, 18, 10], [9, 7, 4]],\n   [1.523364, 10.123598, 12.127336, 2.003738])],\n [('pre-period imbalance is corrected',\n   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],\n   [2.0, 14.0, 14.0, 0.0]),\n  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),\n  ('constant covariate leaves means unadjusted',\n   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],\n   [0.0, 2.0, 3.0, 1.0]),\n  ('arms of different size',\n   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],\n   [1.851852, 5.851852, 6.574074, 0.722222]),\n  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),\n  ('pre/post sample 21', [[3, 12], [2, 5], [21, 16], [9, 6]], [2.6, 12.7, 13.3, 0.6]),\n  ('pre/post sample 22',\n   [[12, 5, 5, 2], [9, 2, 4, 1], [9, 8, 10, 9], [3, 5, 8, 4]],\n   [0.981481, 6.490741, 8.509259, 2.018519]),\n  ('pre/post sample 26',\n   [[11, 8], [5, 7], [7, 15, 12, 10], [2, 9, 4, 7]],\n   [0.670213, 9.276596, 11.111702, 1.835106])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"},"broken":{"sha256":"91d7216ef0582cf409f70034384fbfae851171ee7fd0d490bdf9ecc2d134d504","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(y_c, x_c, y_t, x_t):\n    ys = y_c\n    xs = x_c\n    n = len(xs)\n    mx, my = sum(xs) / n, sum(ys) / n\n    vx = sum((x - mx) ** 2 for x in xs)\n    cxy = sum((x - mx) * (y - my) for x, y in zip(xs, ys))\n    theta = cxy / vx if vx else 0.0\n    def adj(ys_, xs_):\n        mean_x = sum(xs_) / len(xs_)\n        return sum(ys_) / len(ys_) - theta * (mean_x - mx)\n    ac, at = adj(y_c, x_c), adj(y_t, x_t)\n    return [round(theta, 6), round(ac, 6), round(at, 6), round(at - ac, 6)]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('pre-period imbalance is corrected',\n   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],\n   [2.0, 14.0, 14.0, 0.0]),\n  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),\n  ('constant covariate leaves means unadjusted',\n   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],\n   [0.0, 2.0, 3.0, 1.0]),\n  ('arms of different size',\n   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],\n   [1.851852, 5.851852, 6.574074, 0.722222]),\n  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),\n  ('pre/post sample 1',\n   [[8, 8, 10, 16], [3, 5, 7, 8], [6, 10], [3, 7]],\n   [1.319149, 10.170213, 8.659574, -1.510638]),\n  ('pre/post sample 2',\n   [[9, 8, 18], [8, 8, 8], [0, 3, 11, 1, 2], [0, 2, 8, 0, 1]],\n   [1.376593, 6.676516, 6.39409, -0.282426]),\n  ('pre/post sample 3',\n   [[4, 12, 1], [1, 5, 1], [6, 15, 20, 16], [0, 9, 8, 6]],\n   [1.748201, 9.079822, 11.690134, 2.610312])],\n [('pre-period imbalance is corrected',\n   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],\n   [2.0, 14.0, 14.0, 0.0]),\n  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),\n  ('constant covariate leaves means unadjusted',\n   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],\n   [0.0, 2.0, 3.0, 1.0]),\n  ('arms of different size',\n   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],\n   [1.851852, 5.851852, 6.574074, 0.722222]),\n  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),\n  ('pre/post sample 5',\n   [[14, 4, 11, 5], [6, 4, 9, 4], [9, 1, 6, 8, 6], [7, 0, 3, 7, 3]],\n   [1.145522, 7.386298, 6.890962, -0.495336]),\n  ('pre/post sample 6',\n   [[9, 12, 4, 8, 10], [7, 6, 3, 7, 4], [10, 12], [4, 9]],\n   [0.713542, 8.824256, 10.43936, 1.615104]),\n  ('pre/post sample 7',\n   [[10, 6, 11, 10], [7, 3, 5, 4], [12, 6, 12, 8], [9, 2, 4, 6]],\n   [0.666667, 9.416667, 9.333333, -0.083333])],\n [('pre-period imbalance is corrected',\n   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],\n   [2.0, 14.0, 14.0, 0.0]),\n  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),\n  ('constant covariate leaves means unadjusted',\n   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],\n   [0.0, 2.0, 3.0, 1.0]),\n  ('arms of different size',\n   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],\n   [1.851852, 5.851852, 6.574074, 0.722222]),\n  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),\n  ('pre/post sample 11', [[10, 16], [8, 7], [7, 19], [6, 9]], [3.0, 13.0, 13.0, 0.0]),\n  ('pre/post sample 12', [[10, 10], [4, 7], [10, 9], [6, 3]], [0.2, 9.9, 9.6, -0.3]),\n  ('pre/post sample 13', [[8, 5, 9, 7], [6, 5, 7, 5], [16, 11], [5, 8]], [0.375, 7.34375, 13.3125, 5.96875])],\n [('pre-period imbalance is corrected',\n   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],\n   [2.0, 14.0, 14.0, 0.0]),\n  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),\n  ('constant covariate leaves means unadjusted',\n   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],\n   [0.0, 2.0, 3.0, 1.0]),\n  ('arms of different size',\n   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],\n   [1.851852, 5.851852, 6.574074, 0.722222]),\n  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),\n  ('pre/post sample 16',\n   [[18, 4, 1, 10], [8, 1, 0, 5], [16, 13, 15, 3, 7], [5, 8, 5, 0, 2]],\n   [1.789106, 8.746974, 10.402421, 1.655447]),\n  ('pre/post sample 17',\n   [[10, 10, 3], [7, 9, 3], [5, 12, 7], [3, 9, 3]],\n   [1.029412, 6.980392, 8.686275, 1.705882]),\n  ('pre/post sample 19',\n   [[10, 20, 2, 11, 4], [9, 9, 1, 5, 3], [12, 18, 10], [9, 7, 4]],\n   [1.523364, 10.123598, 12.127336, 2.003738])],\n [('pre-period imbalance is corrected',\n   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],\n   [2.0, 14.0, 14.0, 0.0]),\n  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),\n  ('constant covariate leaves means unadjusted',\n   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],\n   [0.0, 2.0, 3.0, 1.0]),\n  ('arms of different size',\n   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],\n   [1.851852, 5.851852, 6.574074, 0.722222]),\n  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),\n  ('pre/post sample 21', [[3, 12], [2, 5], [21, 16], [9, 6]], [2.6, 12.7, 13.3, 0.6]),\n  ('pre/post sample 22',\n   [[12, 5, 5, 2], [9, 2, 4, 1], [9, 8, 10, 9], [3, 5, 8, 4]],\n   [0.981481, 6.490741, 8.509259, 2.018519]),\n  ('pre/post sample 26',\n   [[11, 8], [5, 7], [7, 15, 12, 10], [2, 9, 4, 7]],\n   [0.670213, 9.276596, 11.111702, 1.835106])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"},"fixed":{"sha256":"d84d8a269d558f4adbf0437ed46bab097d72b0ac19651d872ae66af6a94470f0","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(y_c, x_c, y_t, x_t):\n    ys = y_c + y_t\n    xs = x_c + x_t\n    n = len(xs)\n    mx, my = sum(xs) / n, sum(ys) / n\n    vx = sum((x - mx) ** 2 for x in xs)\n    cxy = sum((x - mx) * (y - my) for x, y in zip(xs, ys))\n    theta = cxy / vx if vx else 0.0\n    def adj(ys_, xs_):\n        mean_x = sum(xs_) / len(xs_)\n        return sum(ys_) / len(ys_) - theta * (mean_x - mx)\n    ac, at = adj(y_c, x_c), adj(y_t, x_t)\n    return [round(theta, 6), round(ac, 6), round(at, 6), round(at - ac, 6)]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('pre-period imbalance is corrected',\n   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],\n   [2.0, 14.0, 14.0, 0.0]),\n  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),\n  ('constant covariate leaves means unadjusted',\n   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],\n   [0.0, 2.0, 3.0, 1.0]),\n  ('arms of different size',\n   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],\n   [1.851852, 5.851852, 6.574074, 0.722222]),\n  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),\n  ('pre/post sample 1',\n   [[8, 8, 10, 16], [3, 5, 7, 8], [6, 10], [3, 7]],\n   [1.319149, 10.170213, 8.659574, -1.510638]),\n  ('pre/post sample 2',\n   [[9, 8, 18], [8, 8, 8], [0, 3, 11, 1, 2], [0, 2, 8, 0, 1]],\n   [1.376593, 6.676516, 6.39409, -0.282426]),\n  ('pre/post sample 3',\n   [[4, 12, 1], [1, 5, 1], [6, 15, 20, 16], [0, 9, 8, 6]],\n   [1.748201, 9.079822, 11.690134, 2.610312])],\n [('pre-period imbalance is corrected',\n   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],\n   [2.0, 14.0, 14.0, 0.0]),\n  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),\n  ('constant covariate leaves means unadjusted',\n   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],\n   [0.0, 2.0, 3.0, 1.0]),\n  ('arms of different size',\n   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],\n   [1.851852, 5.851852, 6.574074, 0.722222]),\n  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),\n  ('pre/post sample 5',\n   [[14, 4, 11, 5], [6, 4, 9, 4], [9, 1, 6, 8, 6], [7, 0, 3, 7, 3]],\n   [1.145522, 7.386298, 6.890962, -0.495336]),\n  ('pre/post sample 6',\n   [[9, 12, 4, 8, 10], [7, 6, 3, 7, 4], [10, 12], [4, 9]],\n   [0.713542, 8.824256, 10.43936, 1.615104]),\n  ('pre/post sample 7',\n   [[10, 6, 11, 10], [7, 3, 5, 4], [12, 6, 12, 8], [9, 2, 4, 6]],\n   [0.666667, 9.416667, 9.333333, -0.083333])],\n [('pre-period imbalance is corrected',\n   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],\n   [2.0, 14.0, 14.0, 0.0]),\n  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),\n  ('constant covariate leaves means unadjusted',\n   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],\n   [0.0, 2.0, 3.0, 1.0]),\n  ('arms of different size',\n   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],\n   [1.851852, 5.851852, 6.574074, 0.722222]),\n  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),\n  ('pre/post sample 11', [[10, 16], [8, 7], [7, 19], [6, 9]], [3.0, 13.0, 13.0, 0.0]),\n  ('pre/post sample 12', [[10, 10], [4, 7], [10, 9], [6, 3]], [0.2, 9.9, 9.6, -0.3]),\n  ('pre/post sample 13', [[8, 5, 9, 7], [6, 5, 7, 5], [16, 11], [5, 8]], [0.375, 7.34375, 13.3125, 5.96875])],\n [('pre-period imbalance is corrected',\n   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],\n   [2.0, 14.0, 14.0, 0.0]),\n  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),\n  ('constant covariate leaves means unadjusted',\n   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],\n   [0.0, 2.0, 3.0, 1.0]),\n  ('arms of different size',\n   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],\n   [1.851852, 5.851852, 6.574074, 0.722222]),\n  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),\n  ('pre/post sample 16',\n   [[18, 4, 1, 10], [8, 1, 0, 5], [16, 13, 15, 3, 7], [5, 8, 5, 0, 2]],\n   [1.789106, 8.746974, 10.402421, 1.655447]),\n  ('pre/post sample 17',\n   [[10, 10, 3], [7, 9, 3], [5, 12, 7], [3, 9, 3]],\n   [1.029412, 6.980392, 8.686275, 1.705882]),\n  ('pre/post sample 19',\n   [[10, 20, 2, 11, 4], [9, 9, 1, 5, 3], [12, 18, 10], [9, 7, 4]],\n   [1.523364, 10.123598, 12.127336, 2.003738])],\n [('pre-period imbalance is corrected',\n   [[10, 12, 14], [5, 6, 7], [14, 16, 18], [7, 8, 9]],\n   [2.0, 14.0, 14.0, 0.0]),\n  ('perfectly correlated covariate', [[2, 4, 6], [1, 2, 3], [4, 6, 8], [2, 3, 4]], [2.0, 5.0, 5.0, 0.0]),\n  ('constant covariate leaves means unadjusted',\n   [[1, 2, 3], [5, 5, 5], [2, 3, 4], [5, 5, 5]],\n   [0.0, 2.0, 3.0, 1.0]),\n  ('arms of different size',\n   [[3, 5], [1, 2], [4, 6, 9, 11], [1, 2, 4, 5]],\n   [1.851852, 5.851852, 6.574074, 0.722222]),\n  ('covariate balanced across arms', [[1, 3], [1, 3], [2, 4], [1, 3]], [1.0, 2.0, 3.0, 1.0]),\n  ('pre/post sample 21', [[3, 12], [2, 5], [21, 16], [9, 6]], [2.6, 12.7, 13.3, 0.6]),\n  ('pre/post sample 22',\n   [[12, 5, 5, 2], [9, 2, 4, 1], [9, 8, 10, 9], [3, 5, 8, 4]],\n   [0.981481, 6.490741, 8.509259, 2.018519]),\n  ('pre/post sample 26',\n   [[11, 8], [5, 7], [7, 15, 12, 10], [2, 9, 4, 7]],\n   [0.670213, 9.276596, 11.111702, 1.835106])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"}},"limitations":"A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.","method":"Deterministic executable model with adversarial boundary fixtures.","provenance":{"created_by":"Failure Map","dependencies":"Python standard library","family":"w2-experiment-statistics-cuped-adjustment-estimation-sample","generated_at":"2026-09-29T14:48:56.840758+00:00","license":"CC0-1.0","python":"3.12.14","seed":1,"split":"open-access"},"relevance":"CUPED uses pre-period data to cut variance; mistakes either waste it or bias the estimate.","repair":"Fit theta and the centring mean on the pooled data from both arms.","root_cause":"The pooled lists are built from the control arm alone.","sha256":"b3f937ac868ecd4542a991f32c07338cebd7fc40b6e37a5c8493a5865a2d3265","title":"CUPED covariate adjustment: Theta is fitted on the control arm only · case 01","variant":1,"variant_policy":"Five numbered records share a model and may reuse boundary fixtures.","verification":{"attempt":{"elapsed_ms":39.171,"exit_code":1,"observations":[{"actual":[2.0,16.0,16.0,0.0],"check":"pre-period imbalance is corrected","expected":[2.0,14.0,14.0,0.0],"passed":false},{"actual":[2.0,6.0,6.0,0.0],"check":"perfectly correlated covariate","expected":[2.0,5.0,5.0,0.0],"passed":false},{"actual":[0.0,2.0,3.0,1.0],"check":"constant covariate leaves means unadjusted","expected":[0.0,2.0,3.0,1.0],"passed":true},{"actual":[1.7,6.55,7.5,0.95],"check":"arms of different size","expected":[1.851852,5.851852,6.574074,0.722222],"passed":false},{"actual":[1.0,2.0,3.0,1.0],"check":"covariate balanced across arms","expected":[1.0,2.0,3.0,1.0],"passed":true},{"actual":[1.0,9.75,8.0,-1.75],"check":"pre/post sample 1","expected":[1.319149,10.170213,8.659574,-1.510638],"passed":false},{"actual":[1.308036,4.08006,3.4,-0.68006],"check":"pre/post sample 2","expected":[1.376593,6.676516,6.39409,-0.282426],"passed":false},{"actual":[1.297436,10.099573,14.25,4.150427],"check":"pre/post sample 3","expected":[1.748201,9.079822,11.690134,2.610312],"passed":false}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"pre-period imbalance is corrected\", \"actual\": [2.0, 16.0, 16.0, 0.0], \"expected\": [2.0, 14.0, 14.0, 0.0], \"passed\": false}, {\"check\": \"perfectly correlated covariate\", \"actual\": [2.0, 6.0, 6.0, 0.0], \"expected\": [2.0, 5.0, 5.0, 0.0], \"passed\": false}, {\"check\": \"constant covariate leaves means unadjusted\", \"actual\": [0.0, 2.0, 3.0, 1.0], \"expected\": [0.0, 2.0, 3.0, 1.0], \"passed\": true}, {\"check\": \"arms of different size\", \"actual\": [1.7, 6.55, 7.5, 0.95], \"expected\": [1.851852, 5.851852, 6.574074, 0.722222], \"passed\": false}, {\"check\": \"covariate balanced across arms\", \"actual\": [1.0, 2.0, 3.0, 1.0], \"expected\": [1.0, 2.0, 3.0, 1.0], \"passed\": true}, {\"check\": \"pre/post sample 1\", \"actual\": [1.0, 9.75, 8.0, -1.75], \"expected\": [1.319149, 10.170213, 8.659574, -1.510638], \"passed\": false}, {\"check\": \"pre/post sample 2\", \"actual\": [1.308036, 4.08006, 3.4, -0.68006], \"expected\": [1.376593, 6.676516, 6.39409, -0.282426], \"passed\": false}, {\"check\": \"pre/post sample 3\", \"actual\": [1.297436, 10.099573, 14.25, 4.150427], \"expected\": [1.748201, 9.079822, 11.690134, 2.610312], \"passed\": false}], \"passed\": false}\n"},"broken":{"elapsed_ms":39.168,"exit_code":1,"observations":[{"actual":[2.0,12.0,12.0,0.0],"check":"pre-period imbalance is corrected","expected":[2.0,14.0,14.0,0.0],"passed":false},{"actual":[2.0,4.0,4.0,0.0],"check":"perfectly correlated covariate","expected":[2.0,5.0,5.0,0.0],"passed":false},{"actual":[0.0,2.0,3.0,1.0],"check":"constant covariate leaves means unadjusted","expected":[0.0,2.0,3.0,1.0],"passed":true},{"actual":[2.0,4.0,4.5,0.5],"check":"arms of different size","expected":[1.851852,5.851852,6.574074,0.722222],"passed":false},{"actual":[1.0,2.0,3.0,1.0],"check":"covariate balanced across arms","expected":[1.0,2.0,3.0,1.0],"passed":true},{"actual":[1.389831,10.5,9.042373,-1.457627],"check":"pre/post sample 1","expected":[1.319149,10.170213,8.659574,-1.510638],"passed":false},{"actual":[0.0,11.666667,3.4,-8.266667],"check":"pre/post sample 2","expected":[1.376593,6.676516,6.39409,-0.282426],"passed":false},{"actual":[2.375,5.666667,6.135417,0.46875],"check":"pre/post sample 3","expected":[1.748201,9.079822,11.690134,2.610312],"passed":false}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"pre-period imbalance is corrected\", \"actual\": [2.0, 12.0, 12.0, 0.0], \"expected\": [2.0, 14.0, 14.0, 0.0], \"passed\": false}, {\"check\": \"perfectly correlated covariate\", \"actual\": [2.0, 4.0, 4.0, 0.0], \"expected\": [2.0, 5.0, 5.0, 0.0], \"passed\": false}, {\"check\": \"constant covariate leaves means unadjusted\", \"actual\": [0.0, 2.0, 3.0, 1.0], \"expected\": [0.0, 2.0, 3.0, 1.0], \"passed\": true}, {\"check\": \"arms of different size\", \"actual\": [2.0, 4.0, 4.5, 0.5], \"expected\": [1.851852, 5.851852, 6.574074, 0.722222], \"passed\": false}, {\"check\": \"covariate balanced across arms\", \"actual\": [1.0, 2.0, 3.0, 1.0], \"expected\": [1.0, 2.0, 3.0, 1.0], \"passed\": true}, {\"check\": \"pre/post sample 1\", \"actual\": [1.389831, 10.5, 9.042373, -1.457627], \"expected\": [1.319149, 10.170213, 8.659574, -1.510638], \"passed\": false}, {\"check\": \"pre/post sample 2\", \"actual\": [0.0, 11.666667, 3.4, -8.266667], \"expected\": [1.376593, 6.676516, 6.39409, -0.282426], \"passed\": false}, {\"check\": \"pre/post sample 3\", \"actual\": [2.375, 5.666667, 6.135417, 0.46875], \"expected\": [1.748201, 9.079822, 11.690134, 2.610312], \"passed\": false}], \"passed\": false}\n"},"fixed":{"elapsed_ms":41.928,"exit_code":0,"observations":[{"actual":[2.0,14.0,14.0,0.0],"check":"pre-period imbalance is corrected","expected":[2.0,14.0,14.0,0.0],"passed":true},{"actual":[2.0,5.0,5.0,0.0],"check":"perfectly correlated covariate","expected":[2.0,5.0,5.0,0.0],"passed":true},{"actual":[0.0,2.0,3.0,1.0],"check":"constant covariate leaves means unadjusted","expected":[0.0,2.0,3.0,1.0],"passed":true},{"actual":[1.851852,5.851852,6.574074,0.722222],"check":"arms of different size","expected":[1.851852,5.851852,6.574074,0.722222],"passed":true},{"actual":[1.0,2.0,3.0,1.0],"check":"covariate balanced across arms","expected":[1.0,2.0,3.0,1.0],"passed":true},{"actual":[1.319149,10.170213,8.659574,-1.510638],"check":"pre/post sample 1","expected":[1.319149,10.170213,8.659574,-1.510638],"passed":true},{"actual":[1.376593,6.676516,6.39409,-0.282426],"check":"pre/post sample 2","expected":[1.376593,6.676516,6.39409,-0.282426],"passed":true},{"actual":[1.748201,9.079822,11.690134,2.610312],"check":"pre/post sample 3","expected":[1.748201,9.079822,11.690134,2.610312],"passed":true}],"passed":true,"stderr":"","stdout":"{\"observations\": [{\"check\": \"pre-period imbalance is corrected\", \"actual\": [2.0, 14.0, 14.0, 0.0], \"expected\": [2.0, 14.0, 14.0, 0.0], \"passed\": true}, {\"check\": \"perfectly correlated covariate\", \"actual\": [2.0, 5.0, 5.0, 0.0], \"expected\": [2.0, 5.0, 5.0, 0.0], \"passed\": true}, {\"check\": \"constant covariate leaves means unadjusted\", \"actual\": [0.0, 2.0, 3.0, 1.0], \"expected\": [0.0, 2.0, 3.0, 1.0], \"passed\": true}, {\"check\": \"arms of different size\", \"actual\": [1.851852, 5.851852, 6.574074, 0.722222], \"expected\": [1.851852, 5.851852, 6.574074, 0.722222], \"passed\": true}, {\"check\": \"covariate balanced across arms\", \"actual\": [1.0, 2.0, 3.0, 1.0], \"expected\": [1.0, 2.0, 3.0, 1.0], \"passed\": true}, {\"check\": \"pre/post sample 1\", \"actual\": [1.319149, 10.170213, 8.659574, -1.510638], \"expected\": [1.319149, 10.170213, 8.659574, -1.510638], \"passed\": true}, {\"check\": \"pre/post sample 2\", \"actual\": [1.376593, 6.676516, 6.39409, -0.282426], \"expected\": [1.376593, 6.676516, 6.39409, -0.282426], \"passed\": true}, {\"check\": \"pre/post sample 3\", \"actual\": [1.748201, 9.079822, 11.690134, 2.610312], \"expected\": [1.748201, 9.079822, 11.690134, 2.610312], \"passed\": true}], \"passed\": true}\n"}},"verified":true,"visibility":"public"}