FA-74691 / Experiment statistics / Open access

Complier average causal effect: Treatment outcomes are averaged over adopters · case 01

The ITT estimate mixes a per-protocol treatment mean with an intent-to-treat control mean.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The treatment mean divides by took_t instead of assigned_t.

VERIFIED REPAIR

Divide each arm's outcome sum by its assigned count.

Unsuccessful approach: Dividing the combined difference by all assigned users only works for equal arms.

Case contract

Outcome sums cover every assigned user. ITT = outcome_sum_t / assigned_t - outcome_sum_c / assigned_c. Compliance difference = took_t / assigned_t - took_c / assigned_c (control users can cross over). CACE = ITT / compliance difference when that difference is positive, else None. Nonpositive assignment counts -> None. Return [ITT, compliance difference, CACE] rounded to 6.

Why this case matters

Opt-in features have partial uptake; the effect on adopters needs an instrumental-variable estimate.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(assigned_c, took_c, outcome_sum_c, assigned_t, took_t, outcome_sum_t):
    if assigned_c <= 0 or assigned_t <= 0:
        return None
    itt = outcome_sum_t / max(took_t, 1) - outcome_sum_c / assigned_c
    comp = took_t / assigned_t - took_c / assigned_c
    cace = round(itt / comp, 6) if comp > 0 else None
    return [round(itt, 6), round(comp, 6), cace]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('crossover in control reduces compliance difference', [200, 20, 300, 200, 120, 420], [0.6, 0.5, 1.2]),
  ('unequal arm sizes', [100, 0, 150, 400, 200, 800], [0.5, 0.5, 1.0]),
  ('no uptake difference has no CACE', [100, 50, 100, 100, 50, 120], [0.2, 0.0, None]),
  ('low but positive compliance still yields CACE', [100, 0, 100, 100, 30, 130], [0.3, 0.3, 1.0]),
  ('negative compliance difference', [100, 60, 100, 100, 40, 90], [-0.1, -0.2, None]),
  ('uptake sample 1', [200, 22, 428, 400, 23, 389], [-1.1675, -0.0525, None]),
  ('uptake sample 2', [100, 22, 279, 400, 165, 505], [-1.5275, 0.1925, -7.935065]),
  ('uptake sample 3', [100, 11, 241, 400, 240, 17], [-2.3675, 0.49, -4.831633])],
 [('crossover in control reduces compliance difference', [200, 20, 300, 200, 120, 420], [0.6, 0.5, 1.2]),
  ('unequal arm sizes', [100, 0, 150, 400, 200, 800], [0.5, 0.5, 1.0]),
  ('no uptake difference has no CACE', [100, 50, 100, 100, 50, 120], [0.2, 0.0, None]),
  ('low but positive compliance still yields CACE', [100, 0, 100, 100, 30, 130], [0.3, 0.3, 1.0]),
  ('negative compliance difference', [100, 60, 100, 100, 40, 90], [-0.1, -0.2, None]),
  ('uptake sample 3', [100, 11, 241, 400, 240, 17], [-2.3675, 0.49, -4.831633]),
  ('uptake sample 6', [400, 10, 585, 300, 223, 613], [0.580833, 0.718333, 0.808585]),
  ('uptake sample 11', [400, 10, 635, 100, 93, 173], [0.1425, 0.905, 0.157459])],
 [('crossover in control reduces compliance difference', [200, 20, 300, 200, 120, 420], [0.6, 0.5, 1.2]),
  ('unequal arm sizes', [100, 0, 150, 400, 200, 800], [0.5, 0.5, 1.0]),
  ('no uptake difference has no CACE', [100, 50, 100, 100, 50, 120], [0.2, 0.0, None]),
  ('low but positive compliance still yields CACE', [100, 0, 100, 100, 30, 130], [0.3, 0.3, 1.0]),
  ('negative compliance difference', [100, 60, 100, 100, 40, 90], [-0.1, -0.2, None]),
  ('uptake sample 10', [100, 28, 259, 100, 20, 38], [-2.21, -0.08, None]),
  ('uptake sample 11', [400, 10, 635, 100, 93, 173], [0.1425, 0.905, 0.157459]),
  ('uptake sample 18', [200, 43, 285, 300, 21, 666], [0.795, -0.145, None])],
 [('crossover in control reduces compliance difference', [200, 20, 300, 200, 120, 420], [0.6, 0.5, 1.2]),
  ('unequal arm sizes', [100, 0, 150, 400, 200, 800], [0.5, 0.5, 1.0]),
  ('no uptake difference has no CACE', [100, 50, 100, 100, 50, 120], [0.2, 0.0, None]),
  ('low but positive compliance still yields CACE', [100, 0, 100, 100, 30, 130], [0.3, 0.3, 1.0]),
  ('negative compliance difference', [100, 60, 100, 100, 40, 90], [-0.1, -0.2, None]),
  ('uptake sample 16', [100, 6, 298, 300, 104, 511], [-1.276667, 0.286667, -4.453488]),
  ('uptake sample 17', [200, 26, 414, 400, 300, 725], [-0.2575, 0.62, -0.415323]),
  ('uptake sample 28', [200, 65, 83, 400, 91, 261], [0.2375, -0.0975, None])],
 [('crossover in control reduces compliance difference', [200, 20, 300, 200, 120, 420], [0.6, 0.5, 1.2]),
  ('unequal arm sizes', [100, 0, 150, 400, 200, 800], [0.5, 0.5, 1.0]),
  ('no uptake difference has no CACE', [100, 50, 100, 100, 50, 120], [0.2, 0.0, None]),
  ('low but positive compliance still yields CACE', [100, 0, 100, 100, 30, 130], [0.3, 0.3, 1.0]),
  ('negative compliance difference', [100, 60, 100, 100, 40, 90], [-0.1, -0.2, None]),
  ('uptake sample 21', [100, 18, 78, 100, 98, 114], [0.36, 0.8, 0.45]),
  ('uptake sample 24', [100, 19, 81, 100, 65, 80], [-0.01, 0.46, -0.021739]),
  ('uptake sample 38', [400, 93, 478, 100, 63, 16], [-1.035, 0.3975, -2.603774])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
crossover in control reduces compliance difference	`[2.0, 0.5, 4.0]`	`[0.6, 0.5, 1.2]`	Failed
unequal arm sizes	`[2.5, 0.5, 5.0]`	`[0.5, 0.5, 1.0]`	Failed
no uptake difference has no CACE	`[1.4, 0.0, None]`	`[0.2, 0.0, None]`	Failed
low but positive compliance still yields CACE	`[3.333333, 0.3, 11.111111]`	`[0.3, 0.3, 1.0]`	Failed
negative compliance difference	`[1.25, -0.2, None]`	`[-0.1, -0.2, None]`	Failed
uptake sample 1	`[14.773043, -0.0525, None]`	`[-1.1675, -0.0525, None]`	Failed
uptake sample 2	`[0.270606, 0.1925, 1.405746]`	`[-1.5275, 0.1925, -7.935065]`	Failed
uptake sample 3	`[-2.339167, 0.49, -4.77381]`	`[-2.3675, 0.49, -4.831633]`	Failed

SHA-256 / bbdb17541d43b5eb66fe64d45462ed239a6a5da5171911ea70c69c50d62c1c32

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(assigned_c, took_c, outcome_sum_c, assigned_t, took_t, outcome_sum_t):
    if assigned_c <= 0 or assigned_t <= 0:
        return None
    itt = (outcome_sum_t - outcome_sum_c) * 2 / (assigned_t + assigned_c)
    comp = took_t / assigned_t - took_c / assigned_c
    cace = round(itt / comp, 6) if comp > 0 else None
    return [round(itt, 6), round(comp, 6), cace]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('crossover in control reduces compliance difference', [200, 20, 300, 200, 120, 420], [0.6, 0.5, 1.2]),
  ('unequal arm sizes', [100, 0, 150, 400, 200, 800], [0.5, 0.5, 1.0]),
  ('no uptake difference has no CACE', [100, 50, 100, 100, 50, 120], [0.2, 0.0, None]),
  ('low but positive compliance still yields CACE', [100, 0, 100, 100, 30, 130], [0.3, 0.3, 1.0]),
  ('negative compliance difference', [100, 60, 100, 100, 40, 90], [-0.1, -0.2, None]),
  ('uptake sample 1', [200, 22, 428, 400, 23, 389], [-1.1675, -0.0525, None]),
  ('uptake sample 2', [100, 22, 279, 400, 165, 505], [-1.5275, 0.1925, -7.935065]),
  ('uptake sample 3', [100, 11, 241, 400, 240, 17], [-2.3675, 0.49, -4.831633])],
 [('crossover in control reduces compliance difference', [200, 20, 300, 200, 120, 420], [0.6, 0.5, 1.2]),
  ('unequal arm sizes', [100, 0, 150, 400, 200, 800], [0.5, 0.5, 1.0]),
  ('no uptake difference has no CACE', [100, 50, 100, 100, 50, 120], [0.2, 0.0, None]),
  ('low but positive compliance still yields CACE', [100, 0, 100, 100, 30, 130], [0.3, 0.3, 1.0]),
  ('negative compliance difference', [100, 60, 100, 100, 40, 90], [-0.1, -0.2, None]),
  ('uptake sample 3', [100, 11, 241, 400, 240, 17], [-2.3675, 0.49, -4.831633]),
  ('uptake sample 6', [400, 10, 585, 300, 223, 613], [0.580833, 0.718333, 0.808585]),
  ('uptake sample 11', [400, 10, 635, 100, 93, 173], [0.1425, 0.905, 0.157459])],
 [('crossover in control reduces compliance difference', [200, 20, 300, 200, 120, 420], [0.6, 0.5, 1.2]),
  ('unequal arm sizes', [100, 0, 150, 400, 200, 800], [0.5, 0.5, 1.0]),
  ('no uptake difference has no CACE', [100, 50, 100, 100, 50, 120], [0.2, 0.0, None]),
  ('low but positive compliance still yields CACE', [100, 0, 100, 100, 30, 130], [0.3, 0.3, 1.0]),
  ('negative compliance difference', [100, 60, 100, 100, 40, 90], [-0.1, -0.2, None]),
  ('uptake sample 10', [100, 28, 259, 100, 20, 38], [-2.21, -0.08, None]),
  ('uptake sample 11', [400, 10, 635, 100, 93, 173], [0.1425, 0.905, 0.157459]),
  ('uptake sample 18', [200, 43, 285, 300, 21, 666], [0.795, -0.145, None])],
 [('crossover in control reduces compliance difference', [200, 20, 300, 200, 120, 420], [0.6, 0.5, 1.2]),
  ('unequal arm sizes', [100, 0, 150, 400, 200, 800], [0.5, 0.5, 1.0]),
  ('no uptake difference has no CACE', [100, 50, 100, 100, 50, 120], [0.2, 0.0, None]),
  ('low but positive compliance still yields CACE', [100, 0, 100, 100, 30, 130], [0.3, 0.3, 1.0]),
  ('negative compliance difference', [100, 60, 100, 100, 40, 90], [-0.1, -0.2, None]),
  ('uptake sample 16', [100, 6, 298, 300, 104, 511], [-1.276667, 0.286667, -4.453488]),
  ('uptake sample 17', [200, 26, 414, 400, 300, 725], [-0.2575, 0.62, -0.415323]),
  ('uptake sample 28', [200, 65, 83, 400, 91, 261], [0.2375, -0.0975, None])],
 [('crossover in control reduces compliance difference', [200, 20, 300, 200, 120, 420], [0.6, 0.5, 1.2]),
  ('unequal arm sizes', [100, 0, 150, 400, 200, 800], [0.5, 0.5, 1.0]),
  ('no uptake difference has no CACE', [100, 50, 100, 100, 50, 120], [0.2, 0.0, None]),
  ('low but positive compliance still yields CACE', [100, 0, 100, 100, 30, 130], [0.3, 0.3, 1.0]),
  ('negative compliance difference', [100, 60, 100, 100, 40, 90], [-0.1, -0.2, None]),
  ('uptake sample 21', [100, 18, 78, 100, 98, 114], [0.36, 0.8, 0.45]),
  ('uptake sample 24', [100, 19, 81, 100, 65, 80], [-0.01, 0.46, -0.021739]),
  ('uptake sample 38', [400, 93, 478, 100, 63, 16], [-1.035, 0.3975, -2.603774])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
crossover in control reduces compliance difference	`[0.6, 0.5, 1.2]`	`[0.6, 0.5, 1.2]`	Passed
unequal arm sizes	`[2.6, 0.5, 5.2]`	`[0.5, 0.5, 1.0]`	Failed
no uptake difference has no CACE	`[0.2, 0.0, None]`	`[0.2, 0.0, None]`	Passed
low but positive compliance still yields CACE	`[0.3, 0.3, 1.0]`	`[0.3, 0.3, 1.0]`	Passed
negative compliance difference	`[-0.1, -0.2, None]`	`[-0.1, -0.2, None]`	Passed
uptake sample 1	`[-0.13, -0.0525, None]`	`[-1.1675, -0.0525, None]`	Failed
uptake sample 2	`[0.904, 0.1925, 4.696104]`	`[-1.5275, 0.1925, -7.935065]`	Failed
uptake sample 3	`[-0.896, 0.49, -1.828571]`	`[-2.3675, 0.49, -4.831633]`	Failed

SHA-256 / 54a448cb794c99da43704c9a6e6b91f447fe6b9a777550c01c9d039281d1cb87

3 / The verified repair

Exit 0

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(assigned_c, took_c, outcome_sum_c, assigned_t, took_t, outcome_sum_t):
    if assigned_c <= 0 or assigned_t <= 0:
        return None
    itt = outcome_sum_t / assigned_t - outcome_sum_c / assigned_c
    comp = took_t / assigned_t - took_c / assigned_c
    cace = round(itt / comp, 6) if comp > 0 else None
    return [round(itt, 6), round(comp, 6), cace]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('crossover in control reduces compliance difference', [200, 20, 300, 200, 120, 420], [0.6, 0.5, 1.2]),
  ('unequal arm sizes', [100, 0, 150, 400, 200, 800], [0.5, 0.5, 1.0]),
  ('no uptake difference has no CACE', [100, 50, 100, 100, 50, 120], [0.2, 0.0, None]),
  ('low but positive compliance still yields CACE', [100, 0, 100, 100, 30, 130], [0.3, 0.3, 1.0]),
  ('negative compliance difference', [100, 60, 100, 100, 40, 90], [-0.1, -0.2, None]),
  ('uptake sample 1', [200, 22, 428, 400, 23, 389], [-1.1675, -0.0525, None]),
  ('uptake sample 2', [100, 22, 279, 400, 165, 505], [-1.5275, 0.1925, -7.935065]),
  ('uptake sample 3', [100, 11, 241, 400, 240, 17], [-2.3675, 0.49, -4.831633])],
 [('crossover in control reduces compliance difference', [200, 20, 300, 200, 120, 420], [0.6, 0.5, 1.2]),
  ('unequal arm sizes', [100, 0, 150, 400, 200, 800], [0.5, 0.5, 1.0]),
  ('no uptake difference has no CACE', [100, 50, 100, 100, 50, 120], [0.2, 0.0, None]),
  ('low but positive compliance still yields CACE', [100, 0, 100, 100, 30, 130], [0.3, 0.3, 1.0]),
  ('negative compliance difference', [100, 60, 100, 100, 40, 90], [-0.1, -0.2, None]),
  ('uptake sample 3', [100, 11, 241, 400, 240, 17], [-2.3675, 0.49, -4.831633]),
  ('uptake sample 6', [400, 10, 585, 300, 223, 613], [0.580833, 0.718333, 0.808585]),
  ('uptake sample 11', [400, 10, 635, 100, 93, 173], [0.1425, 0.905, 0.157459])],
 [('crossover in control reduces compliance difference', [200, 20, 300, 200, 120, 420], [0.6, 0.5, 1.2]),
  ('unequal arm sizes', [100, 0, 150, 400, 200, 800], [0.5, 0.5, 1.0]),
  ('no uptake difference has no CACE', [100, 50, 100, 100, 50, 120], [0.2, 0.0, None]),
  ('low but positive compliance still yields CACE', [100, 0, 100, 100, 30, 130], [0.3, 0.3, 1.0]),
  ('negative compliance difference', [100, 60, 100, 100, 40, 90], [-0.1, -0.2, None]),
  ('uptake sample 10', [100, 28, 259, 100, 20, 38], [-2.21, -0.08, None]),
  ('uptake sample 11', [400, 10, 635, 100, 93, 173], [0.1425, 0.905, 0.157459]),
  ('uptake sample 18', [200, 43, 285, 300, 21, 666], [0.795, -0.145, None])],
 [('crossover in control reduces compliance difference', [200, 20, 300, 200, 120, 420], [0.6, 0.5, 1.2]),
  ('unequal arm sizes', [100, 0, 150, 400, 200, 800], [0.5, 0.5, 1.0]),
  ('no uptake difference has no CACE', [100, 50, 100, 100, 50, 120], [0.2, 0.0, None]),
  ('low but positive compliance still yields CACE', [100, 0, 100, 100, 30, 130], [0.3, 0.3, 1.0]),
  ('negative compliance difference', [100, 60, 100, 100, 40, 90], [-0.1, -0.2, None]),
  ('uptake sample 16', [100, 6, 298, 300, 104, 511], [-1.276667, 0.286667, -4.453488]),
  ('uptake sample 17', [200, 26, 414, 400, 300, 725], [-0.2575, 0.62, -0.415323]),
  ('uptake sample 28', [200, 65, 83, 400, 91, 261], [0.2375, -0.0975, None])],
 [('crossover in control reduces compliance difference', [200, 20, 300, 200, 120, 420], [0.6, 0.5, 1.2]),
  ('unequal arm sizes', [100, 0, 150, 400, 200, 800], [0.5, 0.5, 1.0]),
  ('no uptake difference has no CACE', [100, 50, 100, 100, 50, 120], [0.2, 0.0, None]),
  ('low but positive compliance still yields CACE', [100, 0, 100, 100, 30, 130], [0.3, 0.3, 1.0]),
  ('negative compliance difference', [100, 60, 100, 100, 40, 90], [-0.1, -0.2, None]),
  ('uptake sample 21', [100, 18, 78, 100, 98, 114], [0.36, 0.8, 0.45]),
  ('uptake sample 24', [100, 19, 81, 100, 65, 80], [-0.01, 0.46, -0.021739]),
  ('uptake sample 38', [400, 93, 478, 100, 63, 16], [-1.035, 0.3975, -2.603774])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
crossover in control reduces compliance difference	`[0.6, 0.5, 1.2]`	`[0.6, 0.5, 1.2]`	Passed
unequal arm sizes	`[0.5, 0.5, 1.0]`	`[0.5, 0.5, 1.0]`	Passed
no uptake difference has no CACE	`[0.2, 0.0, None]`	`[0.2, 0.0, None]`	Passed
low but positive compliance still yields CACE	`[0.3, 0.3, 1.0]`	`[0.3, 0.3, 1.0]`	Passed
negative compliance difference	`[-0.1, -0.2, None]`	`[-0.1, -0.2, None]`	Passed
uptake sample 1	`[-1.1675, -0.0525, None]`	`[-1.1675, -0.0525, None]`	Passed
uptake sample 2	`[-1.5275, 0.1925, -7.935065]`	`[-1.5275, 0.1925, -7.935065]`	Passed
uptake sample 3	`[-2.3675, 0.49, -4.831633]`	`[-2.3675, 0.49, -4.831633]`	Passed

SHA-256 / 187442bb5def74b8ab2033a09ef77236c52399704355e79781f224a94db96cfd

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:59.114158+00:00.

Case digest / d7cf678b2160b76fb7943b6dd2d6e9c7319aee84e5af5c7bf90f925faa0e490a