FA-74606 / Experiment statistics / Open access
Triggered effect dilution: The trigger rate comes from the treatment arm · case 01
A treatment that changes who triggers distorts the diluted estimate.
ROOT CAUSE
rate = trig_t / n_t.
VERIFIED REPAIR
Pool triggered and total users across both arms.
Unsuccessful approach: Averaging the two arm rates weights a small arm as heavily as a large one.
Case contract
delta_trig is the absolute effect among triggered users. The trigger rate is pooled across arms: (trig_c + trig_t) / (n_c + n_t). The diluted site-wide effect is delta_trig * rate and the relative effect divides that by the overall control mean. No users or zero overall mean -> None. Return [rate, diluted absolute, diluted relative] rounded to 6.
Why this case matters
Triggered analyses are sensitive but must be diluted before claiming site-wide impact.
1 / The failure
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(delta_trig, trig_c, n_c, trig_t, n_t, overall_mean_c):
if n_c + n_t == 0 or overall_mean_c == 0:
return None
rate = trig_t / n_t
absolute = delta_trig * rate
return [round(rate, 6), round(absolute, 6), round(absolute / overall_mean_c, 6)]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
('trigger sample 1', [0.5, 4, 400, 61, 100, 4.0], [0.13, 0.065, 0.01625]),
('trigger sample 2', [0.1, 25, 100, 28, 300, 1.0], [0.1325, 0.01325, 0.01325]),
('trigger sample 3', [0.5, 640, 1000, 682, 1000, 10.0], [0.661, 0.3305, 0.03305])],
[('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
('trigger sample 6', [0.5, 791, 1000, 75, 100, 4.0], [0.787273, 0.393636, 0.098409]),
('trigger sample 7', [2.0, 92, 100, 4, 100, 1.0], [0.48, 0.96, 0.96]),
('trigger sample 9', [2.0, 873, 1000, 77, 100, 1.0], [0.863636, 1.727273, 1.727273])],
[('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
('trigger sample 11', [0.1, 40, 100, 14, 100, 4.0], [0.27, 0.027, 0.00675]),
('trigger sample 13', [2.0, 183, 400, 86, 300, 10.0], [0.384286, 0.768571, 0.076857]),
('trigger sample 19', [-1.0, 294, 1000, 72, 100, 4.0], [0.332727, -0.332727, -0.083182])],
[('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
('trigger sample 16', [2.0, 50, 100, 332, 1000, 10.0], [0.347273, 0.694545, 0.069455]),
('trigger sample 20', [0.1, 357, 400, 89, 300, 4.0], [0.637143, 0.063714, 0.015929]),
('trigger sample 27', [2.0, 77, 100, 143, 1000, 4.0], [0.2, 0.4, 0.1])],
[('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
('trigger sample 21', [-1.0, 26, 100, 60, 100, 4.0], [0.43, -0.43, -0.1075]),
('trigger sample 27', [2.0, 77, 100, 143, 1000, 4.0], [0.2, 0.4, 0.1]),
('trigger sample 36', [-1.0, 456, 1000, 75, 100, 4.0], [0.482727, -0.482727, -0.120682])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| unequal arms pool the trigger rate | [0.3, 0.6, 0.06] | [0.118182, 0.236364, 0.023636] | Failed |
| relative effect uses overall mean | [0.5, 0.25, 0.05] | [0.5, 0.25, 0.05] | Passed |
| negative effect dilutes too | [0.1, -0.1, -0.025] | [0.057143, -0.057143, -0.014286] | Failed |
| nobody triggered | [0.0, 0.0, 0.0] | [0.0, 0.0, 0.0] | Passed |
| zero overall mean | None | None | Passed |
| trigger sample 1 | [0.61, 0.305, 0.07625] | [0.13, 0.065, 0.01625] | Failed |
| trigger sample 2 | [0.093333, 0.009333, 0.009333] | [0.1325, 0.01325, 0.01325] | Failed |
| trigger sample 3 | [0.682, 0.341, 0.0341] | [0.661, 0.3305, 0.03305] | Failed |
SHA-256 / f4cda56d9a534aa6cdc6d07397c36d557a891c43aa2871e147a631a863f115df
2 / The unsuccessful fix
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(delta_trig, trig_c, n_c, trig_t, n_t, overall_mean_c):
if n_c + n_t == 0 or overall_mean_c == 0:
return None
rate = (trig_c / n_c + trig_t / n_t) / 2
absolute = delta_trig * rate
return [round(rate, 6), round(absolute, 6), round(absolute / overall_mean_c, 6)]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
('trigger sample 1', [0.5, 4, 400, 61, 100, 4.0], [0.13, 0.065, 0.01625]),
('trigger sample 2', [0.1, 25, 100, 28, 300, 1.0], [0.1325, 0.01325, 0.01325]),
('trigger sample 3', [0.5, 640, 1000, 682, 1000, 10.0], [0.661, 0.3305, 0.03305])],
[('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
('trigger sample 6', [0.5, 791, 1000, 75, 100, 4.0], [0.787273, 0.393636, 0.098409]),
('trigger sample 7', [2.0, 92, 100, 4, 100, 1.0], [0.48, 0.96, 0.96]),
('trigger sample 9', [2.0, 873, 1000, 77, 100, 1.0], [0.863636, 1.727273, 1.727273])],
[('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
('trigger sample 11', [0.1, 40, 100, 14, 100, 4.0], [0.27, 0.027, 0.00675]),
('trigger sample 13', [2.0, 183, 400, 86, 300, 10.0], [0.384286, 0.768571, 0.076857]),
('trigger sample 19', [-1.0, 294, 1000, 72, 100, 4.0], [0.332727, -0.332727, -0.083182])],
[('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
('trigger sample 16', [2.0, 50, 100, 332, 1000, 10.0], [0.347273, 0.694545, 0.069455]),
('trigger sample 20', [0.1, 357, 400, 89, 300, 4.0], [0.637143, 0.063714, 0.015929]),
('trigger sample 27', [2.0, 77, 100, 143, 1000, 4.0], [0.2, 0.4, 0.1])],
[('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
('trigger sample 21', [-1.0, 26, 100, 60, 100, 4.0], [0.43, -0.43, -0.1075]),
('trigger sample 27', [2.0, 77, 100, 143, 1000, 4.0], [0.2, 0.4, 0.1]),
('trigger sample 36', [-1.0, 456, 1000, 75, 100, 4.0], [0.482727, -0.482727, -0.120682])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| unequal arms pool the trigger rate | [0.2, 0.4, 0.04] | [0.118182, 0.236364, 0.023636] | Failed |
| relative effect uses overall mean | [0.5, 0.25, 0.05] | [0.5, 0.25, 0.05] | Passed |
| negative effect dilutes too | [0.0625, -0.0625, -0.015625] | [0.057143, -0.057143, -0.014286] | Failed |
| nobody triggered | [0.0, 0.0, 0.0] | [0.0, 0.0, 0.0] | Passed |
| zero overall mean | None | None | Passed |
| trigger sample 1 | [0.31, 0.155, 0.03875] | [0.13, 0.065, 0.01625] | Failed |
| trigger sample 2 | [0.171667, 0.017167, 0.017167] | [0.1325, 0.01325, 0.01325] | Failed |
| trigger sample 3 | [0.661, 0.3305, 0.03305] | [0.661, 0.3305, 0.03305] | Passed |
SHA-256 / 23ee91362027e2f0675519c1fa02ea89d239357656b6ada91805ef6a977a9260
3 / The verified repair
Exit 0"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(delta_trig, trig_c, n_c, trig_t, n_t, overall_mean_c):
if n_c + n_t == 0 or overall_mean_c == 0:
return None
rate = (trig_c + trig_t) / (n_c + n_t)
absolute = delta_trig * rate
return [round(rate, 6), round(absolute, 6), round(absolute / overall_mean_c, 6)]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
('trigger sample 1', [0.5, 4, 400, 61, 100, 4.0], [0.13, 0.065, 0.01625]),
('trigger sample 2', [0.1, 25, 100, 28, 300, 1.0], [0.1325, 0.01325, 0.01325]),
('trigger sample 3', [0.5, 640, 1000, 682, 1000, 10.0], [0.661, 0.3305, 0.03305])],
[('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
('trigger sample 6', [0.5, 791, 1000, 75, 100, 4.0], [0.787273, 0.393636, 0.098409]),
('trigger sample 7', [2.0, 92, 100, 4, 100, 1.0], [0.48, 0.96, 0.96]),
('trigger sample 9', [2.0, 873, 1000, 77, 100, 1.0], [0.863636, 1.727273, 1.727273])],
[('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
('trigger sample 11', [0.1, 40, 100, 14, 100, 4.0], [0.27, 0.027, 0.00675]),
('trigger sample 13', [2.0, 183, 400, 86, 300, 10.0], [0.384286, 0.768571, 0.076857]),
('trigger sample 19', [-1.0, 294, 1000, 72, 100, 4.0], [0.332727, -0.332727, -0.083182])],
[('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
('trigger sample 16', [2.0, 50, 100, 332, 1000, 10.0], [0.347273, 0.694545, 0.069455]),
('trigger sample 20', [0.1, 357, 400, 89, 300, 4.0], [0.637143, 0.063714, 0.015929]),
('trigger sample 27', [2.0, 77, 100, 143, 1000, 4.0], [0.2, 0.4, 0.1])],
[('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
('trigger sample 21', [-1.0, 26, 100, 60, 100, 4.0], [0.43, -0.43, -0.1075]),
('trigger sample 27', [2.0, 77, 100, 143, 1000, 4.0], [0.2, 0.4, 0.1]),
('trigger sample 36', [-1.0, 456, 1000, 75, 100, 4.0], [0.482727, -0.482727, -0.120682])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| unequal arms pool the trigger rate | [0.118182, 0.236364, 0.023636] | [0.118182, 0.236364, 0.023636] | Passed |
| relative effect uses overall mean | [0.5, 0.25, 0.05] | [0.5, 0.25, 0.05] | Passed |
| negative effect dilutes too | [0.057143, -0.057143, -0.014286] | [0.057143, -0.057143, -0.014286] | Passed |
| nobody triggered | [0.0, 0.0, 0.0] | [0.0, 0.0, 0.0] | Passed |
| zero overall mean | None | None | Passed |
| trigger sample 1 | [0.13, 0.065, 0.01625] | [0.13, 0.065, 0.01625] | Passed |
| trigger sample 2 | [0.1325, 0.01325, 0.01325] | [0.1325, 0.01325, 0.01325] | Passed |
| trigger sample 3 | [0.661, 0.3305, 0.03305] | [0.661, 0.3305, 0.03305] | Passed |
SHA-256 / dc17d7a03d701660529ff71c09c3e0d12a9e75cbea8173f724c03e468c6a573f
Verification & scope
A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.
Observations recorded using Python 3.12.14 at 2026-09-29T14:48:58.409194+00:00.
Case digest / 20e785fc31c1cb15d12164258c166bdfacdc1b3cbee07705064f7ee089fa5f87