FAILURE MAP
← Case archive

FA-74611 / Experiment statistics / Open access

Triggered effect dilution: Relative impact uses the undiluted effect · case 01

Site-wide relative lift is overstated by the inverse trigger rate.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The relative effect divides delta_trig rather than the diluted effect by the overall mean.

THE FAILURE

The relative effect divides delta_trig rather than the diluted effect by the overall mean.

Unsuccessful approach: Dividing by the treated overall mean (control plus effect) changes the baseline.

Case contract

delta_trig is the absolute effect among triggered users. The trigger rate is pooled across arms: (trig_c + trig_t) / (n_c + n_t). The diluted site-wide effect is delta_trig * rate and the relative effect divides that by the overall control mean. No users or zero overall mean -> None. Return [rate, diluted absolute, diluted relative] rounded to 6.

Why this case matters

Triggered analyses are sensitive but must be diluted before claiming site-wide impact.

1 / The failure

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(delta_trig, trig_c, n_c, trig_t, n_t, overall_mean_c):
    if n_c + n_t == 0 or overall_mean_c == 0:
        return None
    rate = (trig_c + trig_t) / (n_c + n_t)
    absolute = delta_trig * rate
    return [round(rate, 6), round(absolute, 6), round(delta_trig / overall_mean_c, 6)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 1', [0.5, 4, 400, 61, 100, 4.0], [0.13, 0.065, 0.01625]),
  ('trigger sample 2', [0.1, 25, 100, 28, 300, 1.0], [0.1325, 0.01325, 0.01325]),
  ('trigger sample 3', [0.5, 640, 1000, 682, 1000, 10.0], [0.661, 0.3305, 0.03305])],
 [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 4', [-1.0, 1, 1000, 191, 300, 1.0], [0.147692, -0.147692, -0.147692]),
  ('trigger sample 5', [0.5, 661, 1000, 236, 1000, 1.0], [0.4485, 0.22425, 0.22425]),
  ('trigger sample 6', [0.5, 791, 1000, 75, 100, 4.0], [0.787273, 0.393636, 0.098409])],
 [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 11', [0.1, 40, 100, 14, 100, 4.0], [0.27, 0.027, 0.00675]),
  ('trigger sample 12', [2.0, 213, 400, 29, 100, 4.0], [0.484, 0.968, 0.242]),
  ('trigger sample 13', [2.0, 183, 400, 86, 300, 10.0], [0.384286, 0.768571, 0.076857])],
 [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 16', [2.0, 50, 100, 332, 1000, 10.0], [0.347273, 0.694545, 0.069455]),
  ('trigger sample 18', [0.1, 283, 400, 359, 1000, 4.0], [0.458571, 0.045857, 0.011464]),
  ('trigger sample 19', [-1.0, 294, 1000, 72, 100, 4.0], [0.332727, -0.332727, -0.083182])],
 [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 21', [-1.0, 26, 100, 60, 100, 4.0], [0.43, -0.43, -0.1075]),
  ('trigger sample 25', [0.5, 58, 1000, 170, 300, 4.0], [0.175385, 0.087692, 0.021923]),
  ('trigger sample 26', [-1.0, 542, 1000, 68, 300, 1.0], [0.469231, -0.469231, -0.469231])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
unequal arms pool the trigger rate[0.118182, 0.236364, 0.2][0.118182, 0.236364, 0.023636]Failed
relative effect uses overall mean[0.5, 0.25, 0.1][0.5, 0.25, 0.05]Failed
negative effect dilutes too[0.057143, -0.057143, -0.25][0.057143, -0.057143, -0.014286]Failed
nobody triggered[0.0, 0.0, 1.5][0.0, 0.0, 0.0]Failed
zero overall meanNoneNonePassed
trigger sample 1[0.13, 0.065, 0.125][0.13, 0.065, 0.01625]Failed
trigger sample 2[0.1325, 0.01325, 0.1][0.1325, 0.01325, 0.01325]Failed
trigger sample 3[0.661, 0.3305, 0.05][0.661, 0.3305, 0.03305]Failed

SHA-256 / f8956ecad7b87684520f1807c5f6afac96e855ff29dca1889fd8ca56b8074108

2 / The unsuccessful fix

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(delta_trig, trig_c, n_c, trig_t, n_t, overall_mean_c):
    if n_c + n_t == 0 or overall_mean_c == 0:
        return None
    rate = (trig_c + trig_t) / (n_c + n_t)
    absolute = delta_trig * rate
    return [round(rate, 6), round(absolute, 6), round(absolute / (overall_mean_c + absolute), 6)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 1', [0.5, 4, 400, 61, 100, 4.0], [0.13, 0.065, 0.01625]),
  ('trigger sample 2', [0.1, 25, 100, 28, 300, 1.0], [0.1325, 0.01325, 0.01325]),
  ('trigger sample 3', [0.5, 640, 1000, 682, 1000, 10.0], [0.661, 0.3305, 0.03305])],
 [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 4', [-1.0, 1, 1000, 191, 300, 1.0], [0.147692, -0.147692, -0.147692]),
  ('trigger sample 5', [0.5, 661, 1000, 236, 1000, 1.0], [0.4485, 0.22425, 0.22425]),
  ('trigger sample 6', [0.5, 791, 1000, 75, 100, 4.0], [0.787273, 0.393636, 0.098409])],
 [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 11', [0.1, 40, 100, 14, 100, 4.0], [0.27, 0.027, 0.00675]),
  ('trigger sample 12', [2.0, 213, 400, 29, 100, 4.0], [0.484, 0.968, 0.242]),
  ('trigger sample 13', [2.0, 183, 400, 86, 300, 10.0], [0.384286, 0.768571, 0.076857])],
 [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 16', [2.0, 50, 100, 332, 1000, 10.0], [0.347273, 0.694545, 0.069455]),
  ('trigger sample 18', [0.1, 283, 400, 359, 1000, 4.0], [0.458571, 0.045857, 0.011464]),
  ('trigger sample 19', [-1.0, 294, 1000, 72, 100, 4.0], [0.332727, -0.332727, -0.083182])],
 [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 21', [-1.0, 26, 100, 60, 100, 4.0], [0.43, -0.43, -0.1075]),
  ('trigger sample 25', [0.5, 58, 1000, 170, 300, 4.0], [0.175385, 0.087692, 0.021923]),
  ('trigger sample 26', [-1.0, 542, 1000, 68, 300, 1.0], [0.469231, -0.469231, -0.469231])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
unequal arms pool the trigger rate[0.118182, 0.236364, 0.023091][0.118182, 0.236364, 0.023636]Failed
relative effect uses overall mean[0.5, 0.25, 0.047619][0.5, 0.25, 0.05]Failed
negative effect dilutes too[0.057143, -0.057143, -0.014493][0.057143, -0.057143, -0.014286]Failed
nobody triggered[0.0, 0.0, 0.0][0.0, 0.0, 0.0]Passed
zero overall meanNoneNonePassed
trigger sample 1[0.13, 0.065, 0.01599][0.13, 0.065, 0.01625]Failed
trigger sample 2[0.1325, 0.01325, 0.013077][0.1325, 0.01325, 0.01325]Failed
trigger sample 3[0.661, 0.3305, 0.031993][0.661, 0.3305, 0.03305]Failed

SHA-256 / 18f3d0647c664465f378e23725bfc78eaf01180c14c84e664ebf8465cc786671

HELD IN THE MEMBER ARCHIVE

The verified repair and its recorded checks are member-only.

This mechanism has 8 recorded checks per implementation. The open-access tier publishes the failure and the unsuccessful fix; the repaired source that passes every check, and the observations that prove it, are available to members.

Every case sharing this mechanism uses the same contract and the same repair, so this one record is held back for all of them.

Member access is invitation-based. Sign in with your invited account to inspect the repair.

Sign in to the archive ↗

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:58.438897+00:00.

Case digest / 42dbc2b661cb0770d94f4bf104bbe2a70ce6740a10766b784783200e6e03793e