FAILURE MAP
← Case archive

FA-74616 / Experiment statistics / Open access

Triggered effect dilution: The triggered effect is scaled up instead of down · case 01

Reported site-wide gains exceed the triggered gains.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The effect is divided by the trigger rate.

VERIFIED REPAIR

Multiply the triggered effect by the trigger rate.

Unsuccessful approach: Leaving the effect undiluted still claims triggered gains for every user.

Case contract

delta_trig is the absolute effect among triggered users. The trigger rate is pooled across arms: (trig_c + trig_t) / (n_c + n_t). The diluted site-wide effect is delta_trig * rate and the relative effect divides that by the overall control mean. No users or zero overall mean -> None. Return [rate, diluted absolute, diluted relative] rounded to 6.

Why this case matters

Triggered analyses are sensitive but must be diluted before claiming site-wide impact.

1 / The failure

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(delta_trig, trig_c, n_c, trig_t, n_t, overall_mean_c):
    if n_c + n_t == 0 or overall_mean_c == 0:
        return None
    rate = (trig_c + trig_t) / (n_c + n_t)
    absolute = delta_trig / rate if rate else 0.0
    return [round(rate, 6), round(absolute, 6), round(absolute / overall_mean_c, 6)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 1', [0.5, 4, 400, 61, 100, 4.0], [0.13, 0.065, 0.01625]),
  ('trigger sample 2', [0.1, 25, 100, 28, 300, 1.0], [0.1325, 0.01325, 0.01325]),
  ('trigger sample 3', [0.5, 640, 1000, 682, 1000, 10.0], [0.661, 0.3305, 0.03305])],
 [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 4', [-1.0, 1, 1000, 191, 300, 1.0], [0.147692, -0.147692, -0.147692]),
  ('trigger sample 5', [0.5, 661, 1000, 236, 1000, 1.0], [0.4485, 0.22425, 0.22425]),
  ('trigger sample 6', [0.5, 791, 1000, 75, 100, 4.0], [0.787273, 0.393636, 0.098409])],
 [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 11', [0.1, 40, 100, 14, 100, 4.0], [0.27, 0.027, 0.00675]),
  ('trigger sample 12', [2.0, 213, 400, 29, 100, 4.0], [0.484, 0.968, 0.242]),
  ('trigger sample 13', [2.0, 183, 400, 86, 300, 10.0], [0.384286, 0.768571, 0.076857])],
 [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 16', [2.0, 50, 100, 332, 1000, 10.0], [0.347273, 0.694545, 0.069455]),
  ('trigger sample 18', [0.1, 283, 400, 359, 1000, 4.0], [0.458571, 0.045857, 0.011464]),
  ('trigger sample 19', [-1.0, 294, 1000, 72, 100, 4.0], [0.332727, -0.332727, -0.083182])],
 [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 21', [-1.0, 26, 100, 60, 100, 4.0], [0.43, -0.43, -0.1075]),
  ('trigger sample 25', [0.5, 58, 1000, 170, 300, 4.0], [0.175385, 0.087692, 0.021923]),
  ('trigger sample 26', [-1.0, 542, 1000, 68, 300, 1.0], [0.469231, -0.469231, -0.469231])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
unequal arms pool the trigger rate[0.118182, 16.923077, 1.692308][0.118182, 0.236364, 0.023636]Failed
relative effect uses overall mean[0.5, 1.0, 0.2][0.5, 0.25, 0.05]Failed
negative effect dilutes too[0.057143, -17.5, -4.375][0.057143, -0.057143, -0.014286]Failed
nobody triggered[0.0, 0.0, 0.0][0.0, 0.0, 0.0]Passed
zero overall meanNoneNonePassed
trigger sample 1[0.13, 3.846154, 0.961538][0.13, 0.065, 0.01625]Failed
trigger sample 2[0.1325, 0.754717, 0.754717][0.1325, 0.01325, 0.01325]Failed
trigger sample 3[0.661, 0.75643, 0.075643][0.661, 0.3305, 0.03305]Failed

SHA-256 / a705359c9cbaa99832ed56217fffb1a29cf701527b732085cdb03eee0c5106f8

2 / The unsuccessful fix

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(delta_trig, trig_c, n_c, trig_t, n_t, overall_mean_c):
    if n_c + n_t == 0 or overall_mean_c == 0:
        return None
    rate = (trig_c + trig_t) / (n_c + n_t)
    absolute = delta_trig
    return [round(rate, 6), round(absolute, 6), round(absolute / overall_mean_c, 6)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 1', [0.5, 4, 400, 61, 100, 4.0], [0.13, 0.065, 0.01625]),
  ('trigger sample 2', [0.1, 25, 100, 28, 300, 1.0], [0.1325, 0.01325, 0.01325]),
  ('trigger sample 3', [0.5, 640, 1000, 682, 1000, 10.0], [0.661, 0.3305, 0.03305])],
 [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 4', [-1.0, 1, 1000, 191, 300, 1.0], [0.147692, -0.147692, -0.147692]),
  ('trigger sample 5', [0.5, 661, 1000, 236, 1000, 1.0], [0.4485, 0.22425, 0.22425]),
  ('trigger sample 6', [0.5, 791, 1000, 75, 100, 4.0], [0.787273, 0.393636, 0.098409])],
 [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 11', [0.1, 40, 100, 14, 100, 4.0], [0.27, 0.027, 0.00675]),
  ('trigger sample 12', [2.0, 213, 400, 29, 100, 4.0], [0.484, 0.968, 0.242]),
  ('trigger sample 13', [2.0, 183, 400, 86, 300, 10.0], [0.384286, 0.768571, 0.076857])],
 [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 16', [2.0, 50, 100, 332, 1000, 10.0], [0.347273, 0.694545, 0.069455]),
  ('trigger sample 18', [0.1, 283, 400, 359, 1000, 4.0], [0.458571, 0.045857, 0.011464]),
  ('trigger sample 19', [-1.0, 294, 1000, 72, 100, 4.0], [0.332727, -0.332727, -0.083182])],
 [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 21', [-1.0, 26, 100, 60, 100, 4.0], [0.43, -0.43, -0.1075]),
  ('trigger sample 25', [0.5, 58, 1000, 170, 300, 4.0], [0.175385, 0.087692, 0.021923]),
  ('trigger sample 26', [-1.0, 542, 1000, 68, 300, 1.0], [0.469231, -0.469231, -0.469231])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
unequal arms pool the trigger rate[0.118182, 2.0, 0.2][0.118182, 0.236364, 0.023636]Failed
relative effect uses overall mean[0.5, 0.5, 0.1][0.5, 0.25, 0.05]Failed
negative effect dilutes too[0.057143, -1.0, -0.25][0.057143, -0.057143, -0.014286]Failed
nobody triggered[0.0, 3.0, 1.5][0.0, 0.0, 0.0]Failed
zero overall meanNoneNonePassed
trigger sample 1[0.13, 0.5, 0.125][0.13, 0.065, 0.01625]Failed
trigger sample 2[0.1325, 0.1, 0.1][0.1325, 0.01325, 0.01325]Failed
trigger sample 3[0.661, 0.5, 0.05][0.661, 0.3305, 0.03305]Failed

SHA-256 / aa182201627ff7737c8b9aad3ff88fe8efb48cb0cbb8c33a167534e851e15976

3 / The verified repair

Exit 0
"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(delta_trig, trig_c, n_c, trig_t, n_t, overall_mean_c):
    if n_c + n_t == 0 or overall_mean_c == 0:
        return None
    rate = (trig_c + trig_t) / (n_c + n_t)
    absolute = delta_trig * rate
    return [round(rate, 6), round(absolute, 6), round(absolute / overall_mean_c, 6)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 1', [0.5, 4, 400, 61, 100, 4.0], [0.13, 0.065, 0.01625]),
  ('trigger sample 2', [0.1, 25, 100, 28, 300, 1.0], [0.1325, 0.01325, 0.01325]),
  ('trigger sample 3', [0.5, 640, 1000, 682, 1000, 10.0], [0.661, 0.3305, 0.03305])],
 [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 4', [-1.0, 1, 1000, 191, 300, 1.0], [0.147692, -0.147692, -0.147692]),
  ('trigger sample 5', [0.5, 661, 1000, 236, 1000, 1.0], [0.4485, 0.22425, 0.22425]),
  ('trigger sample 6', [0.5, 791, 1000, 75, 100, 4.0], [0.787273, 0.393636, 0.098409])],
 [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 11', [0.1, 40, 100, 14, 100, 4.0], [0.27, 0.027, 0.00675]),
  ('trigger sample 12', [2.0, 213, 400, 29, 100, 4.0], [0.484, 0.968, 0.242]),
  ('trigger sample 13', [2.0, 183, 400, 86, 300, 10.0], [0.384286, 0.768571, 0.076857])],
 [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 16', [2.0, 50, 100, 332, 1000, 10.0], [0.347273, 0.694545, 0.069455]),
  ('trigger sample 18', [0.1, 283, 400, 359, 1000, 4.0], [0.458571, 0.045857, 0.011464]),
  ('trigger sample 19', [-1.0, 294, 1000, 72, 100, 4.0], [0.332727, -0.332727, -0.083182])],
 [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),
  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),
  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),
  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),
  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),
  ('trigger sample 21', [-1.0, 26, 100, 60, 100, 4.0], [0.43, -0.43, -0.1075]),
  ('trigger sample 25', [0.5, 58, 1000, 170, 300, 4.0], [0.175385, 0.087692, 0.021923]),
  ('trigger sample 26', [-1.0, 542, 1000, 68, 300, 1.0], [0.469231, -0.469231, -0.469231])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
unequal arms pool the trigger rate[0.118182, 0.236364, 0.023636][0.118182, 0.236364, 0.023636]Passed
relative effect uses overall mean[0.5, 0.25, 0.05][0.5, 0.25, 0.05]Passed
negative effect dilutes too[0.057143, -0.057143, -0.014286][0.057143, -0.057143, -0.014286]Passed
nobody triggered[0.0, 0.0, 0.0][0.0, 0.0, 0.0]Passed
zero overall meanNoneNonePassed
trigger sample 1[0.13, 0.065, 0.01625][0.13, 0.065, 0.01625]Passed
trigger sample 2[0.1325, 0.01325, 0.01325][0.1325, 0.01325, 0.01325]Passed
trigger sample 3[0.661, 0.3305, 0.03305][0.661, 0.3305, 0.03305]Passed

SHA-256 / fe2e4ab736a5db6001cd508060bfbef41014dc4f968d4992dd5b07aedba34e1e

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:58.438897+00:00.

Case digest / ba8415a2b8de8189e3905fe0801358620f4ec9bfbbd5d6b167f26ad6d337c788