{"abstract":"Site-wide relative lift is overstated by the inverse trigger rate.","category":"Experiment statistics","checks":8,"contract":"delta_trig is the absolute effect among triggered users. The trigger rate is pooled across arms: (trig_c + trig_t) / (n_c + n_t). The diluted site-wide effect is delta_trig * rate and the relative effect divides that by the overall control mean. No users or zero overall mean -> None. Return [rate, diluted absolute, diluted relative] rounded to 6.","contract_signature":"delta_trig, trig_c, n_c, trig_t, n_t, overall_mean_c","evaluation_group":"w2-experiment-statistics-trigger-dilution","failed_approach":"Dividing by the treated overall mean (control plus effect) changes the baseline.","family":"w2-experiment-statistics-trigger-dilution-relative-denominator","id":"FA-74611","implementations":{"attempt":{"sha256":"18f3d0647c664465f378e23725bfc78eaf01180c14c84e664ebf8465cc786671","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(delta_trig, trig_c, n_c, trig_t, n_t, overall_mean_c):\n    if n_c + n_t == 0 or overall_mean_c == 0:\n        return None\n    rate = (trig_c + trig_t) / (n_c + n_t)\n    absolute = delta_trig * rate\n    return [round(rate, 6), round(absolute, 6), round(absolute / (overall_mean_c + absolute), 6)]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),\n  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),\n  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),\n  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),\n  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),\n  ('trigger sample 1', [0.5, 4, 400, 61, 100, 4.0], [0.13, 0.065, 0.01625]),\n  ('trigger sample 2', [0.1, 25, 100, 28, 300, 1.0], [0.1325, 0.01325, 0.01325]),\n  ('trigger sample 3', [0.5, 640, 1000, 682, 1000, 10.0], [0.661, 0.3305, 0.03305])],\n [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),\n  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),\n  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),\n  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),\n  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),\n  ('trigger sample 4', [-1.0, 1, 1000, 191, 300, 1.0], [0.147692, -0.147692, -0.147692]),\n  ('trigger sample 5', [0.5, 661, 1000, 236, 1000, 1.0], [0.4485, 0.22425, 0.22425]),\n  ('trigger sample 6', [0.5, 791, 1000, 75, 100, 4.0], [0.787273, 0.393636, 0.098409])],\n [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),\n  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),\n  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),\n  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),\n  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),\n  ('trigger sample 11', [0.1, 40, 100, 14, 100, 4.0], [0.27, 0.027, 0.00675]),\n  ('trigger sample 12', [2.0, 213, 400, 29, 100, 4.0], [0.484, 0.968, 0.242]),\n  ('trigger sample 13', [2.0, 183, 400, 86, 300, 10.0], [0.384286, 0.768571, 0.076857])],\n [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),\n  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),\n  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),\n  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),\n  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),\n  ('trigger sample 16', [2.0, 50, 100, 332, 1000, 10.0], [0.347273, 0.694545, 0.069455]),\n  ('trigger sample 18', [0.1, 283, 400, 359, 1000, 4.0], [0.458571, 0.045857, 0.011464]),\n  ('trigger sample 19', [-1.0, 294, 1000, 72, 100, 4.0], [0.332727, -0.332727, -0.083182])],\n [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),\n  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),\n  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),\n  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),\n  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),\n  ('trigger sample 21', [-1.0, 26, 100, 60, 100, 4.0], [0.43, -0.43, -0.1075]),\n  ('trigger sample 25', [0.5, 58, 1000, 170, 300, 4.0], [0.175385, 0.087692, 0.021923]),\n  ('trigger sample 26', [-1.0, 542, 1000, 68, 300, 1.0], [0.469231, -0.469231, -0.469231])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"},"broken":{"sha256":"f8956ecad7b87684520f1807c5f6afac96e855ff29dca1889fd8ca56b8074108","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(delta_trig, trig_c, n_c, trig_t, n_t, overall_mean_c):\n    if n_c + n_t == 0 or overall_mean_c == 0:\n        return None\n    rate = (trig_c + trig_t) / (n_c + n_t)\n    absolute = delta_trig * rate\n    return [round(rate, 6), round(absolute, 6), round(delta_trig / overall_mean_c, 6)]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\nfixtures = [[('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),\n  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),\n  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),\n  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),\n  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),\n  ('trigger sample 1', [0.5, 4, 400, 61, 100, 4.0], [0.13, 0.065, 0.01625]),\n  ('trigger sample 2', [0.1, 25, 100, 28, 300, 1.0], [0.1325, 0.01325, 0.01325]),\n  ('trigger sample 3', [0.5, 640, 1000, 682, 1000, 10.0], [0.661, 0.3305, 0.03305])],\n [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),\n  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),\n  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),\n  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),\n  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),\n  ('trigger sample 4', [-1.0, 1, 1000, 191, 300, 1.0], [0.147692, -0.147692, -0.147692]),\n  ('trigger sample 5', [0.5, 661, 1000, 236, 1000, 1.0], [0.4485, 0.22425, 0.22425]),\n  ('trigger sample 6', [0.5, 791, 1000, 75, 100, 4.0], [0.787273, 0.393636, 0.098409])],\n [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),\n  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),\n  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),\n  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),\n  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),\n  ('trigger sample 11', [0.1, 40, 100, 14, 100, 4.0], [0.27, 0.027, 0.00675]),\n  ('trigger sample 12', [2.0, 213, 400, 29, 100, 4.0], [0.484, 0.968, 0.242]),\n  ('trigger sample 13', [2.0, 183, 400, 86, 300, 10.0], [0.384286, 0.768571, 0.076857])],\n [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),\n  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),\n  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),\n  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),\n  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),\n  ('trigger sample 16', [2.0, 50, 100, 332, 1000, 10.0], [0.347273, 0.694545, 0.069455]),\n  ('trigger sample 18', [0.1, 283, 400, 359, 1000, 4.0], [0.458571, 0.045857, 0.011464]),\n  ('trigger sample 19', [-1.0, 294, 1000, 72, 100, 4.0], [0.332727, -0.332727, -0.083182])],\n [('unequal arms pool the trigger rate', [2.0, 100, 1000, 30, 100, 10.0], [0.118182, 0.236364, 0.023636]),\n  ('relative effect uses overall mean', [0.5, 50, 100, 50, 100, 5.0], [0.5, 0.25, 0.05]),\n  ('negative effect dilutes too', [-1.0, 10, 400, 30, 300, 4.0], [0.057143, -0.057143, -0.014286]),\n  ('nobody triggered', [3.0, 0, 100, 0, 100, 2.0], [0.0, 0.0, 0.0]),\n  ('zero overall mean', [1.0, 5, 10, 5, 10, 0.0], None),\n  ('trigger sample 21', [-1.0, 26, 100, 60, 100, 4.0], [0.43, -0.43, -0.1075]),\n  ('trigger sample 25', [0.5, 58, 1000, 170, 300, 4.0], [0.175385, 0.087692, 0.021923]),\n  ('trigger sample 26', [-1.0, 542, 1000, 68, 300, 1.0], [0.469231, -0.469231, -0.469231])]]\nfor label, args, expected in fixtures[N - 1]:\n    check(label, solve(*args), expected)\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"}},"limitations":"A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.","method":"Deterministic executable model with adversarial boundary fixtures.","provenance":{"created_by":"Failure Map","dependencies":"Python standard library","family":"w2-experiment-statistics-trigger-dilution-relative-denominator","generated_at":"2026-09-29T14:48:58.438897+00:00","license":"CC0-1.0","python":"3.12.14","seed":1,"split":"open-access"},"relevance":"Triggered analyses are sensitive but must be diluted before claiming site-wide impact.","root_cause":"The relative effect divides delta_trig rather than the diluted effect by the overall mean.","sha256":"42dbc2b661cb0770d94f4bf104bbe2a70ce6740a10766b784783200e6e03793e","title":"Triggered effect dilution: Relative impact uses the undiluted effect · case 01","variant":1,"variant_policy":"Five numbered records share a model and may reuse boundary fixtures.","verified":true,"visibility":"public","verification":{"attempt":{"elapsed_ms":40.197,"exit_code":1,"observations":[{"actual":[0.118182,0.236364,0.023091],"check":"unequal arms pool the trigger rate","expected":[0.118182,0.236364,0.023636],"passed":false},{"actual":[0.5,0.25,0.047619],"check":"relative effect uses overall mean","expected":[0.5,0.25,0.05],"passed":false},{"actual":[0.057143,-0.057143,-0.014493],"check":"negative effect dilutes too","expected":[0.057143,-0.057143,-0.014286],"passed":false},{"actual":[0.0,0.0,0.0],"check":"nobody triggered","expected":[0.0,0.0,0.0],"passed":true},{"actual":null,"check":"zero overall mean","expected":null,"passed":true},{"actual":[0.13,0.065,0.01599],"check":"trigger sample 1","expected":[0.13,0.065,0.01625],"passed":false},{"actual":[0.1325,0.01325,0.013077],"check":"trigger sample 2","expected":[0.1325,0.01325,0.01325],"passed":false},{"actual":[0.661,0.3305,0.031993],"check":"trigger sample 3","expected":[0.661,0.3305,0.03305],"passed":false}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"unequal arms pool the trigger rate\", \"actual\": [0.118182, 0.236364, 0.023091], \"expected\": [0.118182, 0.236364, 0.023636], \"passed\": false}, {\"check\": \"relative effect uses overall mean\", \"actual\": [0.5, 0.25, 0.047619], \"expected\": [0.5, 0.25, 0.05], \"passed\": false}, {\"check\": \"negative effect dilutes too\", \"actual\": [0.057143, -0.057143, -0.014493], \"expected\": [0.057143, -0.057143, -0.014286], \"passed\": false}, {\"check\": \"nobody triggered\", \"actual\": [0.0, 0.0, 0.0], \"expected\": [0.0, 0.0, 0.0], \"passed\": true}, {\"check\": \"zero overall mean\", \"actual\": null, \"expected\": null, \"passed\": true}, {\"check\": \"trigger sample 1\", \"actual\": [0.13, 0.065, 0.01599], \"expected\": [0.13, 0.065, 0.01625], \"passed\": false}, {\"check\": \"trigger sample 2\", \"actual\": [0.1325, 0.01325, 0.013077], \"expected\": [0.1325, 0.01325, 0.01325], \"passed\": false}, {\"check\": \"trigger sample 3\", \"actual\": [0.661, 0.3305, 0.031993], \"expected\": [0.661, 0.3305, 0.03305], \"passed\": false}], \"passed\": false}\n"},"broken":{"elapsed_ms":36.538,"exit_code":1,"observations":[{"actual":[0.118182,0.236364,0.2],"check":"unequal arms pool the trigger rate","expected":[0.118182,0.236364,0.023636],"passed":false},{"actual":[0.5,0.25,0.1],"check":"relative effect uses overall mean","expected":[0.5,0.25,0.05],"passed":false},{"actual":[0.057143,-0.057143,-0.25],"check":"negative effect dilutes too","expected":[0.057143,-0.057143,-0.014286],"passed":false},{"actual":[0.0,0.0,1.5],"check":"nobody triggered","expected":[0.0,0.0,0.0],"passed":false},{"actual":null,"check":"zero overall mean","expected":null,"passed":true},{"actual":[0.13,0.065,0.125],"check":"trigger sample 1","expected":[0.13,0.065,0.01625],"passed":false},{"actual":[0.1325,0.01325,0.1],"check":"trigger sample 2","expected":[0.1325,0.01325,0.01325],"passed":false},{"actual":[0.661,0.3305,0.05],"check":"trigger sample 3","expected":[0.661,0.3305,0.03305],"passed":false}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"unequal arms pool the trigger rate\", \"actual\": [0.118182, 0.236364, 0.2], \"expected\": [0.118182, 0.236364, 0.023636], \"passed\": false}, {\"check\": \"relative effect uses overall mean\", \"actual\": [0.5, 0.25, 0.1], \"expected\": [0.5, 0.25, 0.05], \"passed\": false}, {\"check\": \"negative effect dilutes too\", \"actual\": [0.057143, -0.057143, -0.25], \"expected\": [0.057143, -0.057143, -0.014286], \"passed\": false}, {\"check\": \"nobody triggered\", \"actual\": [0.0, 0.0, 1.5], \"expected\": [0.0, 0.0, 0.0], \"passed\": false}, {\"check\": \"zero overall mean\", \"actual\": null, \"expected\": null, \"passed\": true}, {\"check\": \"trigger sample 1\", \"actual\": [0.13, 0.065, 0.125], \"expected\": [0.13, 0.065, 0.01625], \"passed\": false}, {\"check\": \"trigger sample 2\", \"actual\": [0.1325, 0.01325, 0.1], \"expected\": [0.1325, 0.01325, 0.01325], \"passed\": false}, {\"check\": \"trigger sample 3\", \"actual\": [0.661, 0.3305, 0.05], \"expected\": [0.661, 0.3305, 0.03305], \"passed\": false}], \"passed\": false}\n"}},"member_only":{"stages":["fixed"],"fields":["implementations.fixed","verification.fixed","harness","repair"],"note":"The verified repair, its recorded checks, the repair description, and the scoring harness are available to members."}}