FA-74561 / Experiment statistics / Open access
Revenue outlier capping: The cap is computed from the control arm alone · case 01
A treatment that creates more big spenders gets its gains clipped by a control-based cap.
ROOT CAUSE
The threshold is taken from sorted(control) only.
VERIFIED REPAIR
Compute one threshold from the pooled values of both arms.
Unsuccessful approach: Separate per-arm caps treat the arms differently and bias the difference.
Case contract
The cap is the nearest-rank pct-th percentile of the pooled values of both arms: sorted pooled value at 1-based rank ceil(pct/100 * N), clamped to [1, N]. Every value is capped (not dropped) at that threshold and each arm mean is taken over all its users. Empty arm -> None. Return [cap, capped control mean, capped treatment mean].
Why this case matters
Capping whales keeps revenue metrics sensitive; a per-arm cap biases the comparison itself.
1 / The failure
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(control, treatment, pct):
if not control or not treatment:
return None
pooled = sorted(control)
rank = math.ceil(pct / 100 * len(pooled))
cap = pooled[min(max(rank, 1), len(pooled)) - 1]
mc = sum(min(v, cap) for v in control) / len(control)
mt = sum(min(v, cap) for v in treatment) / len(treatment)
return [cap, round(mc, 6), round(mt, 6)]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('revenue sample 1', [[1000, 40, 300, 300], [40, 1000, 1000, 20, 5, 0], 100], [1000, 410.0, 344.166667]),
('revenue sample 2', [[20, 1000, 5, 12, 20], [12, 0], 90], [1000, 211.4, 6.0]),
('revenue sample 3', [[1000, 0, 40, 40, 10], [1000, 12, 0], 100], [1000, 218.0, 337.333333])],
[('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 6', [[300, 1000, 0, 300, 10], [0, 12, 10, 20], 90], [1000, 322.0, 10.5]),
('revenue sample 10', [[300, 20, 10], [12, 5, 0], 50], [10, 10.0, 5.0]),
('revenue sample 22', [[300, 10, 0, 0], [0, 5, 12], 80], [12, 5.5, 5.666667])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 11', [[10, 10, 10, 10, 20], [1000, 0, 40, 10], 80], [40, 12.0, 22.5]),
('revenue sample 25', [[5, 300], [12, 12, 0, 10, 10, 1000], 50], [10, 7.5, 8.333333]),
('revenue sample 53', [[300, 40, 10, 5, 12, 12], [0, 300], 80], [300, 63.166667, 150.0])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 9', [[20, 5, 0], [0, 12, 40], 50], [5, 3.333333, 3.333333]),
('revenue sample 16', [[20, 300, 1000], [0, 1000], 95], [1000, 440.0, 500.0]),
('revenue sample 41', [[12, 0], [1000], 99], [1000, 6.0, 1000.0])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 21', [[20, 300], [5, 5, 40, 20, 20], 90], [300, 160.0, 18.0]),
('revenue sample 43', [[20, 10, 0], [40, 12, 40, 10, 300], 50], [12, 7.333333, 11.6]),
('revenue sample 53', [[300, 40, 10, 5, 12, 12], [0, 300], 80], [300, 63.166667, 150.0])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| cap comes from pooled data | [10, 6.25, 7.5] | [20, 8.75, 12.5] | Failed |
| nearest rank rounds up | [2, 1.666667, 2.0] | [3, 2.0, 3.0] | Failed |
| outliers are capped not removed | [1000, 343.333333, 26.666667] | [40, 23.333333, 26.666667] | Failed |
| hundredth percentile keeps the maximum | [2, 1.5, 2.0] | [900, 1.5, 451.5] | Failed |
| tiny percentile uses the minimum | [7, 7.0, 7.0] | [7, 7.0, 7.0] | Passed |
| revenue sample 1 | [1000, 410.0, 344.166667] | [1000, 410.0, 344.166667] | Passed |
| revenue sample 2 | [1000, 211.4, 6.0] | [1000, 211.4, 6.0] | Passed |
| revenue sample 3 | [1000, 218.0, 337.333333] | [1000, 218.0, 337.333333] | Passed |
SHA-256 / b2f3e6261458d5336849228dc4b22ed8e5c5af8ccefc60202e9320b16390affa
2 / The unsuccessful fix
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(control, treatment, pct):
if not control or not treatment:
return None
def capped_mean(values):
ordered = sorted(values)
rank = math.ceil(pct / 100 * len(ordered))
cap = ordered[min(max(rank, 1), len(ordered)) - 1]
return cap, sum(min(v, cap) for v in values) / len(values)
cap_c, mc = capped_mean(control)
cap_t, mt = capped_mean(treatment)
return [max(cap_c, cap_t), round(mc, 6), round(mt, 6)]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('revenue sample 1', [[1000, 40, 300, 300], [40, 1000, 1000, 20, 5, 0], 100], [1000, 410.0, 344.166667]),
('revenue sample 2', [[20, 1000, 5, 12, 20], [12, 0], 90], [1000, 211.4, 6.0]),
('revenue sample 3', [[1000, 0, 40, 40, 10], [1000, 12, 0], 100], [1000, 218.0, 337.333333])],
[('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 6', [[300, 1000, 0, 300, 10], [0, 12, 10, 20], 90], [1000, 322.0, 10.5]),
('revenue sample 10', [[300, 20, 10], [12, 5, 0], 50], [10, 10.0, 5.0]),
('revenue sample 22', [[300, 10, 0, 0], [0, 5, 12], 80], [12, 5.5, 5.666667])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 11', [[10, 10, 10, 10, 20], [1000, 0, 40, 10], 80], [40, 12.0, 22.5]),
('revenue sample 25', [[5, 300], [12, 12, 0, 10, 10, 1000], 50], [10, 7.5, 8.333333]),
('revenue sample 53', [[300, 40, 10, 5, 12, 12], [0, 300], 80], [300, 63.166667, 150.0])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 9', [[20, 5, 0], [0, 12, 40], 50], [5, 3.333333, 3.333333]),
('revenue sample 16', [[20, 300, 1000], [0, 1000], 95], [1000, 440.0, 500.0]),
('revenue sample 41', [[12, 0], [1000], 99], [1000, 6.0, 1000.0])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 21', [[20, 300], [5, 5, 40, 20, 20], 90], [300, 160.0, 18.0]),
('revenue sample 43', [[20, 10, 0], [40, 12, 40, 10, 300], 50], [12, 7.333333, 11.6]),
('revenue sample 53', [[300, 40, 10, 5, 12, 12], [0, 300], 80], [300, 63.166667, 150.0])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| cap comes from pooled data | [20, 6.25, 12.5] | [20, 8.75, 12.5] | Failed |
| nearest rank rounds up | [4, 1.666667, 4.0] | [3, 2.0, 3.0] | Failed |
| outliers are capped not removed | [1000, 343.333333, 26.666667] | [40, 23.333333, 26.666667] | Failed |
| hundredth percentile keeps the maximum | [900, 1.5, 451.5] | [900, 1.5, 451.5] | Passed |
| tiny percentile uses the minimum | [8, 7.0, 8.0] | [7, 7.0, 7.0] | Failed |
| revenue sample 1 | [1000, 410.0, 344.166667] | [1000, 410.0, 344.166667] | Passed |
| revenue sample 2 | [1000, 211.4, 6.0] | [1000, 211.4, 6.0] | Passed |
| revenue sample 3 | [1000, 218.0, 337.333333] | [1000, 218.0, 337.333333] | Passed |
SHA-256 / e75bc3d0cb1b8887ae9d07385ab5a303849f953cb3b11de09bf03e7bbbee9c75
3 / The verified repair
Exit 0"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(control, treatment, pct):
if not control or not treatment:
return None
pooled = sorted(control + treatment)
rank = math.ceil(pct / 100 * len(pooled))
cap = pooled[min(max(rank, 1), len(pooled)) - 1]
mc = sum(min(v, cap) for v in control) / len(control)
mt = sum(min(v, cap) for v in treatment) / len(treatment)
return [cap, round(mc, 6), round(mt, 6)]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('revenue sample 1', [[1000, 40, 300, 300], [40, 1000, 1000, 20, 5, 0], 100], [1000, 410.0, 344.166667]),
('revenue sample 2', [[20, 1000, 5, 12, 20], [12, 0], 90], [1000, 211.4, 6.0]),
('revenue sample 3', [[1000, 0, 40, 40, 10], [1000, 12, 0], 100], [1000, 218.0, 337.333333])],
[('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 6', [[300, 1000, 0, 300, 10], [0, 12, 10, 20], 90], [1000, 322.0, 10.5]),
('revenue sample 10', [[300, 20, 10], [12, 5, 0], 50], [10, 10.0, 5.0]),
('revenue sample 22', [[300, 10, 0, 0], [0, 5, 12], 80], [12, 5.5, 5.666667])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 11', [[10, 10, 10, 10, 20], [1000, 0, 40, 10], 80], [40, 12.0, 22.5]),
('revenue sample 25', [[5, 300], [12, 12, 0, 10, 10, 1000], 50], [10, 7.5, 8.333333]),
('revenue sample 53', [[300, 40, 10, 5, 12, 12], [0, 300], 80], [300, 63.166667, 150.0])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 9', [[20, 5, 0], [0, 12, 40], 50], [5, 3.333333, 3.333333]),
('revenue sample 16', [[20, 300, 1000], [0, 1000], 95], [1000, 440.0, 500.0]),
('revenue sample 41', [[12, 0], [1000], 99], [1000, 6.0, 1000.0])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 21', [[20, 300], [5, 5, 40, 20, 20], 90], [300, 160.0, 18.0]),
('revenue sample 43', [[20, 10, 0], [40, 12, 40, 10, 300], 50], [12, 7.333333, 11.6]),
('revenue sample 53', [[300, 40, 10, 5, 12, 12], [0, 300], 80], [300, 63.166667, 150.0])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| cap comes from pooled data | [20, 8.75, 12.5] | [20, 8.75, 12.5] | Passed |
| nearest rank rounds up | [3, 2.0, 3.0] | [3, 2.0, 3.0] | Passed |
| outliers are capped not removed | [40, 23.333333, 26.666667] | [40, 23.333333, 26.666667] | Passed |
| hundredth percentile keeps the maximum | [900, 1.5, 451.5] | [900, 1.5, 451.5] | Passed |
| tiny percentile uses the minimum | [7, 7.0, 7.0] | [7, 7.0, 7.0] | Passed |
| revenue sample 1 | [1000, 410.0, 344.166667] | [1000, 410.0, 344.166667] | Passed |
| revenue sample 2 | [1000, 211.4, 6.0] | [1000, 211.4, 6.0] | Passed |
| revenue sample 3 | [1000, 218.0, 337.333333] | [1000, 218.0, 337.333333] | Passed |
SHA-256 / 2787d59125f63e79a27a45165903c261941296b7a1bbc53e4536a73fb80645d6
Verification & scope
A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.
Observations recorded using Python 3.12.14 at 2026-09-29T14:48:57.895726+00:00.
Case digest / 2d908bb23923fe17e0790a61150f95442847576266a44f65aaf5d3010bf055e6