FA-74571 / Experiment statistics / Open access
Revenue outlier capping: Outliers are dropped instead of capped · case 01
Revenue means understate treatment impact because whales contribute nothing.
ROOT CAUSE
Values above the cap are excluded from the sum while the denominator keeps all users.
VERIFIED REPAIR
Replace each value by min(value, cap).
Unsuccessful approach: Dropping outliers from both numerator and denominator is trimming, not capping.
Case contract
The cap is the nearest-rank pct-th percentile of the pooled values of both arms: sorted pooled value at 1-based rank ceil(pct/100 * N), clamped to [1, N]. Every value is capped (not dropped) at that threshold and each arm mean is taken over all its users. Empty arm -> None. Return [cap, capped control mean, capped treatment mean].
Why this case matters
Capping whales keeps revenue metrics sensitive; a per-arm cap biases the comparison itself.
1 / The failure
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(control, treatment, pct):
if not control or not treatment:
return None
pooled = sorted(control + treatment)
rank = math.ceil(pct / 100 * len(pooled))
cap = pooled[min(max(rank, 1), len(pooled)) - 1]
mc = sum(v for v in control if v <= cap) / len(control)
mt = sum(v for v in treatment if v <= cap) / len(treatment)
return [cap, round(mc, 6), round(mt, 6)]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('revenue sample 1', [[1000, 40, 300, 300], [40, 1000, 1000, 20, 5, 0], 100], [1000, 410.0, 344.166667]),
('revenue sample 2', [[20, 1000, 5, 12, 20], [12, 0], 90], [1000, 211.4, 6.0]),
('revenue sample 3', [[1000, 0, 40, 40, 10], [1000, 12, 0], 100], [1000, 218.0, 337.333333])],
[('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 6', [[300, 1000, 0, 300, 10], [0, 12, 10, 20], 90], [1000, 322.0, 10.5]),
('revenue sample 15', [[12], [20, 300], 50], [20, 12.0, 20.0]),
('revenue sample 22', [[300, 10, 0, 0], [0, 5, 12], 80], [12, 5.5, 5.666667])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 11', [[10, 10, 10, 10, 20], [1000, 0, 40, 10], 80], [40, 12.0, 22.5]),
('revenue sample 12', [[5, 12, 1000], [12, 300, 0, 0, 12, 5], 95], [1000, 339.0, 54.833333]),
('revenue sample 59', [[300, 0, 0, 1000], [0, 0, 0, 12, 5, 40], 80], [40, 20.0, 9.5])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 10', [[300, 20, 10], [12, 5, 0], 50], [10, 10.0, 5.0]),
('revenue sample 16', [[20, 300, 1000], [0, 1000], 95], [1000, 440.0, 500.0]),
('revenue sample 22', [[300, 10, 0, 0], [0, 5, 12], 80], [12, 5.5, 5.666667])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 21', [[20, 300], [5, 5, 40, 20, 20], 90], [300, 160.0, 18.0]),
('revenue sample 22', [[300, 10, 0, 0], [0, 5, 12], 80], [12, 5.5, 5.666667]),
('revenue sample 39', [[12], [0, 1000, 20], 50], [12, 12.0, 8.0])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| cap comes from pooled data | [20, 3.75, 7.5] | [20, 8.75, 12.5] | Failed |
| nearest rank rounds up | [3, 2.0, 0.0] | [3, 2.0, 3.0] | Failed |
| outliers are capped not removed | [40, 10.0, 26.666667] | [40, 23.333333, 26.666667] | Failed |
| hundredth percentile keeps the maximum | [900, 1.5, 451.5] | [900, 1.5, 451.5] | Passed |
| tiny percentile uses the minimum | [7, 3.5, 0.0] | [7, 7.0, 7.0] | Failed |
| revenue sample 1 | [1000, 410.0, 344.166667] | [1000, 410.0, 344.166667] | Passed |
| revenue sample 2 | [1000, 211.4, 6.0] | [1000, 211.4, 6.0] | Passed |
| revenue sample 3 | [1000, 218.0, 337.333333] | [1000, 218.0, 337.333333] | Passed |
SHA-256 / 16c01ec77a67ff6ab4c92daaa8276b565c614b6b9301f1fa2642c8fe30a0e9ec
2 / The unsuccessful fix
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(control, treatment, pct):
if not control or not treatment:
return None
pooled = sorted(control + treatment)
rank = math.ceil(pct / 100 * len(pooled))
cap = pooled[min(max(rank, 1), len(pooled)) - 1]
mc = sum(v for v in control if v <= cap) / max(1, sum(1 for v in control if v <= cap))
mt = sum(v for v in treatment if v <= cap) / max(1, sum(1 for v in treatment if v <= cap))
return [cap, round(mc, 6), round(mt, 6)]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('revenue sample 1', [[1000, 40, 300, 300], [40, 1000, 1000, 20, 5, 0], 100], [1000, 410.0, 344.166667]),
('revenue sample 2', [[20, 1000, 5, 12, 20], [12, 0], 90], [1000, 211.4, 6.0]),
('revenue sample 3', [[1000, 0, 40, 40, 10], [1000, 12, 0], 100], [1000, 218.0, 337.333333])],
[('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 6', [[300, 1000, 0, 300, 10], [0, 12, 10, 20], 90], [1000, 322.0, 10.5]),
('revenue sample 15', [[12], [20, 300], 50], [20, 12.0, 20.0]),
('revenue sample 22', [[300, 10, 0, 0], [0, 5, 12], 80], [12, 5.5, 5.666667])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 11', [[10, 10, 10, 10, 20], [1000, 0, 40, 10], 80], [40, 12.0, 22.5]),
('revenue sample 12', [[5, 12, 1000], [12, 300, 0, 0, 12, 5], 95], [1000, 339.0, 54.833333]),
('revenue sample 59', [[300, 0, 0, 1000], [0, 0, 0, 12, 5, 40], 80], [40, 20.0, 9.5])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 10', [[300, 20, 10], [12, 5, 0], 50], [10, 10.0, 5.0]),
('revenue sample 16', [[20, 300, 1000], [0, 1000], 95], [1000, 440.0, 500.0]),
('revenue sample 22', [[300, 10, 0, 0], [0, 5, 12], 80], [12, 5.5, 5.666667])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 21', [[20, 300], [5, 5, 40, 20, 20], 90], [300, 160.0, 18.0]),
('revenue sample 22', [[300, 10, 0, 0], [0, 5, 12], 80], [12, 5.5, 5.666667]),
('revenue sample 39', [[12], [0, 1000, 20], 50], [12, 12.0, 8.0])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| cap comes from pooled data | [20, 5.0, 10.0] | [20, 8.75, 12.5] | Failed |
| nearest rank rounds up | [3, 2.0, 0.0] | [3, 2.0, 3.0] | Failed |
| outliers are capped not removed | [40, 15.0, 26.666667] | [40, 23.333333, 26.666667] | Failed |
| hundredth percentile keeps the maximum | [900, 1.5, 451.5] | [900, 1.5, 451.5] | Passed |
| tiny percentile uses the minimum | [7, 7.0, 0.0] | [7, 7.0, 7.0] | Failed |
| revenue sample 1 | [1000, 410.0, 344.166667] | [1000, 410.0, 344.166667] | Passed |
| revenue sample 2 | [1000, 211.4, 6.0] | [1000, 211.4, 6.0] | Passed |
| revenue sample 3 | [1000, 218.0, 337.333333] | [1000, 218.0, 337.333333] | Passed |
SHA-256 / 6ca6a215d8edae971dfa4371a1a557a18507dd2498b0bc44a18ecde99a61f256
3 / The verified repair
Exit 0"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(control, treatment, pct):
if not control or not treatment:
return None
pooled = sorted(control + treatment)
rank = math.ceil(pct / 100 * len(pooled))
cap = pooled[min(max(rank, 1), len(pooled)) - 1]
mc = sum(min(v, cap) for v in control) / len(control)
mt = sum(min(v, cap) for v in treatment) / len(treatment)
return [cap, round(mc, 6), round(mt, 6)]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('revenue sample 1', [[1000, 40, 300, 300], [40, 1000, 1000, 20, 5, 0], 100], [1000, 410.0, 344.166667]),
('revenue sample 2', [[20, 1000, 5, 12, 20], [12, 0], 90], [1000, 211.4, 6.0]),
('revenue sample 3', [[1000, 0, 40, 40, 10], [1000, 12, 0], 100], [1000, 218.0, 337.333333])],
[('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 6', [[300, 1000, 0, 300, 10], [0, 12, 10, 20], 90], [1000, 322.0, 10.5]),
('revenue sample 15', [[12], [20, 300], 50], [20, 12.0, 20.0]),
('revenue sample 22', [[300, 10, 0, 0], [0, 5, 12], 80], [12, 5.5, 5.666667])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 11', [[10, 10, 10, 10, 20], [1000, 0, 40, 10], 80], [40, 12.0, 22.5]),
('revenue sample 12', [[5, 12, 1000], [12, 300, 0, 0, 12, 5], 95], [1000, 339.0, 54.833333]),
('revenue sample 59', [[300, 0, 0, 1000], [0, 0, 0, 12, 5, 40], 80], [40, 20.0, 9.5])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 10', [[300, 20, 10], [12, 5, 0], 50], [10, 10.0, 5.0]),
('revenue sample 16', [[20, 300, 1000], [0, 1000], 95], [1000, 440.0, 500.0]),
('revenue sample 22', [[300, 10, 0, 0], [0, 5, 12], 80], [12, 5.5, 5.666667])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 21', [[20, 300], [5, 5, 40, 20, 20], 90], [300, 160.0, 18.0]),
('revenue sample 22', [[300, 10, 0, 0], [0, 5, 12], 80], [12, 5.5, 5.666667]),
('revenue sample 39', [[12], [0, 1000, 20], 50], [12, 12.0, 8.0])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| cap comes from pooled data | [20, 8.75, 12.5] | [20, 8.75, 12.5] | Passed |
| nearest rank rounds up | [3, 2.0, 3.0] | [3, 2.0, 3.0] | Passed |
| outliers are capped not removed | [40, 23.333333, 26.666667] | [40, 23.333333, 26.666667] | Passed |
| hundredth percentile keeps the maximum | [900, 1.5, 451.5] | [900, 1.5, 451.5] | Passed |
| tiny percentile uses the minimum | [7, 7.0, 7.0] | [7, 7.0, 7.0] | Passed |
| revenue sample 1 | [1000, 410.0, 344.166667] | [1000, 410.0, 344.166667] | Passed |
| revenue sample 2 | [1000, 211.4, 6.0] | [1000, 211.4, 6.0] | Passed |
| revenue sample 3 | [1000, 218.0, 337.333333] | [1000, 218.0, 337.333333] | Passed |
SHA-256 / 8392792d657533d771be93e4bcdbac1c391db4eea990b4ace37020ce993fd91d
Verification & scope
A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.
Observations recorded using Python 3.12.14 at 2026-09-29T14:48:58.002075+00:00.
Case digest / 3735d58e2651140344b9fcdc0a38a037ecf46acbd93162cfd61dc84365444178