FA-74566 / Experiment statistics / Open access
Revenue outlier capping: The cap reads one rank too high · case 01
The cap sits one order statistic above the requested percentile.
ROOT CAUSE
The 1-based rank is used as a 0-based index.
VERIFIED REPAIR
Index the sorted list at rank - 1.
Unsuccessful approach: Using a floor of pct/100 * (N - 1) switches to a different percentile definition.
Case contract
The cap is the nearest-rank pct-th percentile of the pooled values of both arms: sorted pooled value at 1-based rank ceil(pct/100 * N), clamped to [1, N]. Every value is capped (not dropped) at that threshold and each arm mean is taken over all its users. Empty arm -> None. Return [cap, capped control mean, capped treatment mean].
Why this case matters
Capping whales keeps revenue metrics sensitive; a per-arm cap biases the comparison itself.
1 / The failure
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(control, treatment, pct):
if not control or not treatment:
return None
pooled = sorted(control + treatment)
rank = math.ceil(pct / 100 * len(pooled))
cap = pooled[min(max(rank, 1), len(pooled) - 1)]
mc = sum(min(v, cap) for v in control) / len(control)
mt = sum(min(v, cap) for v in treatment) / len(treatment)
return [cap, round(mc, 6), round(mt, 6)]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('revenue sample 1', [[1000, 40, 300, 300], [40, 1000, 1000, 20, 5, 0], 100], [1000, 410.0, 344.166667]),
('revenue sample 2', [[20, 1000, 5, 12, 20], [12, 0], 90], [1000, 211.4, 6.0]),
('revenue sample 3', [[1000, 0, 40, 40, 10], [1000, 12, 0], 100], [1000, 218.0, 337.333333])],
[('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 6', [[300, 1000, 0, 300, 10], [0, 12, 10, 20], 90], [1000, 322.0, 10.5]),
('revenue sample 15', [[12], [20, 300], 50], [20, 12.0, 20.0]),
('revenue sample 20', [[20, 20, 0, 20], [12, 1000], 90], [1000, 15.0, 506.0])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 11', [[10, 10, 10, 10, 20], [1000, 0, 40, 10], 80], [40, 12.0, 22.5]),
('revenue sample 35', [[0, 40, 0, 0, 40, 0], [1000, 5, 40, 10, 10, 12], 95], [1000, 13.333333, 179.5]),
('revenue sample 59', [[300, 0, 0, 1000], [0, 0, 0, 12, 5, 40], 80], [40, 20.0, 9.5])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 10', [[300, 20, 10], [12, 5, 0], 50], [10, 10.0, 5.0]),
('revenue sample 16', [[20, 300, 1000], [0, 1000], 95], [1000, 440.0, 500.0]),
('revenue sample 46', [[1000, 10], [0, 20, 300], 95], [1000, 505.0, 106.666667])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 21', [[20, 300], [5, 5, 40, 20, 20], 90], [300, 160.0, 18.0]),
('revenue sample 43', [[20, 10, 0], [40, 12, 40, 10, 300], 50], [12, 7.333333, 11.6]),
('revenue sample 58', [[20, 0, 20], [0, 300, 12, 10, 40], 95], [300, 13.333333, 72.4])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| cap comes from pooled data | [300, 78.75, 82.5] | [20, 8.75, 12.5] | Failed |
| nearest rank rounds up | [4, 2.0, 4.0] | [3, 2.0, 3.0] | Failed |
| outliers are capped not removed | [1000, 343.333333, 26.666667] | [40, 23.333333, 26.666667] | Failed |
| hundredth percentile keeps the maximum | [900, 1.5, 451.5] | [900, 1.5, 451.5] | Passed |
| tiny percentile uses the minimum | [8, 7.5, 8.0] | [7, 7.0, 7.0] | Failed |
| revenue sample 1 | [1000, 410.0, 344.166667] | [1000, 410.0, 344.166667] | Passed |
| revenue sample 2 | [1000, 211.4, 6.0] | [1000, 211.4, 6.0] | Passed |
| revenue sample 3 | [1000, 218.0, 337.333333] | [1000, 218.0, 337.333333] | Passed |
SHA-256 / edca112811b12e10fafc90bf23dc9d9ac72e520758e1f8a8dc186b7288a0b701
2 / The unsuccessful fix
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(control, treatment, pct):
if not control or not treatment:
return None
pooled = sorted(control + treatment)
rank = math.ceil(pct / 100 * len(pooled))
cap = pooled[int(pct / 100 * (len(pooled) - 1))]
mc = sum(min(v, cap) for v in control) / len(control)
mt = sum(min(v, cap) for v in treatment) / len(treatment)
return [cap, round(mc, 6), round(mt, 6)]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('revenue sample 1', [[1000, 40, 300, 300], [40, 1000, 1000, 20, 5, 0], 100], [1000, 410.0, 344.166667]),
('revenue sample 2', [[20, 1000, 5, 12, 20], [12, 0], 90], [1000, 211.4, 6.0]),
('revenue sample 3', [[1000, 0, 40, 40, 10], [1000, 12, 0], 100], [1000, 218.0, 337.333333])],
[('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 6', [[300, 1000, 0, 300, 10], [0, 12, 10, 20], 90], [1000, 322.0, 10.5]),
('revenue sample 15', [[12], [20, 300], 50], [20, 12.0, 20.0]),
('revenue sample 20', [[20, 20, 0, 20], [12, 1000], 90], [1000, 15.0, 506.0])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 11', [[10, 10, 10, 10, 20], [1000, 0, 40, 10], 80], [40, 12.0, 22.5]),
('revenue sample 35', [[0, 40, 0, 0, 40, 0], [1000, 5, 40, 10, 10, 12], 95], [1000, 13.333333, 179.5]),
('revenue sample 59', [[300, 0, 0, 1000], [0, 0, 0, 12, 5, 40], 80], [40, 20.0, 9.5])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 10', [[300, 20, 10], [12, 5, 0], 50], [10, 10.0, 5.0]),
('revenue sample 16', [[20, 300, 1000], [0, 1000], 95], [1000, 440.0, 500.0]),
('revenue sample 46', [[1000, 10], [0, 20, 300], 95], [1000, 505.0, 106.666667])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 21', [[20, 300], [5, 5, 40, 20, 20], 90], [300, 160.0, 18.0]),
('revenue sample 43', [[20, 10, 0], [40, 12, 40, 10, 300], 50], [12, 7.333333, 11.6]),
('revenue sample 58', [[20, 0, 20], [0, 300, 12, 10, 40], 95], [300, 13.333333, 72.4])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| cap comes from pooled data | [20, 8.75, 12.5] | [20, 8.75, 12.5] | Passed |
| nearest rank rounds up | [3, 2.0, 3.0] | [3, 2.0, 3.0] | Passed |
| outliers are capped not removed | [40, 23.333333, 26.666667] | [40, 23.333333, 26.666667] | Passed |
| hundredth percentile keeps the maximum | [900, 1.5, 451.5] | [900, 1.5, 451.5] | Passed |
| tiny percentile uses the minimum | [7, 7.0, 7.0] | [7, 7.0, 7.0] | Passed |
| revenue sample 1 | [1000, 410.0, 344.166667] | [1000, 410.0, 344.166667] | Passed |
| revenue sample 2 | [20, 15.4, 6.0] | [1000, 211.4, 6.0] | Failed |
| revenue sample 3 | [1000, 218.0, 337.333333] | [1000, 218.0, 337.333333] | Passed |
SHA-256 / b17dd80aa3c7cb43a91c1e2a99b3fd584747a7eb24cdd162c91fbf1f80732c85
3 / The verified repair
Exit 0"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(control, treatment, pct):
if not control or not treatment:
return None
pooled = sorted(control + treatment)
rank = math.ceil(pct / 100 * len(pooled))
cap = pooled[min(max(rank, 1), len(pooled)) - 1]
mc = sum(min(v, cap) for v in control) / len(control)
mt = sum(min(v, cap) for v in treatment) / len(treatment)
return [cap, round(mc, 6), round(mt, 6)]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('revenue sample 1', [[1000, 40, 300, 300], [40, 1000, 1000, 20, 5, 0], 100], [1000, 410.0, 344.166667]),
('revenue sample 2', [[20, 1000, 5, 12, 20], [12, 0], 90], [1000, 211.4, 6.0]),
('revenue sample 3', [[1000, 0, 40, 40, 10], [1000, 12, 0], 100], [1000, 218.0, 337.333333])],
[('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 6', [[300, 1000, 0, 300, 10], [0, 12, 10, 20], 90], [1000, 322.0, 10.5]),
('revenue sample 15', [[12], [20, 300], 50], [20, 12.0, 20.0]),
('revenue sample 20', [[20, 20, 0, 20], [12, 1000], 90], [1000, 15.0, 506.0])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 11', [[10, 10, 10, 10, 20], [1000, 0, 40, 10], 80], [40, 12.0, 22.5]),
('revenue sample 35', [[0, 40, 0, 0, 40, 0], [1000, 5, 40, 10, 10, 12], 95], [1000, 13.333333, 179.5]),
('revenue sample 59', [[300, 0, 0, 1000], [0, 0, 0, 12, 5, 40], 80], [40, 20.0, 9.5])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 10', [[300, 20, 10], [12, 5, 0], 50], [10, 10.0, 5.0]),
('revenue sample 16', [[20, 300, 1000], [0, 1000], 95], [1000, 440.0, 500.0]),
('revenue sample 46', [[1000, 10], [0, 20, 300], 95], [1000, 505.0, 106.666667])],
[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
('empty treatment', [[1], [], 90], None),
('revenue sample 21', [[20, 300], [5, 5, 40, 20, 20], 90], [300, 160.0, 18.0]),
('revenue sample 43', [[20, 10, 0], [40, 12, 40, 10, 300], 50], [12, 7.333333, 11.6]),
('revenue sample 58', [[20, 0, 20], [0, 300, 12, 10, 40], 95], [300, 13.333333, 72.4])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| cap comes from pooled data | [20, 8.75, 12.5] | [20, 8.75, 12.5] | Passed |
| nearest rank rounds up | [3, 2.0, 3.0] | [3, 2.0, 3.0] | Passed |
| outliers are capped not removed | [40, 23.333333, 26.666667] | [40, 23.333333, 26.666667] | Passed |
| hundredth percentile keeps the maximum | [900, 1.5, 451.5] | [900, 1.5, 451.5] | Passed |
| tiny percentile uses the minimum | [7, 7.0, 7.0] | [7, 7.0, 7.0] | Passed |
| revenue sample 1 | [1000, 410.0, 344.166667] | [1000, 410.0, 344.166667] | Passed |
| revenue sample 2 | [1000, 211.4, 6.0] | [1000, 211.4, 6.0] | Passed |
| revenue sample 3 | [1000, 218.0, 337.333333] | [1000, 218.0, 337.333333] | Passed |
SHA-256 / dbc95746b55e0fb2cd766c08494153ec7aa8bd7eb83397b6bcaca4241cb49117
Verification & scope
A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.
Observations recorded using Python 3.12.14 at 2026-09-29T14:48:57.963773+00:00.
Case digest / 32092b9f90654d8c1e8e1a832a2a9e72525e474a904b65166bee60d8fbb7758b