FAILURE MAP
← Case archive

FA-74576 / Experiment statistics / Open access

Revenue outlier capping: Percentile rank is rounded to nearest · case 01

Small experiments cap at a lower order statistic than the nearest-rank rule allows.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The rank is round(pct/100 * N) instead of ceil.

VERIFIED REPAIR

Use ceil(pct/100 * N) for the nearest rank.

Unsuccessful approach: Truncating the rank moves the cap down even further.

Case contract

The cap is the nearest-rank pct-th percentile of the pooled values of both arms: sorted pooled value at 1-based rank ceil(pct/100 * N), clamped to [1, N]. Every value is capped (not dropped) at that threshold and each arm mean is taken over all its users. Empty arm -> None. Return [cap, capped control mean, capped treatment mean].

Why this case matters

Capping whales keeps revenue metrics sensitive; a per-arm cap biases the comparison itself.

1 / The failure

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(control, treatment, pct):
    if not control or not treatment:
        return None
    pooled = sorted(control + treatment)
    rank = round(pct / 100 * len(pooled))
    cap = pooled[min(max(rank, 1), len(pooled)) - 1]
    mc = sum(min(v, cap) for v in control) / len(control)
    mt = sum(min(v, cap) for v in treatment) / len(treatment)
    return [cap, round(mc, 6), round(mt, 6)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
  ('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
  ('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
  ('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
  ('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
  ('revenue sample 1', [[1000, 40, 300, 300], [40, 1000, 1000, 20, 5, 0], 100], [1000, 410.0, 344.166667]),
  ('revenue sample 2', [[20, 1000, 5, 12, 20], [12, 0], 90], [1000, 211.4, 6.0]),
  ('revenue sample 3', [[1000, 0, 40, 40, 10], [1000, 12, 0], 100], [1000, 218.0, 337.333333])],
 [('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
  ('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
  ('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
  ('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
  ('empty treatment', [[1], [], 90], None),
  ('revenue sample 6', [[300, 1000, 0, 300, 10], [0, 12, 10, 20], 90], [1000, 322.0, 10.5]),
  ('revenue sample 14', [[12, 300], [0, 40, 0, 20], 95], [300, 156.0, 15.0]),
  ('revenue sample 35', [[0, 40, 0, 0, 40, 0], [1000, 5, 40, 10, 10, 12], 95], [1000, 13.333333, 179.5])],
 [('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
  ('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
  ('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
  ('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
  ('empty treatment', [[1], [], 90], None),
  ('revenue sample 11', [[10, 10, 10, 10, 20], [1000, 0, 40, 10], 80], [40, 12.0, 22.5]),
  ('revenue sample 12', [[5, 12, 1000], [12, 300, 0, 0, 12, 5], 95], [1000, 339.0, 54.833333]),
  ('revenue sample 30', [[12, 40, 1000, 300, 0], [10, 5, 0, 10], 90], [1000, 270.4, 6.25])],
 [('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
  ('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
  ('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
  ('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
  ('empty treatment', [[1], [], 90], None),
  ('revenue sample 16', [[20, 300, 1000], [0, 1000], 95], [1000, 440.0, 500.0]),
  ('revenue sample 40', [[1000, 0], [5], 80], [1000, 500.0, 5.0]),
  ('revenue sample 55', [[12, 5], [0, 20, 40, 12, 10], 90], [40, 8.5, 16.4])],
 [('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
  ('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
  ('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
  ('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
  ('empty treatment', [[1], [], 90], None),
  ('revenue sample 21', [[20, 300], [5, 5, 40, 20, 20], 90], [300, 160.0, 18.0]),
  ('revenue sample 30', [[12, 40, 1000, 300, 0], [10, 5, 0, 10], 90], [1000, 270.4, 6.25]),
  ('revenue sample 52', [[0, 40, 5], [1000, 20, 300], 95], [1000, 15.0, 440.0])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
cap comes from pooled data[20, 8.75, 12.5][20, 8.75, 12.5]Passed
nearest rank rounds up[2, 1.666667, 2.0][3, 2.0, 3.0]Failed
outliers are capped not removed[40, 23.333333, 26.666667][40, 23.333333, 26.666667]Passed
hundredth percentile keeps the maximum[900, 1.5, 451.5][900, 1.5, 451.5]Passed
tiny percentile uses the minimum[7, 7.0, 7.0][7, 7.0, 7.0]Passed
revenue sample 1[1000, 410.0, 344.166667][1000, 410.0, 344.166667]Passed
revenue sample 2[20, 15.4, 6.0][1000, 211.4, 6.0]Failed
revenue sample 3[1000, 218.0, 337.333333][1000, 218.0, 337.333333]Passed

SHA-256 / 3a99d1099a07595d3815f646d23773cb25faa88fac4ea4f76066017a20fa0ff2

2 / The unsuccessful fix

Exit 1
"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(control, treatment, pct):
    if not control or not treatment:
        return None
    pooled = sorted(control + treatment)
    rank = int(pct / 100 * len(pooled))
    cap = pooled[min(max(rank, 1), len(pooled)) - 1]
    mc = sum(min(v, cap) for v in control) / len(control)
    mt = sum(min(v, cap) for v in treatment) / len(treatment)
    return [cap, round(mc, 6), round(mt, 6)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
  ('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
  ('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
  ('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
  ('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
  ('revenue sample 1', [[1000, 40, 300, 300], [40, 1000, 1000, 20, 5, 0], 100], [1000, 410.0, 344.166667]),
  ('revenue sample 2', [[20, 1000, 5, 12, 20], [12, 0], 90], [1000, 211.4, 6.0]),
  ('revenue sample 3', [[1000, 0, 40, 40, 10], [1000, 12, 0], 100], [1000, 218.0, 337.333333])],
 [('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
  ('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
  ('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
  ('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
  ('empty treatment', [[1], [], 90], None),
  ('revenue sample 6', [[300, 1000, 0, 300, 10], [0, 12, 10, 20], 90], [1000, 322.0, 10.5]),
  ('revenue sample 14', [[12, 300], [0, 40, 0, 20], 95], [300, 156.0, 15.0]),
  ('revenue sample 35', [[0, 40, 0, 0, 40, 0], [1000, 5, 40, 10, 10, 12], 95], [1000, 13.333333, 179.5])],
 [('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
  ('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
  ('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
  ('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
  ('empty treatment', [[1], [], 90], None),
  ('revenue sample 11', [[10, 10, 10, 10, 20], [1000, 0, 40, 10], 80], [40, 12.0, 22.5]),
  ('revenue sample 12', [[5, 12, 1000], [12, 300, 0, 0, 12, 5], 95], [1000, 339.0, 54.833333]),
  ('revenue sample 30', [[12, 40, 1000, 300, 0], [10, 5, 0, 10], 90], [1000, 270.4, 6.25])],
 [('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
  ('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
  ('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
  ('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
  ('empty treatment', [[1], [], 90], None),
  ('revenue sample 16', [[20, 300, 1000], [0, 1000], 95], [1000, 440.0, 500.0]),
  ('revenue sample 40', [[1000, 0], [5], 80], [1000, 500.0, 5.0]),
  ('revenue sample 55', [[12, 5], [0, 20, 40, 12, 10], 90], [40, 8.5, 16.4])],
 [('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
  ('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
  ('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
  ('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
  ('empty treatment', [[1], [], 90], None),
  ('revenue sample 21', [[20, 300], [5, 5, 40, 20, 20], 90], [300, 160.0, 18.0]),
  ('revenue sample 30', [[12, 40, 1000, 300, 0], [10, 5, 0, 10], 90], [1000, 270.4, 6.25]),
  ('revenue sample 52', [[0, 40, 5], [1000, 20, 300], 95], [1000, 15.0, 440.0])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
cap comes from pooled data[20, 8.75, 12.5][20, 8.75, 12.5]Passed
nearest rank rounds up[2, 1.666667, 2.0][3, 2.0, 3.0]Failed
outliers are capped not removed[30, 20.0, 23.333333][40, 23.333333, 26.666667]Failed
hundredth percentile keeps the maximum[900, 1.5, 451.5][900, 1.5, 451.5]Passed
tiny percentile uses the minimum[7, 7.0, 7.0][7, 7.0, 7.0]Passed
revenue sample 1[1000, 410.0, 344.166667][1000, 410.0, 344.166667]Passed
revenue sample 2[20, 15.4, 6.0][1000, 211.4, 6.0]Failed
revenue sample 3[1000, 218.0, 337.333333][1000, 218.0, 337.333333]Passed

SHA-256 / 7550a4d47568ab828f9f76c96f9a7557f0342965428a7bf97f6982188198267a

3 / The verified repair

Exit 0
"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(control, treatment, pct):
    if not control or not treatment:
        return None
    pooled = sorted(control + treatment)
    rank = math.ceil(pct / 100 * len(pooled))
    cap = pooled[min(max(rank, 1), len(pooled)) - 1]
    mc = sum(min(v, cap) for v in control) / len(control)
    mt = sum(min(v, cap) for v in treatment) / len(treatment)
    return [cap, round(mc, 6), round(mt, 6)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
  ('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
  ('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
  ('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
  ('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
  ('revenue sample 1', [[1000, 40, 300, 300], [40, 1000, 1000, 20, 5, 0], 100], [1000, 410.0, 344.166667]),
  ('revenue sample 2', [[20, 1000, 5, 12, 20], [12, 0], 90], [1000, 211.4, 6.0]),
  ('revenue sample 3', [[1000, 0, 40, 40, 10], [1000, 12, 0], 100], [1000, 218.0, 337.333333])],
 [('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
  ('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
  ('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
  ('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
  ('empty treatment', [[1], [], 90], None),
  ('revenue sample 6', [[300, 1000, 0, 300, 10], [0, 12, 10, 20], 90], [1000, 322.0, 10.5]),
  ('revenue sample 14', [[12, 300], [0, 40, 0, 20], 95], [300, 156.0, 15.0]),
  ('revenue sample 35', [[0, 40, 0, 0, 40, 0], [1000, 5, 40, 10, 10, 12], 95], [1000, 13.333333, 179.5])],
 [('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
  ('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
  ('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
  ('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
  ('empty treatment', [[1], [], 90], None),
  ('revenue sample 11', [[10, 10, 10, 10, 20], [1000, 0, 40, 10], 80], [40, 12.0, 22.5]),
  ('revenue sample 12', [[5, 12, 1000], [12, 300, 0, 0, 12, 5], 95], [1000, 339.0, 54.833333]),
  ('revenue sample 30', [[12, 40, 1000, 300, 0], [10, 5, 0, 10], 90], [1000, 270.4, 6.25])],
 [('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
  ('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
  ('hundredth percentile keeps the maximum', [[1, 2], [3, 900], 100], [900, 1.5, 451.5]),
  ('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
  ('empty treatment', [[1], [], 90], None),
  ('revenue sample 16', [[20, 300, 1000], [0, 1000], 95], [1000, 440.0, 500.0]),
  ('revenue sample 40', [[1000, 0], [5], 80], [1000, 500.0, 5.0]),
  ('revenue sample 55', [[12, 5], [0, 20, 40, 12, 10], 90], [40, 8.5, 16.4])],
 [('cap comes from pooled data', [[0, 5, 10, 1000], [5, 5, 20, 300], 75], [20, 8.75, 12.5]),
  ('nearest rank rounds up', [[1, 2, 3], [4, 5], 50], [3, 2.0, 3.0]),
  ('outliers are capped not removed', [[10, 20, 1000], [10, 30, 40], 80], [40, 23.333333, 26.666667]),
  ('tiny percentile uses the minimum', [[7, 9], [8, 10], 1], [7, 7.0, 7.0]),
  ('empty treatment', [[1], [], 90], None),
  ('revenue sample 21', [[20, 300], [5, 5, 40, 20, 20], 90], [300, 160.0, 18.0]),
  ('revenue sample 30', [[12, 40, 1000, 300, 0], [10, 5, 0, 10], 90], [1000, 270.4, 6.25]),
  ('revenue sample 52', [[0, 40, 5], [1000, 20, 300], 95], [1000, 15.0, 440.0])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
Boundary fixtureActualExpectedOutcome
cap comes from pooled data[20, 8.75, 12.5][20, 8.75, 12.5]Passed
nearest rank rounds up[3, 2.0, 3.0][3, 2.0, 3.0]Passed
outliers are capped not removed[40, 23.333333, 26.666667][40, 23.333333, 26.666667]Passed
hundredth percentile keeps the maximum[900, 1.5, 451.5][900, 1.5, 451.5]Passed
tiny percentile uses the minimum[7, 7.0, 7.0][7, 7.0, 7.0]Passed
revenue sample 1[1000, 410.0, 344.166667][1000, 410.0, 344.166667]Passed
revenue sample 2[1000, 211.4, 6.0][1000, 211.4, 6.0]Passed
revenue sample 3[1000, 218.0, 337.333333][1000, 218.0, 337.333333]Passed

SHA-256 / ff01a6beddcd28aa1c7948506fb530b61d74897538486759d3113a75e9f666b6

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:58.002297+00:00.

Case digest / 43793de4a6e424ca7f256e5c85ff192589fb0e63ee5528aace32ae92333e7d52