FA-74476 / Experiment statistics / Open access

Holm correction across metrics: Rejection continues after a failed step · case 01

A large p-value followed by a lucky threshold rejects metrics Holm would keep.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

Each sorted p-value is tested independently; the stop flag is ignored.

VERIFIED REPAIR

Stop rejecting at the first sorted p-value that fails its threshold.

Unsuccessful approach: Making the comparison strict drops rejections of p-values that sit exactly on a threshold.

Case contract

Holm step-down: sort p-values ascending (ties by position); reject while p_(k) <= alpha / (m - k) for k = 0, 1, ... and stop at the first failure. Adjusted p_(k) = running max of min(1, (m - k) p_(k)). Return [rejections, adjusted p-values rounded to 6] in the original metric order.

Why this case matters

Experiments track many metrics; multiplicity control keeps secondary wins honest.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(pvalues, alpha):
    m = len(pvalues)
    order = sorted(range(m), key=lambda i: (pvalues[i], i))
    reject = [False] * m
    adjusted = [0.0] * m
    running = 0.0
    stopped = False
    for rank, i in enumerate(order):
        running = max(running, min(1.0, (m - rank) * pvalues[i]))
        adjusted[i] = round(running, 6)
        if pvalues[i] <= alpha / (m - rank):
            reject[i] = True
        else:
            stopped = True
    return [reject, adjusted]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('all three pass their step thresholds',
   [[0.01, 0.04, 0.012], 0.05],
   [[True, True, True], [0.03, 0.04, 0.03]]),
  ('step-down stops at first non-rejection',
   [[0.03, 0.02, 0.04], 0.05],
   [[False, False, False], [0.06, 0.06, 0.06]]),
  ('boundary p equals alpha over remaining',
   [[0.0125, 0.5, 0.9, 0.9], 0.05],
   [[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]),
  ('adjusted values are monotone', [[0.01, 0.011, 0.5], 0.05], [[True, True, False], [0.03, 0.03, 0.5]]),
  ('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
  ('single metric', [[0.05], 0.05], [[True], [0.05]]),
  ('metric p-value sample 1',
   [[0.025, 0.02, 0.05, 0.6], 0.05],
   [[False, False, False, False], [0.08, 0.08, 0.1, 0.6]]),
  ('metric p-value sample 2',
   [[0.0125, 0.03, 0.0167], 0.05],
   [[True, True, True], [0.0375, 0.0375, 0.0375]])],
 [('step-down stops at first non-rejection',
   [[0.03, 0.02, 0.04], 0.05],
   [[False, False, False], [0.06, 0.06, 0.06]]),
  ('boundary p equals alpha over remaining',
   [[0.0125, 0.5, 0.9, 0.9], 0.05],
   [[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]),
  ('adjusted values are monotone', [[0.01, 0.011, 0.5], 0.05], [[True, True, False], [0.03, 0.03, 0.5]]),
  ('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
  ('original order is preserved', [[0.3, 0.001, 0.02], 0.05], [[False, True, True], [0.3, 0.003, 0.04]]),
  ('single metric', [[0.05], 0.05], [[True], [0.05]]),
  ('metric p-value sample 21',
   [[0.0167, 0.6, 0.2, 0.02, 0.01], 0.05],
   [[False, False, False, False, True], [0.0668, 0.6, 0.4, 0.0668, 0.05]]),
  ('metric p-value sample 43',
   [[0.001, 0.03, 0.03, 0.0167, 0.025], 0.05],
   [[True, False, False, False, False], [0.005, 0.075, 0.075, 0.0668, 0.075]])],
 [('boundary p equals alpha over remaining',
   [[0.0125, 0.5, 0.9, 0.9], 0.05],
   [[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]),
  ('adjusted values are monotone', [[0.01, 0.011, 0.5], 0.05], [[True, True, False], [0.03, 0.03, 0.5]]),
  ('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
  ('original order is preserved', [[0.3, 0.001, 0.02], 0.05], [[False, True, True], [0.3, 0.003, 0.04]]),
  ('holm rejects more than bonferroni', [[0.01, 0.02], 0.05], [[True, True], [0.02, 0.02]]),
  ('single metric', [[0.05], 0.05], [[True], [0.05]]),
  ('metric p-value sample 11', [[0.01, 0.001, 0.02], 0.05], [[True, True, True], [0.02, 0.003, 0.02]]),
  ('metric p-value sample 20', [[0.03, 0.001, 0.04], 0.05], [[False, True, False], [0.06, 0.003, 0.06]])],
 [('adjusted values are monotone', [[0.01, 0.011, 0.5], 0.05], [[True, True, False], [0.03, 0.03, 0.5]]),
  ('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
  ('original order is preserved', [[0.3, 0.001, 0.02], 0.05], [[False, True, True], [0.3, 0.003, 0.04]]),
  ('holm rejects more than bonferroni', [[0.01, 0.02], 0.05], [[True, True], [0.02, 0.02]]),
  ('single metric', [[0.05], 0.05], [[True], [0.05]]),
  ('metric p-value sample 16',
   [[0.03, 0.02, 0.2, 0.6, 0.0167], 0.05],
   [[False, False, False, False, False], [0.09, 0.0835, 0.4, 0.6, 0.0835]]),
  ('metric p-value sample 35', [[0.025, 0.03], 0.05], [[True, True], [0.05, 0.05]]),
  ('metric p-value sample 59',
   [[0.03, 0.02, 0.0125, 0.04, 0.03], 0.05],
   [[False, False, False, False, False], [0.09, 0.08, 0.0625, 0.09, 0.09]])],
 [('all three pass their step thresholds',
   [[0.01, 0.04, 0.012], 0.05],
   [[True, True, True], [0.03, 0.04, 0.03]]),
  ('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
  ('original order is preserved', [[0.3, 0.001, 0.02], 0.05], [[False, True, True], [0.3, 0.003, 0.04]]),
  ('holm rejects more than bonferroni', [[0.01, 0.02], 0.05], [[True, True], [0.02, 0.02]]),
  ('single metric', [[0.05], 0.05], [[True], [0.05]]),
  ('metric p-value sample 4', [[0.0125, 0.0167, 0.05], 0.05], [[True, True, True], [0.0375, 0.0375, 0.05]]),
  ('metric p-value sample 21',
   [[0.0167, 0.6, 0.2, 0.02, 0.01], 0.05],
   [[False, False, False, False, True], [0.0668, 0.6, 0.4, 0.0668, 0.05]]),
  ('metric p-value sample 34',
   [[0.01, 0.0167, 0.001, 0.05, 0.04], 0.05],
   [[True, False, True, False, False], [0.04, 0.0501, 0.005, 0.08, 0.08]])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
all three pass their step thresholds	`[[True, True, True], [0.03, 0.04, 0.03]]`	`[[True, True, True], [0.03, 0.04, 0.03]]`	Passed
step-down stops at first non-rejection	`[[False, False, True], [0.06, 0.06, 0.06]]`	`[[False, False, False], [0.06, 0.06, 0.06]]`	Failed
boundary p equals alpha over remaining	`[[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]`	`[[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]`	Passed
adjusted values are monotone	`[[True, True, False], [0.03, 0.03, 0.5]]`	`[[True, True, False], [0.03, 0.03, 0.5]]`	Passed
adjusted values cap at one	`[[False, False], [0.8, 0.8]]`	`[[False, False], [0.8, 0.8]]`	Passed
single metric	`[[True], [0.05]]`	`[[True], [0.05]]`	Passed
metric p-value sample 1	`[[False, False, False, False], [0.08, 0.08, 0.1, 0.6]]`	`[[False, False, False, False], [0.08, 0.08, 0.1, 0.6]]`	Passed
metric p-value sample 2	`[[True, True, True], [0.0375, 0.0375, 0.0375]]`	`[[True, True, True], [0.0375, 0.0375, 0.0375]]`	Passed

SHA-256 / a8112cb64dc2f56929c712c414eeb83ff03b5fedc0e94ad281253ce0741d8c5d

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(pvalues, alpha):
    m = len(pvalues)
    order = sorted(range(m), key=lambda i: (pvalues[i], i))
    reject = [False] * m
    adjusted = [0.0] * m
    running = 0.0
    stopped = False
    for rank, i in enumerate(order):
        running = max(running, min(1.0, (m - rank) * pvalues[i]))
        adjusted[i] = round(running, 6)
        if not stopped and pvalues[i] < alpha / (m - rank):
            reject[i] = True
        else:
            stopped = True
    return [reject, adjusted]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('all three pass their step thresholds',
   [[0.01, 0.04, 0.012], 0.05],
   [[True, True, True], [0.03, 0.04, 0.03]]),
  ('step-down stops at first non-rejection',
   [[0.03, 0.02, 0.04], 0.05],
   [[False, False, False], [0.06, 0.06, 0.06]]),
  ('boundary p equals alpha over remaining',
   [[0.0125, 0.5, 0.9, 0.9], 0.05],
   [[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]),
  ('adjusted values are monotone', [[0.01, 0.011, 0.5], 0.05], [[True, True, False], [0.03, 0.03, 0.5]]),
  ('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
  ('single metric', [[0.05], 0.05], [[True], [0.05]]),
  ('metric p-value sample 1',
   [[0.025, 0.02, 0.05, 0.6], 0.05],
   [[False, False, False, False], [0.08, 0.08, 0.1, 0.6]]),
  ('metric p-value sample 2',
   [[0.0125, 0.03, 0.0167], 0.05],
   [[True, True, True], [0.0375, 0.0375, 0.0375]])],
 [('step-down stops at first non-rejection',
   [[0.03, 0.02, 0.04], 0.05],
   [[False, False, False], [0.06, 0.06, 0.06]]),
  ('boundary p equals alpha over remaining',
   [[0.0125, 0.5, 0.9, 0.9], 0.05],
   [[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]),
  ('adjusted values are monotone', [[0.01, 0.011, 0.5], 0.05], [[True, True, False], [0.03, 0.03, 0.5]]),
  ('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
  ('original order is preserved', [[0.3, 0.001, 0.02], 0.05], [[False, True, True], [0.3, 0.003, 0.04]]),
  ('single metric', [[0.05], 0.05], [[True], [0.05]]),
  ('metric p-value sample 21',
   [[0.0167, 0.6, 0.2, 0.02, 0.01], 0.05],
   [[False, False, False, False, True], [0.0668, 0.6, 0.4, 0.0668, 0.05]]),
  ('metric p-value sample 43',
   [[0.001, 0.03, 0.03, 0.0167, 0.025], 0.05],
   [[True, False, False, False, False], [0.005, 0.075, 0.075, 0.0668, 0.075]])],
 [('boundary p equals alpha over remaining',
   [[0.0125, 0.5, 0.9, 0.9], 0.05],
   [[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]),
  ('adjusted values are monotone', [[0.01, 0.011, 0.5], 0.05], [[True, True, False], [0.03, 0.03, 0.5]]),
  ('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
  ('original order is preserved', [[0.3, 0.001, 0.02], 0.05], [[False, True, True], [0.3, 0.003, 0.04]]),
  ('holm rejects more than bonferroni', [[0.01, 0.02], 0.05], [[True, True], [0.02, 0.02]]),
  ('single metric', [[0.05], 0.05], [[True], [0.05]]),
  ('metric p-value sample 11', [[0.01, 0.001, 0.02], 0.05], [[True, True, True], [0.02, 0.003, 0.02]]),
  ('metric p-value sample 20', [[0.03, 0.001, 0.04], 0.05], [[False, True, False], [0.06, 0.003, 0.06]])],
 [('adjusted values are monotone', [[0.01, 0.011, 0.5], 0.05], [[True, True, False], [0.03, 0.03, 0.5]]),
  ('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
  ('original order is preserved', [[0.3, 0.001, 0.02], 0.05], [[False, True, True], [0.3, 0.003, 0.04]]),
  ('holm rejects more than bonferroni', [[0.01, 0.02], 0.05], [[True, True], [0.02, 0.02]]),
  ('single metric', [[0.05], 0.05], [[True], [0.05]]),
  ('metric p-value sample 16',
   [[0.03, 0.02, 0.2, 0.6, 0.0167], 0.05],
   [[False, False, False, False, False], [0.09, 0.0835, 0.4, 0.6, 0.0835]]),
  ('metric p-value sample 35', [[0.025, 0.03], 0.05], [[True, True], [0.05, 0.05]]),
  ('metric p-value sample 59',
   [[0.03, 0.02, 0.0125, 0.04, 0.03], 0.05],
   [[False, False, False, False, False], [0.09, 0.08, 0.0625, 0.09, 0.09]])],
 [('all three pass their step thresholds',
   [[0.01, 0.04, 0.012], 0.05],
   [[True, True, True], [0.03, 0.04, 0.03]]),
  ('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
  ('original order is preserved', [[0.3, 0.001, 0.02], 0.05], [[False, True, True], [0.3, 0.003, 0.04]]),
  ('holm rejects more than bonferroni', [[0.01, 0.02], 0.05], [[True, True], [0.02, 0.02]]),
  ('single metric', [[0.05], 0.05], [[True], [0.05]]),
  ('metric p-value sample 4', [[0.0125, 0.0167, 0.05], 0.05], [[True, True, True], [0.0375, 0.0375, 0.05]]),
  ('metric p-value sample 21',
   [[0.0167, 0.6, 0.2, 0.02, 0.01], 0.05],
   [[False, False, False, False, True], [0.0668, 0.6, 0.4, 0.0668, 0.05]]),
  ('metric p-value sample 34',
   [[0.01, 0.0167, 0.001, 0.05, 0.04], 0.05],
   [[True, False, True, False, False], [0.04, 0.0501, 0.005, 0.08, 0.08]])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
all three pass their step thresholds	`[[True, True, True], [0.03, 0.04, 0.03]]`	`[[True, True, True], [0.03, 0.04, 0.03]]`	Passed
step-down stops at first non-rejection	`[[False, False, False], [0.06, 0.06, 0.06]]`	`[[False, False, False], [0.06, 0.06, 0.06]]`	Passed
boundary p equals alpha over remaining	`[[False, False, False, False], [0.05, 1.0, 1.0, 1.0]]`	`[[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]`	Failed
adjusted values are monotone	`[[True, True, False], [0.03, 0.03, 0.5]]`	`[[True, True, False], [0.03, 0.03, 0.5]]`	Passed
adjusted values cap at one	`[[False, False], [0.8, 0.8]]`	`[[False, False], [0.8, 0.8]]`	Passed
single metric	`[[False], [0.05]]`	`[[True], [0.05]]`	Failed
metric p-value sample 1	`[[False, False, False, False], [0.08, 0.08, 0.1, 0.6]]`	`[[False, False, False, False], [0.08, 0.08, 0.1, 0.6]]`	Passed
metric p-value sample 2	`[[True, True, True], [0.0375, 0.0375, 0.0375]]`	`[[True, True, True], [0.0375, 0.0375, 0.0375]]`	Passed

SHA-256 / 47bae650b41e2dc67e4aad388d1688f75e16ac75b7d2bf2d24bc7c8f99f92ca9

3 / The verified repair

Exit 0

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(pvalues, alpha):
    m = len(pvalues)
    order = sorted(range(m), key=lambda i: (pvalues[i], i))
    reject = [False] * m
    adjusted = [0.0] * m
    running = 0.0
    stopped = False
    for rank, i in enumerate(order):
        running = max(running, min(1.0, (m - rank) * pvalues[i]))
        adjusted[i] = round(running, 6)
        if not stopped and pvalues[i] <= alpha / (m - rank):
            reject[i] = True
        else:
            stopped = True
    return [reject, adjusted]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('all three pass their step thresholds',
   [[0.01, 0.04, 0.012], 0.05],
   [[True, True, True], [0.03, 0.04, 0.03]]),
  ('step-down stops at first non-rejection',
   [[0.03, 0.02, 0.04], 0.05],
   [[False, False, False], [0.06, 0.06, 0.06]]),
  ('boundary p equals alpha over remaining',
   [[0.0125, 0.5, 0.9, 0.9], 0.05],
   [[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]),
  ('adjusted values are monotone', [[0.01, 0.011, 0.5], 0.05], [[True, True, False], [0.03, 0.03, 0.5]]),
  ('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
  ('single metric', [[0.05], 0.05], [[True], [0.05]]),
  ('metric p-value sample 1',
   [[0.025, 0.02, 0.05, 0.6], 0.05],
   [[False, False, False, False], [0.08, 0.08, 0.1, 0.6]]),
  ('metric p-value sample 2',
   [[0.0125, 0.03, 0.0167], 0.05],
   [[True, True, True], [0.0375, 0.0375, 0.0375]])],
 [('step-down stops at first non-rejection',
   [[0.03, 0.02, 0.04], 0.05],
   [[False, False, False], [0.06, 0.06, 0.06]]),
  ('boundary p equals alpha over remaining',
   [[0.0125, 0.5, 0.9, 0.9], 0.05],
   [[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]),
  ('adjusted values are monotone', [[0.01, 0.011, 0.5], 0.05], [[True, True, False], [0.03, 0.03, 0.5]]),
  ('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
  ('original order is preserved', [[0.3, 0.001, 0.02], 0.05], [[False, True, True], [0.3, 0.003, 0.04]]),
  ('single metric', [[0.05], 0.05], [[True], [0.05]]),
  ('metric p-value sample 21',
   [[0.0167, 0.6, 0.2, 0.02, 0.01], 0.05],
   [[False, False, False, False, True], [0.0668, 0.6, 0.4, 0.0668, 0.05]]),
  ('metric p-value sample 43',
   [[0.001, 0.03, 0.03, 0.0167, 0.025], 0.05],
   [[True, False, False, False, False], [0.005, 0.075, 0.075, 0.0668, 0.075]])],
 [('boundary p equals alpha over remaining',
   [[0.0125, 0.5, 0.9, 0.9], 0.05],
   [[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]),
  ('adjusted values are monotone', [[0.01, 0.011, 0.5], 0.05], [[True, True, False], [0.03, 0.03, 0.5]]),
  ('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
  ('original order is preserved', [[0.3, 0.001, 0.02], 0.05], [[False, True, True], [0.3, 0.003, 0.04]]),
  ('holm rejects more than bonferroni', [[0.01, 0.02], 0.05], [[True, True], [0.02, 0.02]]),
  ('single metric', [[0.05], 0.05], [[True], [0.05]]),
  ('metric p-value sample 11', [[0.01, 0.001, 0.02], 0.05], [[True, True, True], [0.02, 0.003, 0.02]]),
  ('metric p-value sample 20', [[0.03, 0.001, 0.04], 0.05], [[False, True, False], [0.06, 0.003, 0.06]])],
 [('adjusted values are monotone', [[0.01, 0.011, 0.5], 0.05], [[True, True, False], [0.03, 0.03, 0.5]]),
  ('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
  ('original order is preserved', [[0.3, 0.001, 0.02], 0.05], [[False, True, True], [0.3, 0.003, 0.04]]),
  ('holm rejects more than bonferroni', [[0.01, 0.02], 0.05], [[True, True], [0.02, 0.02]]),
  ('single metric', [[0.05], 0.05], [[True], [0.05]]),
  ('metric p-value sample 16',
   [[0.03, 0.02, 0.2, 0.6, 0.0167], 0.05],
   [[False, False, False, False, False], [0.09, 0.0835, 0.4, 0.6, 0.0835]]),
  ('metric p-value sample 35', [[0.025, 0.03], 0.05], [[True, True], [0.05, 0.05]]),
  ('metric p-value sample 59',
   [[0.03, 0.02, 0.0125, 0.04, 0.03], 0.05],
   [[False, False, False, False, False], [0.09, 0.08, 0.0625, 0.09, 0.09]])],
 [('all three pass their step thresholds',
   [[0.01, 0.04, 0.012], 0.05],
   [[True, True, True], [0.03, 0.04, 0.03]]),
  ('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
  ('original order is preserved', [[0.3, 0.001, 0.02], 0.05], [[False, True, True], [0.3, 0.003, 0.04]]),
  ('holm rejects more than bonferroni', [[0.01, 0.02], 0.05], [[True, True], [0.02, 0.02]]),
  ('single metric', [[0.05], 0.05], [[True], [0.05]]),
  ('metric p-value sample 4', [[0.0125, 0.0167, 0.05], 0.05], [[True, True, True], [0.0375, 0.0375, 0.05]]),
  ('metric p-value sample 21',
   [[0.0167, 0.6, 0.2, 0.02, 0.01], 0.05],
   [[False, False, False, False, True], [0.0668, 0.6, 0.4, 0.0668, 0.05]]),
  ('metric p-value sample 34',
   [[0.01, 0.0167, 0.001, 0.05, 0.04], 0.05],
   [[True, False, True, False, False], [0.04, 0.0501, 0.005, 0.08, 0.08]])]]
for label, args, expected in fixtures[N - 1]:
    check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
all three pass their step thresholds	`[[True, True, True], [0.03, 0.04, 0.03]]`	`[[True, True, True], [0.03, 0.04, 0.03]]`	Passed
step-down stops at first non-rejection	`[[False, False, False], [0.06, 0.06, 0.06]]`	`[[False, False, False], [0.06, 0.06, 0.06]]`	Passed
boundary p equals alpha over remaining	`[[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]`	`[[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]`	Passed
adjusted values are monotone	`[[True, True, False], [0.03, 0.03, 0.5]]`	`[[True, True, False], [0.03, 0.03, 0.5]]`	Passed
adjusted values cap at one	`[[False, False], [0.8, 0.8]]`	`[[False, False], [0.8, 0.8]]`	Passed
single metric	`[[True], [0.05]]`	`[[True], [0.05]]`	Passed
metric p-value sample 1	`[[False, False, False, False], [0.08, 0.08, 0.1, 0.6]]`	`[[False, False, False, False], [0.08, 0.08, 0.1, 0.6]]`	Passed
metric p-value sample 2	`[[True, True, True], [0.0375, 0.0375, 0.0375]]`	`[[True, True, True], [0.0375, 0.0375, 0.0375]]`	Passed

SHA-256 / 38cfad1862300e7c86dbe559dd6d468046ae36c2c17662990af516585176f187

Verification & scope

A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:57.189582+00:00.

Case digest / daf726f900378ff67086d18232ff6514d9210bb10e82e577efb2a14cf96e162e