FA-74486 / Experiment statistics / Open access
Holm correction across metrics: Adjusted p-values are not made monotone · case 01
A metric with a smaller raw p-value reports a larger adjusted p-value than a later one.
ROOT CAUSE
Each adjusted value is min(1, (m - k) p) without the running maximum.
VERIFIED REPAIR
Carry the running maximum over the sorted adjusted values.
Unsuccessful approach: Keeping the running maximum but dropping the cap reports adjusted values above one.
Case contract
Holm step-down: sort p-values ascending (ties by position); reject while p_(k) <= alpha / (m - k) for k = 0, 1, ... and stop at the first failure. Adjusted p_(k) = running max of min(1, (m - k) p_(k)). Return [rejections, adjusted p-values rounded to 6] in the original metric order.
Why this case matters
Experiments track many metrics; multiplicity control keeps secondary wins honest.
1 / The failure
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(pvalues, alpha):
m = len(pvalues)
order = sorted(range(m), key=lambda i: (pvalues[i], i))
reject = [False] * m
adjusted = [0.0] * m
running = 0.0
stopped = False
for rank, i in enumerate(order):
running = min(1.0, (m - rank) * pvalues[i])
adjusted[i] = round(running, 6)
if not stopped and pvalues[i] <= alpha / (m - rank):
reject[i] = True
else:
stopped = True
return [reject, adjusted]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('all three pass their step thresholds',
[[0.01, 0.04, 0.012], 0.05],
[[True, True, True], [0.03, 0.04, 0.03]]),
('step-down stops at first non-rejection',
[[0.03, 0.02, 0.04], 0.05],
[[False, False, False], [0.06, 0.06, 0.06]]),
('boundary p equals alpha over remaining',
[[0.0125, 0.5, 0.9, 0.9], 0.05],
[[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]),
('adjusted values are monotone', [[0.01, 0.011, 0.5], 0.05], [[True, True, False], [0.03, 0.03, 0.5]]),
('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
('metric p-value sample 1',
[[0.025, 0.02, 0.05, 0.6], 0.05],
[[False, False, False, False], [0.08, 0.08, 0.1, 0.6]]),
('metric p-value sample 2', [[0.0125, 0.03, 0.0167], 0.05], [[True, True, True], [0.0375, 0.0375, 0.0375]]),
('metric p-value sample 3', [[0.025], 0.05], [[True], [0.025]])],
[('step-down stops at first non-rejection',
[[0.03, 0.02, 0.04], 0.05],
[[False, False, False], [0.06, 0.06, 0.06]]),
('boundary p equals alpha over remaining',
[[0.0125, 0.5, 0.9, 0.9], 0.05],
[[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]),
('adjusted values are monotone', [[0.01, 0.011, 0.5], 0.05], [[True, True, False], [0.03, 0.03, 0.5]]),
('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
('original order is preserved', [[0.3, 0.001, 0.02], 0.05], [[False, True, True], [0.3, 0.003, 0.04]]),
('metric p-value sample 4', [[0.0125, 0.0167, 0.05], 0.05], [[True, True, True], [0.0375, 0.0375, 0.05]]),
('metric p-value sample 6', [[0.01, 0.05], 0.05], [[True, True], [0.02, 0.05]]),
('metric p-value sample 7', [[0.02, 0.0167], 0.05], [[True, True], [0.0334, 0.0334]])],
[('boundary p equals alpha over remaining',
[[0.0125, 0.5, 0.9, 0.9], 0.05],
[[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]),
('adjusted values are monotone', [[0.01, 0.011, 0.5], 0.05], [[True, True, False], [0.03, 0.03, 0.5]]),
('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
('original order is preserved', [[0.3, 0.001, 0.02], 0.05], [[False, True, True], [0.3, 0.003, 0.04]]),
('holm rejects more than bonferroni', [[0.01, 0.02], 0.05], [[True, True], [0.02, 0.02]]),
('metric p-value sample 11', [[0.01, 0.001, 0.02], 0.05], [[True, True, True], [0.02, 0.003, 0.02]]),
('metric p-value sample 12',
[[0.04, 0.025, 0.01, 0.0167, 0.0125], 0.05],
[[False, False, True, False, True], [0.0501, 0.0501, 0.05, 0.0501, 0.05]]),
('metric p-value sample 16',
[[0.03, 0.02, 0.2, 0.6, 0.0167], 0.05],
[[False, False, False, False, False], [0.09, 0.0835, 0.4, 0.6, 0.0835]])],
[('boundary p equals alpha over remaining',
[[0.0125, 0.5, 0.9, 0.9], 0.05],
[[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]),
('adjusted values are monotone', [[0.01, 0.011, 0.5], 0.05], [[True, True, False], [0.03, 0.03, 0.5]]),
('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
('original order is preserved', [[0.3, 0.001, 0.02], 0.05], [[False, True, True], [0.3, 0.003, 0.04]]),
('holm rejects more than bonferroni', [[0.01, 0.02], 0.05], [[True, True], [0.02, 0.02]]),
('single metric', [[0.05], 0.05], [[True], [0.05]]),
('metric p-value sample 16',
[[0.03, 0.02, 0.2, 0.6, 0.0167], 0.05],
[[False, False, False, False, False], [0.09, 0.0835, 0.4, 0.6, 0.0835]]),
('metric p-value sample 28', [[0.01, 0.05, 0.03], 0.05], [[True, False, False], [0.03, 0.06, 0.06]])],
[('all three pass their step thresholds',
[[0.01, 0.04, 0.012], 0.05],
[[True, True, True], [0.03, 0.04, 0.03]]),
('boundary p equals alpha over remaining',
[[0.0125, 0.5, 0.9, 0.9], 0.05],
[[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]),
('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
('original order is preserved', [[0.3, 0.001, 0.02], 0.05], [[False, True, True], [0.3, 0.003, 0.04]]),
('holm rejects more than bonferroni', [[0.01, 0.02], 0.05], [[True, True], [0.02, 0.02]]),
('single metric', [[0.05], 0.05], [[True], [0.05]]),
('metric p-value sample 21',
[[0.0167, 0.6, 0.2, 0.02, 0.01], 0.05],
[[False, False, False, False, True], [0.0668, 0.6, 0.4, 0.0668, 0.05]]),
('metric p-value sample 48',
[[0.02, 0.2, 0.0167, 0.2, 0.025], 0.05],
[[False, False, False, False, False], [0.0835, 0.4, 0.0835, 0.4, 0.0835]])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| all three pass their step thresholds | [[True, True, True], [0.03, 0.04, 0.024]] | [[True, True, True], [0.03, 0.04, 0.03]] | Failed |
| step-down stops at first non-rejection | [[False, False, False], [0.06, 0.06, 0.04]] | [[False, False, False], [0.06, 0.06, 0.06]] | Failed |
| boundary p equals alpha over remaining | [[True, False, False, False], [0.05, 1.0, 1.0, 0.9]] | [[True, False, False, False], [0.05, 1.0, 1.0, 1.0]] | Failed |
| adjusted values are monotone | [[True, True, False], [0.03, 0.022, 0.5]] | [[True, True, False], [0.03, 0.03, 0.5]] | Failed |
| adjusted values cap at one | [[False, False], [0.8, 0.6]] | [[False, False], [0.8, 0.8]] | Failed |
| metric p-value sample 1 | [[False, False, False, False], [0.075, 0.08, 0.1, 0.6]] | [[False, False, False, False], [0.08, 0.08, 0.1, 0.6]] | Failed |
| metric p-value sample 2 | [[True, True, True], [0.0375, 0.03, 0.0334]] | [[True, True, True], [0.0375, 0.0375, 0.0375]] | Failed |
| metric p-value sample 3 | [[True], [0.025]] | [[True], [0.025]] | Passed |
SHA-256 / 2286904f5db44c6f7fc7abfa5b6620a81aef742c8d38b756f23e3f98915ddc9d
2 / The unsuccessful fix
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(pvalues, alpha):
m = len(pvalues)
order = sorted(range(m), key=lambda i: (pvalues[i], i))
reject = [False] * m
adjusted = [0.0] * m
running = 0.0
stopped = False
for rank, i in enumerate(order):
running = max(running, (m - rank) * pvalues[i])
adjusted[i] = round(running, 6)
if not stopped and pvalues[i] <= alpha / (m - rank):
reject[i] = True
else:
stopped = True
return [reject, adjusted]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('all three pass their step thresholds',
[[0.01, 0.04, 0.012], 0.05],
[[True, True, True], [0.03, 0.04, 0.03]]),
('step-down stops at first non-rejection',
[[0.03, 0.02, 0.04], 0.05],
[[False, False, False], [0.06, 0.06, 0.06]]),
('boundary p equals alpha over remaining',
[[0.0125, 0.5, 0.9, 0.9], 0.05],
[[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]),
('adjusted values are monotone', [[0.01, 0.011, 0.5], 0.05], [[True, True, False], [0.03, 0.03, 0.5]]),
('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
('metric p-value sample 1',
[[0.025, 0.02, 0.05, 0.6], 0.05],
[[False, False, False, False], [0.08, 0.08, 0.1, 0.6]]),
('metric p-value sample 2', [[0.0125, 0.03, 0.0167], 0.05], [[True, True, True], [0.0375, 0.0375, 0.0375]]),
('metric p-value sample 3', [[0.025], 0.05], [[True], [0.025]])],
[('step-down stops at first non-rejection',
[[0.03, 0.02, 0.04], 0.05],
[[False, False, False], [0.06, 0.06, 0.06]]),
('boundary p equals alpha over remaining',
[[0.0125, 0.5, 0.9, 0.9], 0.05],
[[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]),
('adjusted values are monotone', [[0.01, 0.011, 0.5], 0.05], [[True, True, False], [0.03, 0.03, 0.5]]),
('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
('original order is preserved', [[0.3, 0.001, 0.02], 0.05], [[False, True, True], [0.3, 0.003, 0.04]]),
('metric p-value sample 4', [[0.0125, 0.0167, 0.05], 0.05], [[True, True, True], [0.0375, 0.0375, 0.05]]),
('metric p-value sample 6', [[0.01, 0.05], 0.05], [[True, True], [0.02, 0.05]]),
('metric p-value sample 7', [[0.02, 0.0167], 0.05], [[True, True], [0.0334, 0.0334]])],
[('boundary p equals alpha over remaining',
[[0.0125, 0.5, 0.9, 0.9], 0.05],
[[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]),
('adjusted values are monotone', [[0.01, 0.011, 0.5], 0.05], [[True, True, False], [0.03, 0.03, 0.5]]),
('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
('original order is preserved', [[0.3, 0.001, 0.02], 0.05], [[False, True, True], [0.3, 0.003, 0.04]]),
('holm rejects more than bonferroni', [[0.01, 0.02], 0.05], [[True, True], [0.02, 0.02]]),
('metric p-value sample 11', [[0.01, 0.001, 0.02], 0.05], [[True, True, True], [0.02, 0.003, 0.02]]),
('metric p-value sample 12',
[[0.04, 0.025, 0.01, 0.0167, 0.0125], 0.05],
[[False, False, True, False, True], [0.0501, 0.0501, 0.05, 0.0501, 0.05]]),
('metric p-value sample 16',
[[0.03, 0.02, 0.2, 0.6, 0.0167], 0.05],
[[False, False, False, False, False], [0.09, 0.0835, 0.4, 0.6, 0.0835]])],
[('boundary p equals alpha over remaining',
[[0.0125, 0.5, 0.9, 0.9], 0.05],
[[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]),
('adjusted values are monotone', [[0.01, 0.011, 0.5], 0.05], [[True, True, False], [0.03, 0.03, 0.5]]),
('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
('original order is preserved', [[0.3, 0.001, 0.02], 0.05], [[False, True, True], [0.3, 0.003, 0.04]]),
('holm rejects more than bonferroni', [[0.01, 0.02], 0.05], [[True, True], [0.02, 0.02]]),
('single metric', [[0.05], 0.05], [[True], [0.05]]),
('metric p-value sample 16',
[[0.03, 0.02, 0.2, 0.6, 0.0167], 0.05],
[[False, False, False, False, False], [0.09, 0.0835, 0.4, 0.6, 0.0835]]),
('metric p-value sample 28', [[0.01, 0.05, 0.03], 0.05], [[True, False, False], [0.03, 0.06, 0.06]])],
[('all three pass their step thresholds',
[[0.01, 0.04, 0.012], 0.05],
[[True, True, True], [0.03, 0.04, 0.03]]),
('boundary p equals alpha over remaining',
[[0.0125, 0.5, 0.9, 0.9], 0.05],
[[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]),
('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
('original order is preserved', [[0.3, 0.001, 0.02], 0.05], [[False, True, True], [0.3, 0.003, 0.04]]),
('holm rejects more than bonferroni', [[0.01, 0.02], 0.05], [[True, True], [0.02, 0.02]]),
('single metric', [[0.05], 0.05], [[True], [0.05]]),
('metric p-value sample 21',
[[0.0167, 0.6, 0.2, 0.02, 0.01], 0.05],
[[False, False, False, False, True], [0.0668, 0.6, 0.4, 0.0668, 0.05]]),
('metric p-value sample 48',
[[0.02, 0.2, 0.0167, 0.2, 0.025], 0.05],
[[False, False, False, False, False], [0.0835, 0.4, 0.0835, 0.4, 0.0835]])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| all three pass their step thresholds | [[True, True, True], [0.03, 0.04, 0.03]] | [[True, True, True], [0.03, 0.04, 0.03]] | Passed |
| step-down stops at first non-rejection | [[False, False, False], [0.06, 0.06, 0.06]] | [[False, False, False], [0.06, 0.06, 0.06]] | Passed |
| boundary p equals alpha over remaining | [[True, False, False, False], [0.05, 1.5, 1.8, 1.8]] | [[True, False, False, False], [0.05, 1.0, 1.0, 1.0]] | Failed |
| adjusted values are monotone | [[True, True, False], [0.03, 0.03, 0.5]] | [[True, True, False], [0.03, 0.03, 0.5]] | Passed |
| adjusted values cap at one | [[False, False], [0.8, 0.8]] | [[False, False], [0.8, 0.8]] | Passed |
| metric p-value sample 1 | [[False, False, False, False], [0.08, 0.08, 0.1, 0.6]] | [[False, False, False, False], [0.08, 0.08, 0.1, 0.6]] | Passed |
| metric p-value sample 2 | [[True, True, True], [0.0375, 0.0375, 0.0375]] | [[True, True, True], [0.0375, 0.0375, 0.0375]] | Passed |
| metric p-value sample 3 | [[True], [0.025]] | [[True], [0.025]] | Passed |
SHA-256 / d255f5200ea7ccfaaf53ddc06f2ff8383ce3a8d604eddfe5821b808c0e97bd60
3 / The verified repair
Exit 0"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(pvalues, alpha):
m = len(pvalues)
order = sorted(range(m), key=lambda i: (pvalues[i], i))
reject = [False] * m
adjusted = [0.0] * m
running = 0.0
stopped = False
for rank, i in enumerate(order):
running = max(running, min(1.0, (m - rank) * pvalues[i]))
adjusted[i] = round(running, 6)
if not stopped and pvalues[i] <= alpha / (m - rank):
reject[i] = True
else:
stopped = True
return [reject, adjusted]
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('all three pass their step thresholds',
[[0.01, 0.04, 0.012], 0.05],
[[True, True, True], [0.03, 0.04, 0.03]]),
('step-down stops at first non-rejection',
[[0.03, 0.02, 0.04], 0.05],
[[False, False, False], [0.06, 0.06, 0.06]]),
('boundary p equals alpha over remaining',
[[0.0125, 0.5, 0.9, 0.9], 0.05],
[[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]),
('adjusted values are monotone', [[0.01, 0.011, 0.5], 0.05], [[True, True, False], [0.03, 0.03, 0.5]]),
('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
('metric p-value sample 1',
[[0.025, 0.02, 0.05, 0.6], 0.05],
[[False, False, False, False], [0.08, 0.08, 0.1, 0.6]]),
('metric p-value sample 2', [[0.0125, 0.03, 0.0167], 0.05], [[True, True, True], [0.0375, 0.0375, 0.0375]]),
('metric p-value sample 3', [[0.025], 0.05], [[True], [0.025]])],
[('step-down stops at first non-rejection',
[[0.03, 0.02, 0.04], 0.05],
[[False, False, False], [0.06, 0.06, 0.06]]),
('boundary p equals alpha over remaining',
[[0.0125, 0.5, 0.9, 0.9], 0.05],
[[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]),
('adjusted values are monotone', [[0.01, 0.011, 0.5], 0.05], [[True, True, False], [0.03, 0.03, 0.5]]),
('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
('original order is preserved', [[0.3, 0.001, 0.02], 0.05], [[False, True, True], [0.3, 0.003, 0.04]]),
('metric p-value sample 4', [[0.0125, 0.0167, 0.05], 0.05], [[True, True, True], [0.0375, 0.0375, 0.05]]),
('metric p-value sample 6', [[0.01, 0.05], 0.05], [[True, True], [0.02, 0.05]]),
('metric p-value sample 7', [[0.02, 0.0167], 0.05], [[True, True], [0.0334, 0.0334]])],
[('boundary p equals alpha over remaining',
[[0.0125, 0.5, 0.9, 0.9], 0.05],
[[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]),
('adjusted values are monotone', [[0.01, 0.011, 0.5], 0.05], [[True, True, False], [0.03, 0.03, 0.5]]),
('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
('original order is preserved', [[0.3, 0.001, 0.02], 0.05], [[False, True, True], [0.3, 0.003, 0.04]]),
('holm rejects more than bonferroni', [[0.01, 0.02], 0.05], [[True, True], [0.02, 0.02]]),
('metric p-value sample 11', [[0.01, 0.001, 0.02], 0.05], [[True, True, True], [0.02, 0.003, 0.02]]),
('metric p-value sample 12',
[[0.04, 0.025, 0.01, 0.0167, 0.0125], 0.05],
[[False, False, True, False, True], [0.0501, 0.0501, 0.05, 0.0501, 0.05]]),
('metric p-value sample 16',
[[0.03, 0.02, 0.2, 0.6, 0.0167], 0.05],
[[False, False, False, False, False], [0.09, 0.0835, 0.4, 0.6, 0.0835]])],
[('boundary p equals alpha over remaining',
[[0.0125, 0.5, 0.9, 0.9], 0.05],
[[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]),
('adjusted values are monotone', [[0.01, 0.011, 0.5], 0.05], [[True, True, False], [0.03, 0.03, 0.5]]),
('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
('original order is preserved', [[0.3, 0.001, 0.02], 0.05], [[False, True, True], [0.3, 0.003, 0.04]]),
('holm rejects more than bonferroni', [[0.01, 0.02], 0.05], [[True, True], [0.02, 0.02]]),
('single metric', [[0.05], 0.05], [[True], [0.05]]),
('metric p-value sample 16',
[[0.03, 0.02, 0.2, 0.6, 0.0167], 0.05],
[[False, False, False, False, False], [0.09, 0.0835, 0.4, 0.6, 0.0835]]),
('metric p-value sample 28', [[0.01, 0.05, 0.03], 0.05], [[True, False, False], [0.03, 0.06, 0.06]])],
[('all three pass their step thresholds',
[[0.01, 0.04, 0.012], 0.05],
[[True, True, True], [0.03, 0.04, 0.03]]),
('boundary p equals alpha over remaining',
[[0.0125, 0.5, 0.9, 0.9], 0.05],
[[True, False, False, False], [0.05, 1.0, 1.0, 1.0]]),
('adjusted values cap at one', [[0.4, 0.6], 0.05], [[False, False], [0.8, 0.8]]),
('original order is preserved', [[0.3, 0.001, 0.02], 0.05], [[False, True, True], [0.3, 0.003, 0.04]]),
('holm rejects more than bonferroni', [[0.01, 0.02], 0.05], [[True, True], [0.02, 0.02]]),
('single metric', [[0.05], 0.05], [[True], [0.05]]),
('metric p-value sample 21',
[[0.0167, 0.6, 0.2, 0.02, 0.01], 0.05],
[[False, False, False, False, True], [0.0668, 0.6, 0.4, 0.0668, 0.05]]),
('metric p-value sample 48',
[[0.02, 0.2, 0.0167, 0.2, 0.025], 0.05],
[[False, False, False, False, False], [0.0835, 0.4, 0.0835, 0.4, 0.0835]])]]
for label, args, expected in fixtures[N - 1]:
check(label, solve(*args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| all three pass their step thresholds | [[True, True, True], [0.03, 0.04, 0.03]] | [[True, True, True], [0.03, 0.04, 0.03]] | Passed |
| step-down stops at first non-rejection | [[False, False, False], [0.06, 0.06, 0.06]] | [[False, False, False], [0.06, 0.06, 0.06]] | Passed |
| boundary p equals alpha over remaining | [[True, False, False, False], [0.05, 1.0, 1.0, 1.0]] | [[True, False, False, False], [0.05, 1.0, 1.0, 1.0]] | Passed |
| adjusted values are monotone | [[True, True, False], [0.03, 0.03, 0.5]] | [[True, True, False], [0.03, 0.03, 0.5]] | Passed |
| adjusted values cap at one | [[False, False], [0.8, 0.8]] | [[False, False], [0.8, 0.8]] | Passed |
| metric p-value sample 1 | [[False, False, False, False], [0.08, 0.08, 0.1, 0.6]] | [[False, False, False, False], [0.08, 0.08, 0.1, 0.6]] | Passed |
| metric p-value sample 2 | [[True, True, True], [0.0375, 0.0375, 0.0375]] | [[True, True, True], [0.0375, 0.0375, 0.0375]] | Passed |
| metric p-value sample 3 | [[True], [0.025]] | [[True], [0.025]] | Passed |
SHA-256 / f6c95938ea32dcd32f9de19630d1b6cea8218fe1ac376cf798b16c1e028e3fcd
Verification & scope
A deterministic toy experiment-analysis model with a stipulated contract; results are rounded and are not a substitute for a validated statistics package. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.
Observations recorded using Python 3.12.14 at 2026-09-29T14:48:57.270741+00:00.
Case digest / 643615471dbe490ebb1ed1fd84e0d630c6440263b3376e878dae4924ba4468c5