FA-74478 / Experiment statistics / Member archive

Holm correction across metrics: Rejection continues after a failed step · case 03

A large p-value followed by a lucky threshold rejects metrics Holm would keep.

Member previewVariant 3 · 3 implementations · 8 checks per implementation

Case contract

Holm step-down: sort p-values ascending (ties by position); reject while p_(k) <= alpha / (m - k) for k = 0, 1, ... and stop at the first failure. Adjusted p_(k) = running max of min(1, (m - k) p_(k)). Return [rejections, adjusted p-values rounded to 6] in the original metric order.

Why this case matters

Experiments track many metrics; multiplicity control keeps secondary wins honest.

One recorded failure

Sample boundary fixture

This sample comes from the broken implementation of a controlled reproducer.

Boundary fixture	Actual	Expected	Outcome
metric p-value sample 20	`[[false, true, true], [0.06, 0.003, 0.06]]`	`[[false, true, false], [0.06, 0.003, 0.06]]`	Failed

MEMBER ARCHIVE

The complete case is available to members.

This record includes three runnable implementations, regression fixtures, execution results, and source hashes.

Member access is invitation-based. Sign in with your invited account to inspect the sources.