FA-74479 / Experiment statistics / Member archive

Holm correction across metrics: Rejection continues after a failed step · case 04

A large p-value followed by a lucky threshold rejects metrics Holm would keep.

Member previewVariant 4 · 3 implementations · 8 checks per implementation

Case contract

Holm step-down: sort p-values ascending (ties by position); reject while p_(k) <= alpha / (m - k) for k = 0, 1, ... and stop at the first failure. Adjusted p_(k) = running max of min(1, (m - k) p_(k)). Return [rejections, adjusted p-values rounded to 6] in the original metric order.

Why this case matters

Experiments track many metrics; multiplicity control keeps secondary wins honest.

One recorded failure

Sample boundary fixture

This sample comes from the broken implementation of a controlled reproducer.

Boundary fixture	Actual	Expected	Outcome
metric p-value sample 59	`[[false, false, false, true, false], [0.09, 0.08, 0.0625, 0.09, 0.09]]`	`[[false, false, false, false, false], [0.09, 0.08, 0.0625, 0.09, 0.09]]`	Failed

MEMBER ARCHIVE

The complete case is available to members.

This record includes three runnable implementations, regression fixtures, execution results, and source hashes.

Member access is invitation-based. Sign in with your invited account to inspect the sources.