FA-74490 / Experiment statistics / Member archive

Holm correction across metrics: Adjusted p-values are not made monotone · case 05

A metric with a smaller raw p-value reports a larger adjusted p-value than a later one.

Member previewVariant 5 · 3 implementations · 8 checks per implementation

Case contract

Holm step-down: sort p-values ascending (ties by position); reject while p_(k) <= alpha / (m - k) for k = 0, 1, ... and stop at the first failure. Adjusted p_(k) = running max of min(1, (m - k) p_(k)). Return [rejections, adjusted p-values rounded to 6] in the original metric order.

Why this case matters

Experiments track many metrics; multiplicity control keeps secondary wins honest.

One recorded failure

Sample boundary fixture

This sample comes from the broken implementation of a controlled reproducer.

Boundary fixture	Actual	Expected	Outcome
all three pass their step thresholds	`[[true, true, true], [0.03, 0.04, 0.024]]`	`[[true, true, true], [0.03, 0.04, 0.03]]`	Failed

MEMBER ARCHIVE

The complete case is available to members.

This record includes three runnable implementations, regression fixtures, execution results, and source hashes.

Member access is invitation-based. Sign in with your invited account to inspect the sources.