FA-74488 / Experiment statistics / Member archive

Holm correction across metrics: Adjusted p-values are not made monotone · case 03

A metric with a smaller raw p-value reports a larger adjusted p-value than a later one.

Member previewVariant 3 · 3 implementations · 8 checks per implementation

Case contract

Holm step-down: sort p-values ascending (ties by position); reject while p_(k) <= alpha / (m - k) for k = 0, 1, ... and stop at the first failure. Adjusted p_(k) = running max of min(1, (m - k) p_(k)). Return [rejections, adjusted p-values rounded to 6] in the original metric order.

Why this case matters

Experiments track many metrics; multiplicity control keeps secondary wins honest.

One recorded failure

Sample boundary fixture

This sample comes from the broken implementation of a controlled reproducer.

Boundary fixture	Actual	Expected	Outcome
boundary p equals alpha over remaining	`[[true, false, false, false], [0.05, 1.0, 1.0, 0.9]]`	`[[true, false, false, false], [0.05, 1.0, 1.0, 1.0]]`	Failed

MEMBER ARCHIVE

The complete case is available to members.

This record includes three runnable implementations, regression fixtures, execution results, and source hashes.

Member access is invitation-based. Sign in with your invited account to inspect the sources.