FA-74789 / Experiment statistics / Member archive

Always-valid sequential p-value: The p-value can rise again · case 04

A result that was significant yesterday becomes insignificant today, breaking always-valid guarantees.

Member previewVariant 4 · 3 implementations · 8 checks per implementation

Case contract

diffs is a stream of paired differences with known variance sigma2; the normal-mixture likelihood ratio after n observations with running mean m is sqrt(sigma2 / (sigma2 + n tau2)) exp(n^2 tau2 m^2 / (2 sigma2 (sigma2 + n tau2))). The always-valid p-value starts at 1 and is the running minimum of 1 / ratio. Return the p-value after each observation rounded to 6.

Why this case matters

Always-valid p-values let teams monitor continuously without inflating false positives.

One recorded failure

Sample boundary fixture

This sample comes from the broken implementation of a controlled reproducer.

Boundary fixture	Actual	Expected	Outcome
p-value never increases after a reversal	`[0.52026, 0.120349, 0.649305, 1.0]`	`[0.52026, 0.120349, 0.120349, 0.120349]`	Failed

MEMBER ARCHIVE

The complete case is available to members.

This record includes three runnable implementations, regression fixtures, execution results, and source hashes.

Member access is invitation-based. Sign in with your invited account to inspect the sources.