FA-98434 / Token sampling filters / Member archive

Token sampling: Tail-free pruning measures drops from the unsorted vocabulary · case 04

Tail-free pruning measures drops from the unsorted vocabulary.

Member previewVariant 4 · 3 implementations · 6 checks per implementation

Case contract

Sort positive probabilities descending with lexical ties. Normalize adjacent first differences by their sum and keep the shortest prefix through the first normalized second-difference ratio exceeding zeta; always keep the first token and keep all if no ratio exceeds it.

Why this case matters

A finite, deterministic token-selection contract isolates one decoding operation from model inference and runtime behavior.

One recorded failure

Sample boundary fixture

This sample comes from the broken implementation of a controlled reproducer.

Boundary fixture	Actual	Expected	Outcome
zero token omitted	`["a", "z"]`	`["a"]`	Failed

MEMBER ARCHIVE

The complete case is available to members.

This record includes three runnable implementations, regression fixtures, execution results, and source hashes.

Member access is invitation-based. Sign in with your invited account to inspect the sources.