FA-98370 / Token sampling filters / Member archive

Token sampling: Typical sampling ranks by raw surprisal distance · case 05

Typical sampling ranks by raw surprisal distance.

Member previewVariant 5 · 3 implementations · 6 checks per implementation

Case contract

Compute entropy H=-sum(p log p), rank by abs(-log(p)-H) then token ID, and return the shortest nonempty prefix whose probability mass reaches mass; empty candidates return empty.

Why this case matters

A finite, deterministic token-selection contract isolates one decoding operation from model inference and runtime behavior.

One recorded failure

Sample boundary fixture

This sample comes from the broken implementation of a controlled reproducer.

Boundary fixture	Actual	Expected	Outcome
entropy-centered selection	`["a", "b"]`	`["b", "a"]`	Failed

MEMBER ARCHIVE

The complete case is available to members.

This record includes three runnable implementations, regression fixtures, execution results, and source hashes.

Member access is invitation-based. Sign in with your invited account to inspect the sources.