{"abstract":"Tail-free pruning measures drops from the unsorted vocabulary.","category":"Token sampling filters","checks":6,"contract":"Sort positive probabilities descending with lexical ties. Normalize adjacent first differences by their sum and keep the shortest prefix through the first normalized second-difference ratio exceeding zeta; always keep the first token and keep all if no ratio exceeds it.","contract_signature":"probabilities, zeta","evaluation_group":"w3-tsf-tail-free-curvature","failed_approach":"The attempted repair cuts at the first-difference index instead of retaining through the corresponding ratio boundary.","family":"w3-token-sampling-filters-tail-free-curvature","id":"FA-98431","implementations":{"attempt":{"sha256":"55bb853135789171be11a806c8760a6c3565e1c16c066a09500ab884f1febdcb","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\nimport math\nN = 1\nobservations = []\ndef solve(probabilities, zeta):\n    ranked=sorted((t for t,p in probabilities.items() if p>0),key=lambda t:(-probabilities[t],t))\n    diff=[probabilities[ranked[i]]-probabilities[ranked[i+1]] for i in range(len(ranked)-1)]\n    total=sum(diff) or 1\n    ratios=[abs(diff[i]-diff[i-1])/total for i in range(1,len(diff))]\n    cut=next((i for i,r in enumerate(ratios) if r>zeta),len(ranked))\n    return ranked[:max(1,cut)]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\ncheck('large tail drop',solve({'a':.6,'b':.3,'c':.08,'d':.02},.2),['a','b'])\ncheck('flat differences',solve({'a':.4,'b':.3,'c':.2,'d':.1},.01),['a','b','c','d'])\ncheck('singleton',solve({'x':1.0},.1),['x'])\ncheck('empty',solve({},.1),[])\ncheck('zero token omitted',solve({'a':1.0,'z':0.0},.1),['a'])\ncheck('tie order',solve({'b':.5,'a':.5,'c':0},.1),['a','b'])\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"},"broken":{"sha256":"074e57b5cd0653b167818d346a2c9aec6163fe36892526adf86583af96a9d265","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\nimport math\nN = 1\nobservations = []\ndef solve(probabilities, zeta):\n    import math\n    ranked=list(probabilities)\n    diff=[probabilities[ranked[i]]-probabilities[ranked[i+1]] for i in range(len(ranked)-1)]\n    total=sum(diff) or 1\n    ratios=[abs(diff[i]-diff[i-1])/total for i in range(1,len(diff))]\n    cut=next((i+1 for i,r in enumerate(ratios) if r>zeta),len(ranked))\n    return ranked[:cut]\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\ncheck('large tail drop',solve({'a':.6,'b':.3,'c':.08,'d':.02},.2),['a','b'])\ncheck('flat differences',solve({'a':.4,'b':.3,'c':.2,'d':.1},.01),['a','b','c','d'])\ncheck('singleton',solve({'x':1.0},.1),['x'])\ncheck('empty',solve({},.1),[])\ncheck('zero token omitted',solve({'a':1.0,'z':0.0},.1),['a'])\ncheck('tie order',solve({'b':.5,'a':.5,'c':0},.1),['a','b'])\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"}},"limitations":"These are toy finite-vocabulary contracts with supplied scores, histories, and draws; they do not claim production model quality or compatibility. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.","method":"Deterministic executable model with adversarial boundary fixtures.","provenance":{"created_by":"Failure Map","dependencies":"Python standard library","family":"w3-token-sampling-filters-tail-free-curvature","generated_at":"2026-09-30T16:48:12.803749+00:00","license":"CC0-1.0","python":"3.12.14","seed":1,"split":"open-access"},"relevance":"A finite, deterministic token-selection contract isolates one decoding operation from model inference and runtime behavior.","root_cause":"The filter depends on rank adjacent probability drops, so arbitrary vocabulary order yields unrelated differences.","sha256":"c885f4604cf11c7ef3176825094410ed5ed2455f50fe2b2697fcc816b9588934","title":"Token sampling: Tail-free pruning measures drops from the unsorted vocabulary · case 01","variant":1,"variant_policy":"The same explicitly authored fixtures repeat across all five numbered records; runtime N is unused. These records share one mechanism and must remain together in evaluation splits.","verified":true,"visibility":"public","verification":{"attempt":{"elapsed_ms":26.476,"exit_code":1,"observations":[{"actual":["a"],"check":"large tail drop","expected":["a","b"],"passed":false},{"actual":["a","b","c","d"],"check":"flat differences","expected":["a","b","c","d"],"passed":true},{"actual":["x"],"check":"singleton","expected":["x"],"passed":true},{"actual":[],"check":"empty","expected":[],"passed":true},{"actual":["a"],"check":"zero token omitted","expected":["a"],"passed":true},{"actual":["a","b"],"check":"tie order","expected":["a","b"],"passed":true}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"large tail drop\", \"actual\": [\"a\"], \"expected\": [\"a\", \"b\"], \"passed\": false}, {\"check\": \"flat differences\", \"actual\": [\"a\", \"b\", \"c\", \"d\"], \"expected\": [\"a\", \"b\", \"c\", \"d\"], \"passed\": true}, {\"check\": \"singleton\", \"actual\": [\"x\"], \"expected\": [\"x\"], \"passed\": true}, {\"check\": \"empty\", \"actual\": [], \"expected\": [], \"passed\": true}, {\"check\": \"zero token omitted\", \"actual\": [\"a\"], \"expected\": [\"a\"], \"passed\": true}, {\"check\": \"tie order\", \"actual\": [\"a\", \"b\"], \"expected\": [\"a\", \"b\"], \"passed\": true}], \"passed\": false}\n"},"broken":{"elapsed_ms":26.708,"exit_code":1,"observations":[{"actual":["a","b"],"check":"large tail drop","expected":["a","b"],"passed":true},{"actual":["a","b","c","d"],"check":"flat differences","expected":["a","b","c","d"],"passed":true},{"actual":["x"],"check":"singleton","expected":["x"],"passed":true},{"actual":[],"check":"empty","expected":[],"passed":true},{"actual":["a","z"],"check":"zero token omitted","expected":["a"],"passed":false},{"actual":["b"],"check":"tie order","expected":["a","b"],"passed":false}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"large tail drop\", \"actual\": [\"a\", \"b\"], \"expected\": [\"a\", \"b\"], \"passed\": true}, {\"check\": \"flat differences\", \"actual\": [\"a\", \"b\", \"c\", \"d\"], \"expected\": [\"a\", \"b\", \"c\", \"d\"], \"passed\": true}, {\"check\": \"singleton\", \"actual\": [\"x\"], \"expected\": [\"x\"], \"passed\": true}, {\"check\": \"empty\", \"actual\": [], \"expected\": [], \"passed\": true}, {\"check\": \"zero token omitted\", \"actual\": [\"a\", \"z\"], \"expected\": [\"a\"], \"passed\": false}, {\"check\": \"tie order\", \"actual\": [\"b\"], \"expected\": [\"a\", \"b\"], \"passed\": false}], \"passed\": false}\n"}},"member_only":{"stages":["fixed"],"fields":["implementations.fixed","verification.fixed","harness","repair"],"note":"The verified repair, its recorded checks, the repair description, and the scoring harness are available to members."}}