{"abstract":"length penalty counts prompt tokens as generated output.","category":"Beam search bookkeeping","checks":6,"contract":"Rows [id,score,total_length]; normalize by max(1,total_length-prompt_length) and return scores by ID.","contract_signature":"rows, prompt_length","evaluation_group":"beam-generated_length","failed_approach":"Subtracting one token regardless of prompt size leaves the penalty distorted.","family":"w3-beam-search-bookkeeping-length-excludes-prompt","id":"FA-98481","implementations":{"attempt":{"sha256":"2834ace972badf0f4fc8b8ca4e9d322c263c79ad7a2909c2d81b55ac716fe37b","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(rows, prompt_length):\n    return {r[0]:r[1]/max(1,r[2]-1) for r in rows}\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\ncheck('shared prompt removed',solve([['a',N,N+2]],N),{'a':N/2})\ncheck('prompt only floor',solve([['a',N,N]],N),{'a':float(N)})\ncheck('two generated',solve([['b',2*N,N+2]],N),{'b':float(N)})\ncheck('empty input',solve([],N),{})\ncheck('zero prompt',solve([['c',N,2]],0),{'c':N/2})\ncheck('long prefix',solve([['d',2*N,3*N]],2*N),{'d':2.0})\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"},"broken":{"sha256":"f78ce5141068858b2603d384bbd1f535f8cb34cf759dea0015905b74b4076800","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(rows, prompt_length):\n    return {r[0]:r[1]/r[2] for r in rows}\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\ncheck('shared prompt removed',solve([['a',N,N+2]],N),{'a':N/2})\ncheck('prompt only floor',solve([['a',N,N]],N),{'a':float(N)})\ncheck('two generated',solve([['b',2*N,N+2]],N),{'b':float(N)})\ncheck('empty input',solve([],N),{})\ncheck('zero prompt',solve([['c',N,2]],0),{'c':N/2})\ncheck('long prefix',solve([['d',2*N,3*N]],2*N),{'d':2.0})\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"}},"limitations":"This is a finite toy beam-state contract and makes no claim about neural model quality or production decoding. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.","method":"Deterministic executable model with adversarial boundary fixtures.","provenance":{"created_by":"Failure Map","dependencies":"Python standard library","family":"w3-beam-search-bookkeeping-length-excludes-prompt","generated_at":"2026-09-30T16:48:13.668688+00:00","license":"CC0-1.0","python":"3.12.14","seed":1,"split":"open-access"},"relevance":"Beam decoders rely on consistent hypothesis state while expanding and ranking candidates.","root_cause":"The length field includes a fixed prompt prefix that is shared by all candidates.","sha256":"8f04ea2cca9cde5bb8c9d5448add0a9fb2c8ab14b9611f251b4ef3aecaf5fcd2","title":"Beam search: length penalty counts prompt tokens as generated output · case 01","variant":1,"variant_policy":"The fixture code references N, supplied as 1 through 5. Fixed boundary checks may repeat. These records share one mechanism and must remain together in evaluation splits.","verified":true,"visibility":"public","verification":{"attempt":{"elapsed_ms":28.721,"exit_code":1,"observations":[{"actual":{"a":0.5},"check":"shared prompt removed","expected":{"a":0.5},"passed":true},{"actual":{"a":1.0},"check":"prompt only floor","expected":{"a":1.0},"passed":true},{"actual":{"b":1.0},"check":"two generated","expected":{"b":1.0},"passed":true},{"actual":{},"check":"empty input","expected":{},"passed":true},{"actual":{"c":1.0},"check":"zero prompt","expected":{"c":0.5},"passed":false},{"actual":{"d":1.0},"check":"long prefix","expected":{"d":2.0},"passed":false}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"shared prompt removed\", \"actual\": {\"a\": 0.5}, \"expected\": {\"a\": 0.5}, \"passed\": true}, {\"check\": \"prompt only floor\", \"actual\": {\"a\": 1.0}, \"expected\": {\"a\": 1.0}, \"passed\": true}, {\"check\": \"two generated\", \"actual\": {\"b\": 1.0}, \"expected\": {\"b\": 1.0}, \"passed\": true}, {\"check\": \"empty input\", \"actual\": {}, \"expected\": {}, \"passed\": true}, {\"check\": \"zero prompt\", \"actual\": {\"c\": 1.0}, \"expected\": {\"c\": 0.5}, \"passed\": false}, {\"check\": \"long prefix\", \"actual\": {\"d\": 1.0}, \"expected\": {\"d\": 2.0}, \"passed\": false}], \"passed\": false}\n"},"broken":{"elapsed_ms":28.703,"exit_code":1,"observations":[{"actual":{"a":0.3333333333333333},"check":"shared prompt removed","expected":{"a":0.5},"passed":false},{"actual":{"a":1.0},"check":"prompt only floor","expected":{"a":1.0},"passed":true},{"actual":{"b":0.6666666666666666},"check":"two generated","expected":{"b":1.0},"passed":false},{"actual":{},"check":"empty input","expected":{},"passed":true},{"actual":{"c":0.5},"check":"zero prompt","expected":{"c":0.5},"passed":true},{"actual":{"d":0.6666666666666666},"check":"long prefix","expected":{"d":2.0},"passed":false}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"shared prompt removed\", \"actual\": {\"a\": 0.3333333333333333}, \"expected\": {\"a\": 0.5}, \"passed\": false}, {\"check\": \"prompt only floor\", \"actual\": {\"a\": 1.0}, \"expected\": {\"a\": 1.0}, \"passed\": true}, {\"check\": \"two generated\", \"actual\": {\"b\": 0.6666666666666666}, \"expected\": {\"b\": 1.0}, \"passed\": false}, {\"check\": \"empty input\", \"actual\": {}, \"expected\": {}, \"passed\": true}, {\"check\": \"zero prompt\", \"actual\": {\"c\": 0.5}, \"expected\": {\"c\": 0.5}, \"passed\": true}, {\"check\": \"long prefix\", \"actual\": {\"d\": 0.6666666666666666}, \"expected\": {\"d\": 2.0}, \"passed\": false}], \"passed\": false}\n"}},"member_only":{"stages":["fixed"],"fields":["implementations.fixed","verification.fixed","harness","repair"],"note":"The verified repair, its recorded checks, the repair description, and the scoring harness are available to members."}}