FA-3261 / Tokenization / Open access

Whitespace splitting ignores shell quote boundaries · case 01

Whitespace splitting ignores shell quote boundaries.

Verified by executionVariant 1 · 4 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The implementation applies an operation whose text or grammar semantics violate this contract: Tokenize POSIX shell words without comment processing; perform no expansion.

VERIFIED REPAIR

Implement the complete stated contract, including the boundary fixtures: Tokenize POSIX shell words without comment processing; perform no expansion.

Unsuccessful approach: The attempted repair handles the primary example but still violates a separate boundary of the same contract.

Case contract

Tokenize POSIX shell words without comment processing; perform no expansion.

Why this case matters

A local executable model for consumers of structured text; oracle values are authored literals, not outputs copied from the repaired implementation.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json
import re, unicodedata, json, csv, io, html, base64, binascii, codecs, struct
from urllib.parse import quote, unquote, unquote_to_bytes, urlencode, parse_qsl, urlsplit, urlunsplit
from email.header import decode_header, make_header
from email.utils import getaddresses
from xml.etree import ElementTree as ET
import shlex, string, textwrap

N = 1
observations = []
def solve(x):
    try:
        return x.split()
    except Exception as exc:
        return {"error": type(exc).__name__}
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('boundary 1', solve('a "b c"'), ['a', 'b c'])
check('boundary 2', solve("'' x"), ['', 'x'])
check('boundary 3', solve('a\\ b'), ['a b'])
check('boundary 4', solve('x #y'), ['x', '#y'])
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
boundary 1	`['a', '"b', 'c"']`	`['a', 'b c']`	Failed
boundary 2	`["''", 'x']`	`['', 'x']`	Failed
boundary 3	`['a\\', 'b']`	`['a b']`	Failed
boundary 4	`['x', '#y']`	`['x', '#y']`	Passed

SHA-256 / f90df1724d8fbb914f08f0747747ee378f531c1c01491b4a2c76a2fe1bbfa78d

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json
import re, unicodedata, json, csv, io, html, base64, binascii, codecs, struct
from urllib.parse import quote, unquote, unquote_to_bytes, urlencode, parse_qsl, urlsplit, urlunsplit
from email.header import decode_header, make_header
from email.utils import getaddresses
from xml.etree import ElementTree as ET
import shlex, string, textwrap

N = 1
observations = []
def solve(x):
    try:
        return [v.replace(chr(34),chr(0)).replace(chr(0),chr(32)).strip() for v in x.split()]
    except Exception as exc:
        return {"error": type(exc).__name__}
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('boundary 1', solve('a "b c"'), ['a', 'b c'])
check('boundary 2', solve("'' x"), ['', 'x'])
check('boundary 3', solve('a\\ b'), ['a b'])
check('boundary 4', solve('x #y'), ['x', '#y'])
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
boundary 1	`['a', 'b', 'c']`	`['a', 'b c']`	Failed
boundary 2	`["''", 'x']`	`['', 'x']`	Failed
boundary 3	`['a\\', 'b']`	`['a b']`	Failed
boundary 4	`['x', '#y']`	`['x', '#y']`	Passed

SHA-256 / e81f4bcca2d4ae48327f9b5ac73b6e240234ee4ffd9832223dbcc9cbd7711485

3 / The verified repair

Exit 0

"""Failure Map reference implementation. Python standard library only."""
import json
import re, unicodedata, json, csv, io, html, base64, binascii, codecs, struct
from urllib.parse import quote, unquote, unquote_to_bytes, urlencode, parse_qsl, urlsplit, urlunsplit
from email.header import decode_header, make_header
from email.utils import getaddresses
from xml.etree import ElementTree as ET
import shlex, string, textwrap

N = 1
observations = []
def solve(x):
    try:
        return shlex.split(x,comments=False,posix=True)
    except Exception as exc:
        return {"error": type(exc).__name__}
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('boundary 1', solve('a "b c"'), ['a', 'b c'])
check('boundary 2', solve("'' x"), ['', 'x'])
check('boundary 3', solve('a\\ b'), ['a b'])
check('boundary 4', solve('x #y'), ['x', '#y'])
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
boundary 1	`['a', 'b c']`	`['a', 'b c']`	Passed
boundary 2	`['', 'x']`	`['', 'x']`	Passed
boundary 3	`['a b']`	`['a b']`	Passed
boundary 4	`['x', '#y']`	`['x', '#y']`	Passed

SHA-256 / fd275232dcbfd4f95393849912cb97b54afa5c6b9b35b217fb876dbd3b5b32c9

Verification & scope

This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:37:23.516983+00:00.

Case digest / 349b5646b292d0eb1a22a786d2b1e093fefc4ff355996a16945308544fe61087