FA-2996 / Text processing / Open access

Whitespace tokenization creates empty tokens · case 01

Whitespace tokenization creates empty tokens.

Verified by executionVariant 1 · 4 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The implementation applies an operation whose text or grammar semantics violate this contract: Split on runs of Unicode whitespace with no leading/trailing empty fields.

VERIFIED REPAIR

Implement the complete stated contract, including the boundary fixtures: Split on runs of Unicode whitespace with no leading/trailing empty fields.

Unsuccessful approach: The attempted repair handles the primary example but still violates a separate boundary of the same contract.

Case contract

Split on runs of Unicode whitespace with no leading/trailing empty fields.

Why this case matters

A local executable model for consumers of structured text; oracle values are authored literals, not outputs copied from the repaired implementation.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json
import re, unicodedata, json, csv, io, html, base64, binascii, codecs, struct
from urllib.parse import quote, unquote, unquote_to_bytes, urlencode, parse_qsl, urlsplit, urlunsplit
from email.header import decode_header, make_header
from email.utils import getaddresses
from xml.etree import ElementTree as ET
import shlex, string, textwrap

N = 1
observations = []
def solve(x):
    try:
        return x.split(' ')
    except Exception as exc:
        return {"error": type(exc).__name__}
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('boundary 1', solve(' a  b '), ['a', 'b'])
check('boundary 2', solve('\tA\xa0B'), ['A', 'B'])
check('boundary 3', solve(''), [])
check('boundary 4', solve('  '), [])
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
boundary 1	`['', 'a', '', 'b', '']`	`['a', 'b']`	Failed
boundary 2	`['\tA\xa0B']`	`['A', 'B']`	Failed
boundary 3	`['']`	`[]`	Failed
boundary 4	`['', '', '']`	`[]`	Failed

SHA-256 / a9e35113a056864e6464c8a4b2a7499f214c43b0a4849813fff8ccf1fc76f4e8

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json
import re, unicodedata, json, csv, io, html, base64, binascii, codecs, struct
from urllib.parse import quote, unquote, unquote_to_bytes, urlencode, parse_qsl, urlsplit, urlunsplit
from email.header import decode_header, make_header
from email.utils import getaddresses
from xml.etree import ElementTree as ET
import shlex, string, textwrap

N = 1
observations = []
def solve(x):
    try:
        return re.split(r'\s+',x.strip())
    except Exception as exc:
        return {"error": type(exc).__name__}
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('boundary 1', solve(' a  b '), ['a', 'b'])
check('boundary 2', solve('\tA\xa0B'), ['A', 'B'])
check('boundary 3', solve(''), [])
check('boundary 4', solve('  '), [])
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
boundary 1	`['a', 'b']`	`['a', 'b']`	Passed
boundary 2	`['A', 'B']`	`['A', 'B']`	Passed
boundary 3	`['']`	`[]`	Failed
boundary 4	`['']`	`[]`	Failed

SHA-256 / e0a9fcce0f12d766cb852019b29934ad1033a8d0a868c147cc21a528f24078b5

3 / The verified repair

Exit 0

"""Failure Map reference implementation. Python standard library only."""
import json
import re, unicodedata, json, csv, io, html, base64, binascii, codecs, struct
from urllib.parse import quote, unquote, unquote_to_bytes, urlencode, parse_qsl, urlsplit, urlunsplit
from email.header import decode_header, make_header
from email.utils import getaddresses
from xml.etree import ElementTree as ET
import shlex, string, textwrap

N = 1
observations = []
def solve(x):
    try:
        return x.split()
    except Exception as exc:
        return {"error": type(exc).__name__}
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('boundary 1', solve(' a  b '), ['a', 'b'])
check('boundary 2', solve('\tA\xa0B'), ['A', 'B'])
check('boundary 3', solve(''), [])
check('boundary 4', solve('  '), [])
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
boundary 1	`['a', 'b']`	`['a', 'b']`	Passed
boundary 2	`['A', 'B']`	`['A', 'B']`	Passed
boundary 3	`[]`	`[]`	Passed
boundary 4	`[]`	`[]`	Passed

SHA-256 / b565c6c6b3cbf4442b5451686ea29952bf50af3791a991877181a1cd1ff3de7d

Verification & scope

This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:37:19.908125+00:00.

Case digest / 404cf0494b91f7856f16fe9b56bd9a5d3e965641e1f42dc41c3d2a4e6e922965