FA-201 / Encodings / Open access

BOM cleanup removes a legitimate character inside the document · case 01

A leading encoding signature is exposed as text, or cleanup also strips matching interior characters.

Verified by executionVariant 1 · 7 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The implementation treats a document-start byte signature as a character to retain or remove everywhere.

VERIFIED REPAIR

Consume one UTF-8 BOM only at byte offset zero, then strictly decode the remaining bytes.

Unsuccessful approach: A global replacement deletes U+FEFF characters that are part of the content and removes multiple leading signatures.

Case contract

Decode UTF-8 bytes, removing at most one EF BB BF signature at the start. Preserve every subsequent U+FEFF character; return None for invalid UTF-8.

Why this case matters

Encoding signatures are positional metadata. Global text cleanup cannot preserve the distinction between metadata and document content.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(data):
    try:
        return data.decode('utf-8')
    except UnicodeDecodeError:
        return None
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
bom = b'\xef\xbb\xbf'
text = 'part' * N
check('leading BOM removed', solve(bom + text.encode()), text)
check('interior character preserved', solve(text.encode() + bom + b'end'), text + '\ufeffend')
check('only one leading signature consumed', solve(bom + bom + text.encode()), '\ufeff' + text)
check('signature-only input', solve(bom), '')
check('ordinary UTF-8 retained', solve(('é' * N).encode()), 'é' * N)
check('empty input', solve(b''), '')
check('invalid UTF-8 rejected', solve(bom + b'\xff'), None)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
leading BOM removed	`part`	`part`	Failed
interior character preserved	`partend`	`partend`	Passed
only one leading signature consumed	`part`	`part`	Failed
signature-only input			Failed
ordinary UTF-8 retained	`é`	`é`	Passed
empty input			Passed
invalid UTF-8 rejected	`None`	`None`	Passed

SHA-256 / 31a1fbb2215e6eee0f9a48c2bba0152505a3759a733415c48595911dac11ee8e

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(data):
    try:
        return data.decode('utf-8').replace('\ufeff', '')
    except UnicodeDecodeError:
        return None
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
bom = b'\xef\xbb\xbf'
text = 'part' * N
check('leading BOM removed', solve(bom + text.encode()), text)
check('interior character preserved', solve(text.encode() + bom + b'end'), text + '\ufeffend')
check('only one leading signature consumed', solve(bom + bom + text.encode()), '\ufeff' + text)
check('signature-only input', solve(bom), '')
check('ordinary UTF-8 retained', solve(('é' * N).encode()), 'é' * N)
check('empty input', solve(b''), '')
check('invalid UTF-8 rejected', solve(bom + b'\xff'), None)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
leading BOM removed	`part`	`part`	Passed
interior character preserved	`partend`	`partend`	Failed
only one leading signature consumed	`part`	`part`	Failed
signature-only input			Passed
ordinary UTF-8 retained	`é`	`é`	Passed
empty input			Passed
invalid UTF-8 rejected	`None`	`None`	Passed

SHA-256 / 824958450ddf0a061e366b0ebb323b0a74e0aebab07f5eec4f0f4ad04cefdb9a

3 / The verified repair

Exit 0

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(data):
    try:
        return data.decode('utf-8-sig')
    except UnicodeDecodeError:
        return None
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
bom = b'\xef\xbb\xbf'
text = 'part' * N
check('leading BOM removed', solve(bom + text.encode()), text)
check('interior character preserved', solve(text.encode() + bom + b'end'), text + '\ufeffend')
check('only one leading signature consumed', solve(bom + bom + text.encode()), '\ufeff' + text)
check('signature-only input', solve(bom), '')
check('ordinary UTF-8 retained', solve(('é' * N).encode()), 'é' * N)
check('empty input', solve(b''), '')
check('invalid UTF-8 rejected', solve(bom + b'\xff'), None)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
leading BOM removed	`part`	`part`	Passed
interior character preserved	`partend`	`partend`	Passed
only one leading signature consumed	`part`	`part`	Passed
signature-only input			Passed
ordinary UTF-8 retained	`é`	`é`	Passed
empty input			Passed
invalid UTF-8 rejected	`None`	`None`	Passed

SHA-256 / 6d458b9067745a530cf910477020a85169666e2266205f39fc4c0a51267124a1

Verification & scope

This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:36:51.041225+00:00.

Case digest / 8e68ddfcd98fbb43f296c5829eca36e27eabf770274be32b804cc85f3993ac7d