FA-100251 / Dataset deduplication grouping / Open access
Dataset deduplication: formatting whitespace creates duplicate records · case 01
formatting whitespace creates duplicate records.
ROOT CAUSE
Runs of whitespace are not normalized before the identity comparison.
THE FAILURE
Runs of whitespace are not normalized before the identity comparison.
Unsuccessful approach: Trimming ends still treats different internal whitespace runs as distinct.
Case contract
Return distinct keys after collapsing each whitespace run to one space and trimming ends; case is preserved.
Why this case matters
This deterministic toy operation makes one dataset equivalence or representative rule explicit.
1 / The failure
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(values):
return len(set(values))
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('tabs and spaces',solve(['a b','a\tb']),1)
check('outer whitespace',solve([' x ','x']),1)
check('internal word boundary',solve(['ab','a b']),2)
check('empty strings',solve(['',' ']),1)
check('empty input',solve([]),0)
check('single token',solve(['a']),1)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| tabs and spaces | 2 | 1 | Failed |
| outer whitespace | 2 | 1 | Failed |
| internal word boundary | 2 | 2 | Passed |
| empty strings | 2 | 1 | Failed |
| empty input | 0 | 0 | Passed |
| single token | 1 | 1 | Passed |
SHA-256 / 9a7cb6ca50d5cdd72b35d9d214c8f4e2de158a07e5fd1667f392b324ebdcb498
2 / The unsuccessful fix
Exit 1"""Failure Map reference implementation. Python standard library only."""
import json
N = 1
observations = []
def solve(values):
return len({s.strip() for s in values})
def check(label, actual, expected):
observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('tabs and spaces',solve(['a b','a\tb']),1)
check('outer whitespace',solve([' x ','x']),1)
check('internal word boundary',solve(['ab','a b']),2)
check('empty strings',solve(['',' ']),1)
check('empty input',solve([]),0)
check('single token',solve(['a']),1)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)
| Boundary fixture | Actual | Expected | Outcome |
|---|---|---|---|
| tabs and spaces | 2 | 1 | Failed |
| outer whitespace | 1 | 1 | Passed |
| internal word boundary | 2 | 2 | Passed |
| empty strings | 1 | 1 | Passed |
| empty input | 0 | 0 | Passed |
| single token | 1 | 1 | Passed |
SHA-256 / aa5ea745fcaf10af57918c2ef05e3d5f8850d7ad117f6bc1c44b4265167be782
HELD IN THE MEMBER ARCHIVE
The verified repair and its recorded checks are member-only.
This mechanism has 6 recorded checks per implementation. The open-access tier publishes the failure and the unsuccessful fix; the repaired source that passes every check, and the observations that prove it, are available to members.
Every case sharing this mechanism uses the same contract and the same repair, so this one record is held back for all of them.
Member access is invitation-based. Sign in with your invited account to inspect the repair.
Sign in to the archive ↗Verification & scope
Inputs are small in-memory fixtures with the stated contract; results do not claim production deduplication quality. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.
Observations recorded using Python 3.12.14 at 2026-09-30T16:48:33.560960+00:00.
Case digest / 1e8ee8574c591b9145bfa2349e828bc3287f5c33fff7395708748bbe4c3a62df