FA-100311 / Dataset deduplication grouping / Open access

Dataset deduplication: final token shingle is omitted · case 01

final token shingle is omitted.

Verified by executionVariant 1 · 6 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The range endpoint treats the final valid start as out of bounds.

THE FAILURE

The range endpoint treats the final valid start as out of bounds.

Unsuccessful approach: Repairing ordinary longer windows but retaining a strict exact-width guard still misses the sole valid window.

Case contract

Return sorted tuple shingles for every contiguous window of positive width; if input is shorter, return empty.

Why this case matters

This deterministic toy operation makes one dataset equivalence or representative rule explicit.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(tokens, width):
    return sorted(tuple(tokens[i:i+width]) for i in range(max(0,len(tokens)-width)))
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('last window',solve(list(range(N+2)),2),[tuple(range(i,i+2)) for i in range(N+1)])
check('exactly one window',solve([1,2],2),[(1,2)])
check('too short',solve([1],2),[])
check('empty tokens',solve([],1),[])
check('width one',solve([1,2],1),[(1,),(2,)])
check('width exceeds length',solve([1,2],3),[])
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
last window	`[[0, 1]]`	`[[0, 1], [1, 2]]`	Failed
exactly one window	`[]`	`[[1, 2]]`	Failed
too short	`[]`	`[]`	Passed
empty tokens	`[]`	`[]`	Passed
width one	`[[1]]`	`[[1], [2]]`	Failed
width exceeds length	`[]`	`[]`	Passed

SHA-256 / 6d724fe002ef4ac4b60ae822b7d97a5907e4feeb9520a7c94f5f6f71ded0333a

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json

N = 1
observations = []
def solve(tokens, width):
    starts=list(range(max(0,len(tokens)-width)))
    if len(tokens)>width: starts.append(len(tokens)-width)
    return sorted(tuple(tokens[i:i+width]) for i in starts)
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
check('last window',solve(list(range(N+2)),2),[tuple(range(i,i+2)) for i in range(N+1)])
check('exactly one window',solve([1,2],2),[(1,2)])
check('too short',solve([1],2),[])
check('empty tokens',solve([],1),[])
check('width one',solve([1,2],1),[(1,),(2,)])
check('width exceeds length',solve([1,2],3),[])
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
last window	`[[0, 1], [1, 2]]`	`[[0, 1], [1, 2]]`	Passed
exactly one window	`[]`	`[[1, 2]]`	Failed
too short	`[]`	`[]`	Passed
empty tokens	`[]`	`[]`	Passed
width one	`[[1], [2]]`	`[[1], [2]]`	Passed
width exceeds length	`[]`	`[]`	Passed

SHA-256 / f32183b81cca048a0aea2dd65a95512293dff7bba532307a604a52a82c163e80

HELD IN THE MEMBER ARCHIVE

The verified repair and its recorded checks are member-only.

This mechanism has 6 recorded checks per implementation. The open-access tier publishes the failure and the unsuccessful fix; the repaired source that passes every check, and the observations that prove it, are available to members.

Every case sharing this mechanism uses the same contract and the same repair, so this one record is held back for all of them.

Member access is invitation-based. Sign in with your invited account to inspect the repair.

Verification & scope

Inputs are small in-memory fixtures with the stated contract; results do not claim production deduplication quality. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-30T16:48:34.014397+00:00.

Case digest / c51831052949112510733347b5aa2507afd5e1996edfe1ebae8df534ba46c212