{"abstract":"domain holdout leaks through a second domain label.","category":"Dataset split isolation","checks":6,"contract":"Rows [id,domains]; hold out the lexically greatest domain and return rows containing it.","contract_signature":"rows","evaluation_group":"ds-domain-holdout","failed_approach":"Using only the first domain misses rows where the held-out domain appears later.","family":"w3-dataset-split-isolation-domain-holdout","id":"FA-97891","implementations":{"attempt":{"sha256":"1ccc573658c1316e53c36fb5e6f9a50e64f56910cea3d6ade9fb7bc93e6e1521","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(rows):\n    held=max((x for i,d in rows for x in d),default='')\n    return sorted(i for i,d in rows if held in d and d[0]==held)\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\ncheck('later domain member',solve([['a',['x','z']],['b',['z']]]),['a','b'])\ncheck('empty',solve([]),[])\ncheck('empty memberships',solve([['a',[]],['b',[]]]),[])\ncheck('singleton',solve([['a',['z']]]),['a'])\ncheck('overlap domain',solve([['a',['a','z']],['b',['b']]]),['a'])\ncheck('held domain only',solve([['a',['x']],['b',['y','z']]]),['b'])\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"},"broken":{"sha256":"644aceb835296675c4b68478849ca0ba74740063e93994c3c7d68a1b74e26ce4","source":"\"\"\"Failure Map reference implementation. Python standard library only.\"\"\"\nimport json\n\nN = 1\nobservations = []\ndef solve(rows):\n    held=max((d[0] for i,d in rows if d),default='')\n    return sorted(i for i,d in rows if d and d[0]==held)\ndef check(label, actual, expected):\n    observations.append({\"check\": label, \"actual\": actual, \"expected\": expected, \"passed\": actual == expected})\ncheck('later domain member',solve([['a',['x','z']],['b',['z']]]),['a','b'])\ncheck('empty',solve([]),[])\ncheck('empty memberships',solve([['a',[]],['b',[]]]),[])\ncheck('singleton',solve([['a',['z']]]),['a'])\ncheck('overlap domain',solve([['a',['a','z']],['b',['b']]]),['a'])\ncheck('held domain only',solve([['a',['x']],['b',['y','z']]]),['b'])\nprint(json.dumps({\"observations\": observations, \"passed\": all(x[\"passed\"] for x in observations)}, ensure_ascii=False))\nraise SystemExit(0 if all(x[\"passed\"] for x in observations) else 1)\n"}},"limitations":"Inputs are small in-memory records with self-contained contracts; no claim is made about production sampling quality. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.","method":"Deterministic executable model with adversarial boundary fixtures.","provenance":{"created_by":"Failure Map","dependencies":"Python standard library","family":"w3-dataset-split-isolation-domain-holdout","generated_at":"2026-09-30T16:48:07.116943+00:00","license":"CC0-1.0","python":"3.12.14","seed":1,"split":"open-access"},"relevance":"This toy split allocator makes membership rules explicit for reproducible evaluation fixtures.","root_cause":"Each row can belong to multiple domains and one matching label is missed.","sha256":"7ef6728cb4de1520baf277cc083e15f2358f1f47611cf8951cab6d970b842e17","title":"Dataset split: domain holdout leaks through a second domain label · case 01","variant":1,"variant_policy":"The same explicitly authored fixtures repeat across all five numbered records; runtime N is unused. These records share one mechanism and must remain together in evaluation splits.","verified":true,"visibility":"public","verification":{"attempt":{"elapsed_ms":26.119,"exit_code":1,"observations":[{"actual":["b"],"check":"later domain member","expected":["a","b"],"passed":false},{"actual":[],"check":"empty","expected":[],"passed":true},{"actual":[],"check":"empty memberships","expected":[],"passed":true},{"actual":["a"],"check":"singleton","expected":["a"],"passed":true},{"actual":[],"check":"overlap domain","expected":["a"],"passed":false},{"actual":[],"check":"held domain only","expected":["b"],"passed":false}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"later domain member\", \"actual\": [\"b\"], \"expected\": [\"a\", \"b\"], \"passed\": false}, {\"check\": \"empty\", \"actual\": [], \"expected\": [], \"passed\": true}, {\"check\": \"empty memberships\", \"actual\": [], \"expected\": [], \"passed\": true}, {\"check\": \"singleton\", \"actual\": [\"a\"], \"expected\": [\"a\"], \"passed\": true}, {\"check\": \"overlap domain\", \"actual\": [], \"expected\": [\"a\"], \"passed\": false}, {\"check\": \"held domain only\", \"actual\": [], \"expected\": [\"b\"], \"passed\": false}], \"passed\": false}\n"},"broken":{"elapsed_ms":27.259,"exit_code":1,"observations":[{"actual":["b"],"check":"later domain member","expected":["a","b"],"passed":false},{"actual":[],"check":"empty","expected":[],"passed":true},{"actual":[],"check":"empty memberships","expected":[],"passed":true},{"actual":["a"],"check":"singleton","expected":["a"],"passed":true},{"actual":["b"],"check":"overlap domain","expected":["a"],"passed":false},{"actual":["b"],"check":"held domain only","expected":["b"],"passed":true}],"passed":false,"stderr":"","stdout":"{\"observations\": [{\"check\": \"later domain member\", \"actual\": [\"b\"], \"expected\": [\"a\", \"b\"], \"passed\": false}, {\"check\": \"empty\", \"actual\": [], \"expected\": [], \"passed\": true}, {\"check\": \"empty memberships\", \"actual\": [], \"expected\": [], \"passed\": true}, {\"check\": \"singleton\", \"actual\": [\"a\"], \"expected\": [\"a\"], \"passed\": true}, {\"check\": \"overlap domain\", \"actual\": [\"b\"], \"expected\": [\"a\"], \"passed\": false}, {\"check\": \"held domain only\", \"actual\": [\"b\"], \"expected\": [\"b\"], \"passed\": true}], \"passed\": false}\n"}},"member_only":{"stages":["fixed"],"fields":["implementations.fixed","verification.fixed","harness","repair"],"note":"The verified repair, its recorded checks, the repair description, and the scoring harness are available to members."}}