GapQA: A Span-Deletion Benchmark with Recoverability Known by Construction

Language models increasingly answer from contexts assembled by retrieval, truncation or summarisation, and whether the evidence for an answer survived that assembly is rarely known. GapQA deletes spans from annotated question-answering contexts so that recoverability is known by construction: an item is recoverable exactly when every evidence unit survives. Version 1 draws on six training sources and four held-out sets in three languages (4,000 training, 592 development and 4,682 test items), stores deletions as character offsets so that every tokenizer sees the same holes, removes the 16.5% of candidate questions that any of seven open models answers without context, and blocks title leakage between splits. Fixing gap lengths before placement and pairing each set of lengths across the two classes makes hole size uninformative: size-only predictors score an area under the ROC curve (AUROC) of 0.500. A BM25 floor reaches 0.643 in-domain but stays below 0.70 on five lexically hard sources (macro-average 0.570). Zero-shot, six open models of at most 3.2B parameters reach 0.555-0.830 and a written omission marker does not help; after low-rank fine-tuning they reach 0.871-0.953 in-domain and 0.840-0.922 on held-out sources and languages, with QASPER hardest. A pre-registered case study finds no gain from a learned gap-token embedding over the written marker across six models and lower AUROC on the held-out sets, significantly on two of four; a descriptive probe places the learned vector among untrained glyph embeddings. GapQA, its construction code and every score are released, and all registered tests are reported against their criteria.