Measuring the Indus corpus before testing it: edition disagreement, near-copy leakage and detection limits

Statistical and machine-learning work on the Indus script rests on a corpus of 2,270 distinct short texts. We measure three properties of this corpus that decide what such work can show. (1) The editions disagree. Matching 469 objects across the two standard transcriptions through the 1930s excavation registers, independently of their signs, we find that the editions read 12.9% of signs differently (95% CI 11.1–14.9%; 14.0% out of sample), and one text in three differs somewhere. (2) Random splits leak. Because moulded tablets and fixed formulae put near-copies of a text on different objects, a random split by object leaves a near-copy of 37% of test texts in training even after exact repeats are removed (median of 200 splits; 95% range 33–42%), and of 62% if they are not. (3) Many questions lie below the detection limit. Planted-answer controls show that a genre difference must exceed 0.57 bits per text to be found, that a test for a relational reading of a joining sign fails at Indus size and works on a corpus four times larger, and that a sign-order statistic passes on four of seven non-linguistic codes. We release a family-level split and a corrected concordance between the two sign lists, and recommend reporting practice for future quantitative work on the script.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-19
DOI
https://doi.org/10.5281/zenodo.24371566
Primary Topic
Digital Humanities and Scholarship
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Measuring the Indus corpus before testing it: edition disagreement, near-copy leakage and detection limits

Sriharsha Garlapati
Zenodo (CERN European Organization for Nuclear Research)
Digital Humanities and Scholarship
preprint

Measuring the Indus corpus before testing it: edition disagreement, near-copy leakage and detection limits

Sriharsha Garlapati
preprint en

Abstract

Statistical and machine-learning work on the Indus script rests on a corpus of 2,270 distinct short texts. We measure three properties of this corpus that decide what such work can show. (1) The editions disagree. Matching 469 objects across the two standard transcriptions through the 1930s excavation registers, independently of their signs, we find that the editions read 12.9% of signs differently (95% CI 11.1–14.9%; 14.0% out of sample), and one text in three differs somewhere. (2) Random splits leak. Because moulded tablets and fixed formulae put near-copies of a text on different objects, a random split by object leaves a near-copy of 37% of test texts in training even after exact repeats are removed (median of 200 splits; 95% range 33–42%), and of 62% if they are not. (3) Many questions lie below the detection limit. Planted-answer controls show that a genre difference must exceed 0.57 bits per text to be found, that a test for a relational reading of a joining sign fails at Indus size and works on a corpus four times larger, and that a sign-order statistic passes on four of seven non-linguistic codes. We release a family-level split and a corrected concordance between the two sign lists, and recommend reporting practice for future quantitative work on the script.

Zenodo (CERN European Organization for Nuclear Research)
Digital Humanities and Scholarship
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon