SignDarpan: A Free, On-Device Indian Sign Language Coach -- Signer-Disjoint Benchmarking, Representation Transfer, and Fail-Closed Deployment

We report three findings from building and deploying SignDarpan, a free, non-commercial, in-browser tool that helps hearing families and educators of Deaf children practice 262 Indian Sign Language (ISL) signs against real Deaf-signer video. (1) To our knowledge, the first signer-disjoint evaluation of INCLUDE, the standard ISL benchmark, whose official split shares signers - and even single recording bursts - across train and test. Changing only the split, top-1 accuracy falls from 92.5% to 69.25%, and a 7-fold leave-one-signer-out sweep averages 68.2% (range 50.8-80.3%). Screening all 140 papers that cite INCLUDE, we find no signer-disjoint evaluation. (2) At low signer count, representation beats architecture: the Kaggle GISLR-winning Conv1D-Transformer gains 12.2 signer-disjoint points when its raw-joint input is replaced by a signer-invariant feature vector, on the identical architecture and recipe. (3) A deployment post-mortem: a one-line server misconfiguration silently stopped the model from ever executing in production while a geometric fallback impersonated it - exposed not by CI but by a Deaf-expert red team at the Indian Sign Language Research and Training Centre (ISLRTC), New Delhi. On an exhaustive 159,210-pair impostor sweep, the fail-closed judge we rebuilt takes false accepts from 23 to 0 and true accepts from 60.0% to 84.9%, the latter after an ablation retired refusal rules our measurements showed were inert. Applying finding (1) to that audit, however, the guarantee proves to be a property of the evaluation regime rather than of the system: rebuilt signer-disjoint and tested on an unseen signer, the same policy yields 161 false accepts and 62.9% true accepts. Silent fallbacks let deployed ML lie; a safety mechanism nobody has ablated has a known cost and a guessed benefit; and evidence standards for social-good ML must include unseen-user evaluation.