Three Times the Data, the Same Mistakes: A Manually Audited Evaluation of Seerie, an Offline Scam-Prevention Assistant for India

Seerie is a scam-prevention assistant for India that runs entirely on an Android phone, with no internet connection after the one-time model download and no conversation data leaving the device. This version is a newer model: a QLoRA fine-tune of Qwen3-1.7B on 107,575 conversations in English, Hinglish and Indic scripts (corpus v349), deployed as a 1.11 GB 4-bit GGUF model through llama.cpp. Every answer of the fine-tuned model and of the untuned base model to a 100-prompt suite was rated by hand. The fine-tuned model is safe on 87 of 100 prompts (95% CI 79-92) against 35 for the base model. On the 70 prompts shared with the previous model (36,669 conversations) it is no better - 59 against 58 safe - and it still fabricates legal sections. A held-out test of 34 unseen topics looks near-perfect (271 of 272 automatic passes, ROUGE-L 0.94), but 181 of the answers copy training replies word for word: holding out topics did not hold out the text. The automatic harness again agrees with the manual rating barely above chance (Cohen’s kappa 0.12). This record contains the research paper (PDF and LaTeX source), the on-device model (seerie-v349-q4_k_m.gguf, 4-bit GGUF, 1.11 GB - the file the Seerie Android app runs), all 200 manually rated answers, the raw evaluation outputs, the scripts that recompute every number, and the figures. See README.md for details, intended use and limitations.