Benchmark Shortcuts and Append Robustness in Phishing Detection: Evidence from University Scam Emails

Phishing detectors trained on public benchmarks score highly in-domain yet miss scams aimed at university users. We audit TF-IDF, MiniLM and RoBERTa detectors with 5146 scams published by five U.S. universities, 594 scams posted since 2023, 2684 legitimate university announcements and mailing-list messages, and 17,880 labelled job advertisements. In one benchmark, message year separates phishing from legitimate mail; in both, donor-identifying terms, 2% of the vocabulary, carry 14–16% of the legitimacy weight. Trained on that benchmark, TF-IDF and RoBERTa detect scams containing academic vocabulary about 15 percentage points less often. Thresholds set for a 5% false-positive rate on benchmark mail flag between 1.0% and 87.8% of legitimate job advertisements. Calibrated instead on job advertisements grouped by company, TF-IDF flags 2.1% of university announcements but 65.3% of short legitimate financial messages, and appended legitimate-looking words cut its recall from 0.828 to 0.123. Bounding legitimacy evidence yields closed-form certificates for whitespace-delimited appends, prepends and interior insertions. A budget-split ensemble certifies 0.259 recall against appends or prepends of any length at 0.706 clean recall, whereas adversarially trained TF-IDF falls to 0.078 under a 40-token white-box append. Monotone components, however, flag many long formal letters.