Phishing in the LLM Era: Which Website Features Still Separate Phishing from Legitimate When Attackers Clean Up Their Act?

Classical phishing detectors were trained against sloppy attackers: IP-literal hosts, hyphen-stuffed domains, URL shorteners, absurd lengths. Large language models and cheap tooling have industrialised the opposite kind of phishing — grammatically clean, lexically tidy, HTTPS-served. This paper asks which parts of a classical feature set still earn their keep when the lexical sloppiness is removed. On the UCI Phishing Websites corpus (11,055 sites, 30 ternary features), we partition the features into three families — lexical/URL-structure (LEX, 14), host and reputation (HOST, 9), and HTML/JavaScript content behaviour (CONTENT, 7) — and run a graded ablation over three tabular models (logistic regression, random forest, gradient boosting) with operating thresholds calibrated on a separate validation split at 1% and 5% false-positive rates. Three findings survive every model. First, the CONTENT family is nearly worthless on this corpus: deleting it costs at most 0.003 AUC, and alone it reaches only 0.59–0.61 AUC, barely above chance. Second, LEX and HOST are both load-bearing and partially redundant: each alone reaches 0.92–0.97 AUC, and deleting either costs 0.017–0.052. Third, the clean-attacker test splits by how it is run. Among real test samples, the 127 phishing pages with the fewest lexical flaws are still caught at 96.1% recall (random forest, 5% FPR threshold) — their host features betray them. But under a counterfactual edit that sets the lexical features of every phishing page to legitimate-like values, detection at the strict 1% FPR threshold falls to zero for every model, and at the 5% threshold only the random forest degrades gracefully (recall 0.984 → 0.852 for a 9-feature clean-up, → 0.420 for a full lexical clean-up) while logistic regression and gradient boosting collapse (0.19 and 0.10). We also report a negative result about the dataset itself: it contains not one phishing sample that is clean on all nine strict lexical features, so the truly clean attacker is, in 2015 data, a counterfactual — and conclusions about the LLM era drawn from this corpus must be read with that limit in view.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-13
DOI
https://doi.org/10.5281/zenodo.24209074
Primary Topic
Spam and Phishing Detection
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Phishing in the LLM Era: Which Website Features Still Separate Phishing from Legitimate When Attackers Clean Up Their Act?

Saagar Sengupta
1 links
Zenodo (CERN European Organization for Nuclear Research)
Spam and Phishing Detection
preprint

Phishing in the LLM Era: Which Website Features Still Separate Phishing from Legitimate When Attackers Clean Up Their Act?

Saagar Sengupta
preprint en

Abstract

Classical phishing detectors were trained against sloppy attackers: IP-literal hosts, hyphen-stuffed domains, URL shorteners, absurd lengths. Large language models and cheap tooling have industrialised the opposite kind of phishing — grammatically clean, lexically tidy, HTTPS-served. This paper asks which parts of a classical feature set still earn their keep when the lexical sloppiness is removed. On the UCI Phishing Websites corpus (11,055 sites, 30 ternary features), we partition the features into three families — lexical/URL-structure (LEX, 14), host and reputation (HOST, 9), and HTML/JavaScript content behaviour (CONTENT, 7) — and run a graded ablation over three tabular models (logistic regression, random forest, gradient boosting) with operating thresholds calibrated on a separate validation split at 1% and 5% false-positive rates. Three findings survive every model. First, the CONTENT family is nearly worthless on this corpus: deleting it costs at most 0.003 AUC, and alone it reaches only 0.59–0.61 AUC, barely above chance. Second, LEX and HOST are both load-bearing and partially redundant: each alone reaches 0.92–0.97 AUC, and deleting either costs 0.017–0.052. Third, the clean-attacker test splits by how it is run. Among real test samples, the 127 phishing pages with the fewest lexical flaws are still caught at 96.1% recall (random forest, 5% FPR threshold) — their host features betray them. But under a counterfactual edit that sets the lexical features of every phishing page to legitimate-like values, detection at the strict 1% FPR threshold falls to zero for every model, and at the 5% threshold only the random forest degrades gracefully (recall 0.984 → 0.852 for a 9-feature clean-up, → 0.420 for a full lexical clean-up) while logistic regression and gradient boosting collapse (0.19 and 0.10). We also report a negative result about the dataset itself: it contains not one phishing sample that is clean on all nine strict lexical features, so the truly clean attacker is, in 2015 data, a counterfactual — and conclusions about the LLM era drawn from this corpus must be read with that limit in view.

Zenodo (CERN European Organization for Nuclear Research)
Spam and Phishing Detection
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon