How much model does warning triage need? A cost-aware comparison from rules to locally run language models

Static analysis tools report far more warnings than developers act on, and machine learning is widely proposed to identify the actionable ones. Published comparisons rarely include a simple label-based reference, and rarely report quality on unseen projects together with compute time. This study compares five levels of model complexity on two open datasets, NASCAR (Java) and AW4C (C): a warning-type prior, a linear model on term frequency-inverse document frequency (TF-IDF) features, a frozen code encoder, fine-tuned code encoders, and open-weight large language models (LLMs) run locally. All models were scored on the same 1,000 warnings per dataset and protocol, using the Matthews correlation coefficient (MCC) with project-level bootstrap intervals. Three findings emerged. First, in the original AW4C release the number of lines in the supplied code context predicted the label with an MCC of 0.91 on unseen projects, and fine-tuned CodeBERT reached 0.81; in a variant rebuilt with a uniform context window, which also differs in composition, the corresponding values were 0.12 and 0.30. Second, on unseen NASCAR projects an advantage of fine-tuned encoders over the warning-type prior was not established: the paired difference in MCC was −0.08 (95% CI −0.21 to 0.02) for CodeBERT. Third, locally run LLMs without task-specific training scored near zero on NASCAR with the default threshold and required about 47 times the mean processing time of a fine-tuned encoder. Model complexity should be assessed against a simple reference under cross-project evaluation.