Entity recovery in criminal investigations: Evaluating NER and investigator labels on real case texts

We investigate how effectively automatic named entity recognition can recover entities selected by investigators in real criminal cases. Two homicide cases and one synthetic case were processed with five NER models, and extracted entities were matched to investigator labels using a lightweight entity resolution and similarity scoring method. Police trained models aligned best with investigator annotations, while general-purpose models produced larger but less relevant entity sets. Manual validation confirmed that many labels are not text recoverable, establishing a realistic upper bound on NER performance. The results indicate that extraction and investigative relevance are distinct tasks, motivating a two-stage pipeline and future work on relevance modelling and improved resolution.