An NLP-Based Approach for Surgical Site Infection Surveillance Using Hospital Discharge Letters

Abstract Background Surgical Site Infections (SSIs) represent a significant proportion of healthcare-associated infections and pose a major burden in hospital settings. Traditional SSI surveillance, based on manual review of free-text clinical records, is time-consuming and poorly scalable. This study aims to develop and evaluate a semi-automated surveillance system based on artificial intelligence (AI) for SSI detection using Italian-language clinical records from a large tertiary hospital. Methods A retrospective analysis was conducted on anonymized unstructured hospital discharge letters from Pisa Hospital (Italy). Clinical text was processed and vectorized using TF-IDF, Word2Vec, and BERT (including pre-training and fine-tuning). Classifiers included Logistic Regression, Decision Trees, Random Forests, and XGBoost. The final performance was evaluated using stratified 10-fold cross-validation, F1-score, and AUC. Given the 1:100 imbalance between SSI and non-SSI records, oversampling and undersampling techniques were tested. Additionally, two Large Language Models (LLMs)-OpenBioLLM and Minerva-were evaluated using zero-shot and few-shot prompting. Results The best performance was achieved by a BERT model further pre-trained on the domain-specific corpus and fine-tuned for classification (BERT-FT), with XGBoost as the classifier: F1-score = 0.79, Precision = 0.86, Recall = 0.74. Traditional vectorization methods (TF-IDF and Word2Vec) yielded significantly lower F1-scores (maximum = 0.34). LLMs underperformed, classifying all cases as negative, likely due to limitations in Italian medical language comprehension and token constraints. Conclusions This study shows the feasibility of using AI for surveillance of SSI on unstructured hospital discharge letters. The optimized BERT model significantly outperformed both traditional NLP techniques and generic LLM. BERT-based systems represent a promising avenue for scalable, real-time surveillance of infections in healthcare. Key messages • A fine-tuned BERT model achieved the best classification results (F1 = 0.79), outperforming all traditional NLP and LLM-based approaches. • AI systems can support semi-automated SSI surveillance in real-world hospital contexts, especially when tailored to domain-specific and language-specific characteristics.