Enhancing cancer symptom detection in electronic health records using artificial intelligence

This dissertation investigates the application of artificial intelligence (AI), including machine learning (ML), deep learning (DL), and large language models (LLMs), to advance cancer symptom detection and prediction. By addressing the complexities of unstructured clinical notes, this research bridges gaps in existing methodologies and contributes to improving personalized symptom management. The study emphasizes dual-level analyses: document-level modeling to identify broad symptom patterns and token-level processing to perform detailed symptom profiling. A systematic review identifies prevailing trends, challenges, and opportunities in ML and DL-based symptom prediction. Building on these findings, the research develops Symptom-BERT, a domain-specific model fine-tuned on cancer-related EHRs. This model achieves superior internal validation (F1-score: 0.933, AUC: 0.929) and robust external validation with synthetic notes (F1-score: 0.831, AUC: 0.834). Comparative analysis of fine-tuning and prompt-based learning approaches highlights trade-offs in computational efficiency, precision, and adaptability. This demonstrates that fine-tuning excels in detecting physical symptoms with consistent documentation. In contrast, prompt learning offers flexibility for broader applications. The dissertation underscores the role of AI in bridging structured and unstructured data, enhancing predictive accuracy, and supporting proactive healthcare interventions. Key contributions include improved psychological symptom detection, scalable AI frameworks for unstructured clinical text, and insights into multimodal data integration for future research. Despite challenges such as data bias, resource intensity, and interpretability, the findings advance symptom science and establish a foundation for integrating AI-driven methods into clinical workflows. These contributions offer significant implications for oncology and other medical domains, enabling more accurate, efficient, and patient-centered healthcare.