MODELING AND INFERENCE FOR REAL-WORLD HEALTH DATA: ADDRESSING COMPLEX TRAJECTORIES, MISSINGNESS, AND UNSTRUCTURED TEXT

The proliferation of complex, high-dimensional, and longitudinal data from sources like clinical trials and Electronic Health Records (EHRs) presents significant analytical challenges. Standard analytical methods are often insufficient, as their underlying assumptions fail to capture the nuanced dynamics of non-linear disease progression, complex microbial ecosystems, and unstructured clinical text. This dissertation addresses these limitations by developing and validating three novel computational frameworks, each tailored to the unique complexities of oncology clinical trials, longitudinal microbiome studies, and unstructured EHR analysis, respectively. In the first project, we address the limitations of conventional models in oncology, which often fail to capture the complex, non-linear nature of treatment response. We introduce a cure rate joint model for the simultaneous analysis of longitudinal tumor burden and time-to-event data. This model integrates a random change-point structure to capture tumor shrinkage followed by potential regrowth, and a cure-rate component for patients achieving long-term disease control. The primary innovation is leveraging survival data to impose a biologically plausible constraint on the change point's timing, ensuring tumor regrowth precedes the clinically observed progression event. Applied to a non-small cell lung cancer trial, the framework yields more accurate estimations of treatment effects and offers a deeper understanding of disease dynamics. The second project confronts the pervasive challenge of irregularly-sampled data in longitudinal microbiome studies. We propose the Bidirectional GRU-ODE-Bayes (BGOB) model, a novel deep-learning-based interpolation framework. BGOB leverages a Neural Ordinary Differential Equation (ODE) architecture to learn the underlying continuous-time trajectories of microbial dynamics directly from sparse observations. Its innovative bidirectional structure processes each time-series both forwards and backwards, enhancing accuracy, particularly for early time points. By interpolating both missing time points (``unavailable'' data) and unobserved biological zeros (``undetectable'' data), BGOB transforms fragmented datasets into complete, regularly-spaced data matrices. As demonstrated through applications to early childhood caries and inflammatory bowel disease cohorts, these imputed datasets significantly improve the statistical power of downstream analyses like differential abundance testing and clustering. The third project introduces the precLLM framework to address critical barriers of patient privacy and computational cost, enabling smaller, locally-deployed Large Language Models (LLMs) for unstructured Electronic Health Record (EHR) analysis. The core innovation is a smart preprocessing step—using techniques like regular expressions and Retrieval-Augmented Generation—that filters long clinical notes to isolate the most relevant text before LLM inference. Evaluations on private and public EHR datasets demonstrate that this preprocessing step dramatically enhances the accuracy of smaller models for clinical phenotyping, often outperforming much larger models and proving more efficient than fine-tuning with limited data. We also discuss its potential for enabling more complex longitudinal analyses of patient trajectories within EHR data.