112 Multimodal AI biomarker fusing radiology, pathology, and molecular information for immune checkpoint inhibitor (ICI) response prediction in lung adenocarcinoma (LUAD)

<h3>Background</h3> LUAD, the most common subtype of NSCLC, has benefited from ICIs, but current biomarkers (e.g. PD-L1) fail to fully identify responders. Application of AI in routinely collected radiology images and tumor tissue samples has substantially advanced, showing promise to improve guidance of ICI. CT scans offer macroscale phenotypic data of lesions and their surrounding habitat, while hematoxylin and eosin (H&amp;E) stained whole slide images (WSIs) provide detailed microstructural information on the tumor-immune environment. However, unimodal biomarkers may not be sufficiently prognostic or predictive individually. In this study, we developed a multimodal AI biomarker system integrating radiology, pathology and molecular data for holistic ICI response prediction and validated on an external test set. <h3>Methods</h3> A multimodal AI system integrating CT scans, H&amp;E stained WSIs, and molecular biomarker data was built to predict objective response to ICI monotherapy in LUAD using available multimodal data. First, separate unimodal predictors were independently trained with predominantly late stage NSCLC patients. Firstly we trained a Radiomic neural network using features extracted from within the tumor and its vasculature on baseline CT scans (N=252). Secondly, a Pathomic neural network was trained with features of immune cells and their interaction with tumors on pre-treatment WSIs of NSCLC patients (N=57). Finally a molecular classifier using PD-L1 TPS score was trained as well (N=189). A multimodal fusion system was then developed to combine predictions from available data modalities to generate a fusion response score, with capability of handling missing data types. The fusion model was validated on a testing set of late stage LUAD patients from an external institute (N=58). <h3>Results</h3> In the external test set, 98%, 50%, and 79% had radiology, pathology, and PDL1 data available, respectively. Unimodal radiomic and pathomics models met or exceeded the performance of PD-L1 (binarized with TPS&gt;=50% threshold) (figure 1). The multimodal AI predictor significantly exceeded any single modality (AUC=0.87), with strongest performance in the subset of patients with all three modalities available (AUC=0.90, n=25): a 20-29% improvement in AUC over any unimodal model. Performance of the multimodal system remained robust even for patients with one missing data type (AUC=0.81, n=24). <h3>Conclusions</h3> The multimodal AI system integrated models trained with imaging data with PDL1, showing added value for ICI response prediction compared to using PDL1 alone for LUAD patients. While its performance was most enhanced with full multimodal data, the system’s ability to accurately predict response despite missing modalities can greatly improve its clinical applicability.