Histopathobiome – integrating histopathology and microbiome data via multimodal deep learning

We introduce Histopathobiome, a term representing the integration of histopathology and microbiome data to explore tissue-microbe interactions. Using a dataset of colon biopsy whole-slide images paired with microbiota composition samples, we assess the benefits of combining these modalities to distinguish patients with inflammatory bowel disease (IBD) subtype – ulcerative colitis (UC) from non-IBD controls. Initially, we evaluate the unimodal performance of state-of-the-art algorithms using vectors representing bacterial species abundances or histopathology slide-level embeddings. We compare single-modality models with bimodal networks with various fusion strategies. Our results prove that histopathology and microbiome data are complementary in UC classification. By demonstrating improved performance over single-modality approaches, we prove that bimodal deep learning models can be used to learn meaningful and interpretable cross-modal tissue-microbe patterns.