FViT: A Foundation-Tokenized Vision Transformer with Retrieval-Augmented Multi-Scale Ensembling for Deep Learning-Based Spatial Gene Expression Prediction from Histopathology

Spatial transcriptomics links gene expression to tissue morphology but is too costly for routine clinical use, motivating prediction of expression from hematoxylin-and-eosin (H&E) images. FViT (Foundation-tokenized Vision Transformer) tokenizes a spot and its k nearest neighbours with the frozen foundation model Phikon, fuses the tokens with a small trainable Transformer with a relative-position embedding, predicts expression directly and by k-nearest-neighbour retrieval of measured training expression, and combines three neighbourhood sizes (k = 8, 16, 24) in a weighted ensemble. We evaluated FViT on two public cohorts, Her2ST (HER2-positive breast cancer, eight patients) and cutaneous squamous cell carcinoma (cSCC, four patients), with leave-one-patient-out validation in which the gene panel, the retrieval parameters and the ensemble weights are selected without the evaluated patient. The ensemble reached a Pearson correlation of 0.180 (Her2ST) and 0.213 (cSCC) on the 50 most highly expressed genes (HEGs), but only 0.067 and 0.059 over all 1000 panel genes. Controls showed that neighbourhood fusion with relative position raised the HEG of direct prediction by +0.038 (Her2ST) and +0.045 (cSCC) over a centre-only model, that retrieval changed it by +0.002 to +0.004 on Her2ST and +0.007 to +0.010 on cSCC, and that the multi-scale ensemble differed from an ensemble of three seeds of a single neighbourhood size by −0.003 to +0.008. Under the same folds and panels, the released mclSTExp model (one seed, released settings) reached 0.141 and 0.177. In a section-level protocol, 99% of the retrieved weight came from other sections of the query’s own patient, and the ensemble’s retrieval gain fell from +0.013 to −0.002 when these sections were masked. Published results use section-level protocols and other gene panels, so they are not directly comparable. With eight and four patients, the findings are exploratory.