Integrating Unsupervised Clustering with Transformer-Based Models for De Novo Cell Population Identification

Single-cell transcriptome sequencing has been widely applied across the life sciences. It enables accurate and robust identification of cell types in heterogeneous biological tissues through unsupervised clustering, shallow learning, and deep learning. While deep learning does not outperform shallow learning in classification performance, it streamlines cell type identification by eliminating cumbersome preprocessing procedures. However, the availability of training datasets with comprehensive cell type annotations is always limited, so new datasets inevitably contain cell populations that are not pre-defined in the training data. To address this limitation, we integrated unsupervised clustering with transformer-based models to identify cell populations unrecognized by classical methods. Specifically, we developed the Nankai GeneRank Classifier by fine-tuning the pre-trained Geneformer-V1 model, and then clustered cells based on its softmax-normalized class probability matrix (SCPM). Applied to a public colorectal cancer dataset, the new method, named SCPM-based clustering, resolved cell populations that cannot be unambiguously identified by classical methods alone and led to the discovery of novel candidate markers of colorectal cancer stem cells, demonstrating its utility.

Authors

Institutions

Publication Details

Journal
bioRxiv (Cold Spring Harbor Laboratory)
Published
2026-11-16
DOI
https://doi.org/10.64898/2026.11.12.766182
Primary Topic
Single-cell and spatial transcriptomics
Type
preprint

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Integrating Unsupervised Clustering with Transformer-Based Models for De Novo Cell Population Identification

Guofeng Xu, Yanting Guo, Jia Chang, GuangYou Duan et al.
2 links
bioRxiv (Cold Spring Harbor Laboratory)
Single-cell and spatial transcriptomics
preprint

Integrating Unsupervised Clustering with Transformer-Based Models for De Novo Cell Population Identification

Guofeng Xu, Yanting Guo, Jia Chang, GuangYou Duan, Chang Liu, Yangguang Han, Shan Gao, Xin Li
preprint en

Abstract

Single-cell transcriptome sequencing has been widely applied across the life sciences. It enables accurate and robust identification of cell types in heterogeneous biological tissues through unsupervised clustering, shallow learning, and deep learning. While deep learning does not outperform shallow learning in classification performance, it streamlines cell type identification by eliminating cumbersome preprocessing procedures. However, the availability of training datasets with comprehensive cell type annotations is always limited, so new datasets inevitably contain cell populations that are not pre-defined in the training data. To address this limitation, we integrated unsupervised clustering with transformer-based models to identify cell populations unrecognized by classical methods. Specifically, we developed the Nankai GeneRank Classifier by fine-tuning the pre-trained Geneformer-V1 model, and then clustered cells based on its softmax-normalized class probability matrix (SCPM). Applied to a public colorectal cancer dataset, the new method, named SCPM-based clustering, resolved cell populations that cannot be unambiguously identified by classical methods alone and led to the discovery of novel candidate markers of colorectal cancer stem cells, demonstrating its utility.

bioRxiv (Cold Spring Harbor Laboratory)
Yunnan University (CN), Nankai University (CN), Qilu Normal University (CN)
Nankai University
Single-cell and spatial transcriptomics
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon