LUNA-FM: a subword foundation model for long non-coding RNAs for functional inference
RNA foundation models (FMs) have shown promise for RNA identification and functional annotation. However, many use single-nucleotide tokenization, which produces long input sequences and can increase computational costs when modelling long RNAs, including long non-coding RNAs (lncRNAs). To address these limitations, we introduce LUNA-FM (Length-efficient sUbword-based Nucleic Acid Foundation Model), an RNA language model pretrained on approximately 28 million redundancy-filtered ncRNA sequences. LUNA-FM combines subword tokenization with a long context transformer, enabling efficient modelling of long RNA transcripts. We systematically evaluate tokenization strategies and vocabulary sizes to investigate the trade-offs between sequence compression, representation granularity, computational efficiency, and downstream performance. Across multiple RNA prediction tasks, including transcript classification, subcellular localization, and RNA-protein interaction prediction, LUNA-FM achieved competitive performance compared with existing RNA FMs while reducing effective input length and fine-tuning cost. We also found that frequency-based subword tokenization generally produced more stable and effective representations than likelihood-based approaches, although the optimal vocabulary size remained task dependent. Training-set motif discovery identified candidate motifs enriched relative to matched low-attribution and dinucleotide-shuffled controls of preeclampsia-associated lncRNAs. Overall, our findings highlight the task-dependent effects of subword tokenization and vocabulary size, extend RNA FM evaluation to longer transcript inputs, and establish LUNA-FM as a competitive model with favourable recorded fine-tuning times.
Authors
- Naima Vahab
- Sonika Tyagi (ORCID: https://orcid.org/0000-0003-0181-6258)
- Tabinda Sarwar (ORCID: https://orcid.org/0000-0001-7313-5350)
- Estrid He
Institutions
- RMIT University (AU)
Publication Details
- Journal
- RNA Biology
- Published
- 2026-11-13
- DOI
- https://doi.org/10.1080/15476286.2026.2763191
- Primary Topic
- Cancer-related molecular mechanisms research
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- RMIT University