LUNA-FM: a subword foundation model for long non-coding RNAs for functional inference

RNA foundation models (FMs) have shown promise for RNA identification and functional annotation. However, many use single-nucleotide tokenization, which produces long input sequences and can increase computational costs when modelling long RNAs, including long non-coding RNAs (lncRNAs). To address these limitations, we introduce LUNA-FM (Length-efficient sUbword-based Nucleic Acid Foundation Model), an RNA language model pretrained on approximately 28 million redundancy-filtered ncRNA sequences. LUNA-FM combines subword tokenization with a long context transformer, enabling efficient modelling of long RNA transcripts. We systematically evaluate tokenization strategies and vocabulary sizes to investigate the trade-offs between sequence compression, representation granularity, computational efficiency, and downstream performance. Across multiple RNA prediction tasks, including transcript classification, subcellular localization, and RNA-protein interaction prediction, LUNA-FM achieved competitive performance compared with existing RNA FMs while reducing effective input length and fine-tuning cost. We also found that frequency-based subword tokenization generally produced more stable and effective representations than likelihood-based approaches, although the optimal vocabulary size remained task dependent. Training-set motif discovery identified candidate motifs enriched relative to matched low-attribution and dinucleotide-shuffled controls of preeclampsia-associated lncRNAs. Overall, our findings highlight the task-dependent effects of subword tokenization and vocabulary size, extend RNA FM evaluation to longer transcript inputs, and establish LUNA-FM as a competitive model with favourable recorded fine-tuning times.

Authors

Institutions

Publication Details

Journal
RNA Biology
Published
2026-11-13
DOI
https://doi.org/10.1080/15476286.2026.2763191
Primary Topic
Cancer-related molecular mechanisms research
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

LUNA-FM: a subword foundation model for long non-coding RNAs for functional inference

Naima Vahab, Sonika Tyagi, Tabinda Sarwar, Estrid He
5 links
RNA Biology
Cancer-related molecular mechanisms research
article

LUNA-FM: a subword foundation model for long non-coding RNAs for functional inference

Naima Vahab, Sonika Tyagi, Tabinda Sarwar, Estrid He
article en

Abstract

RNA foundation models (FMs) have shown promise for RNA identification and functional annotation. However, many use single-nucleotide tokenization, which produces long input sequences and can increase computational costs when modelling long RNAs, including long non-coding RNAs (lncRNAs). To address these limitations, we introduce LUNA-FM (Length-efficient sUbword-based Nucleic Acid Foundation Model), an RNA language model pretrained on approximately 28 million redundancy-filtered ncRNA sequences. LUNA-FM combines subword tokenization with a long context transformer, enabling efficient modelling of long RNA transcripts. We systematically evaluate tokenization strategies and vocabulary sizes to investigate the trade-offs between sequence compression, representation granularity, computational efficiency, and downstream performance. Across multiple RNA prediction tasks, including transcript classification, subcellular localization, and RNA-protein interaction prediction, LUNA-FM achieved competitive performance compared with existing RNA FMs while reducing effective input length and fine-tuning cost. We also found that frequency-based subword tokenization generally produced more stable and effective representations than likelihood-based approaches, although the optimal vocabulary size remained task dependent. Training-set motif discovery identified candidate motifs enriched relative to matched low-attribution and dinucleotide-shuffled controls of preeclampsia-associated lncRNAs. Overall, our findings highlight the task-dependent effects of subword tokenization and vocabulary size, extend RNA FM evaluation to longer transcript inputs, and establish LUNA-FM as a competitive model with favourable recorded fine-tuning times.

RNA Biology
RMIT University (AU)
RMIT University
Cancer-related molecular mechanisms research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon