LUNA-FM: a subword foundation model for long non-coding RNAs for functional inference

RNA foundation models (FMs) have shown promise for RNA identification and functional annotation. However, many use single-nucleotide tokenization, which produces long input sequences and can increase computational costs when modelling long RNAs, including long non-coding RNAs (lncRNAs). To address these limitations, we introduce LUNA-FM (Length-efficient sUbword-based Nucleic Acid Foundation Model), an RNA language model pretrained on approximately 28 million redundancy-filtered ncRNA sequences. LUNA-FM combines subword tokenization with a long context transformer, enabling efficient modelling of long RNA transcripts. We systematically evaluate tokenization strategies and vocabulary sizes to investigate the trade-offs between sequence compression, representation granularity, computational efficiency, and downstream performance. Across multiple RNA prediction tasks, including transcript classification, subcellular localization, and RNA-protein interaction prediction, LUNA-FM achieved competitive performance compared with existing RNA FMs while reducing effective input length and fine-tuning cost. We also found that frequency-based subword tokenization generally produced more stable and effective representations than likelihood-based approaches, although the optimal vocabulary size remained task dependent. Training-set motif discovery identified candidate motifs enriched relative to matched low-attribution and dinucleotide-shuffled controls of preeclampsia-associated lncRNAs. Overall, our findings highlight the task-dependent effects of subword tokenization and vocabulary size, extend RNA FM evaluation to longer transcript inputs, and establish LUNA-FM as a competitive model with favourable recorded fine-tuning times. LUNA-FM combines a long-context transformer architecture with subword tokenization for RNA modelling, incorporating alternating attention, improved activation functions, and padding-aware computation to support efficient representation learning across diverse RNA tasks.Across multiple downstream evaluations, LUNA-FM demonstrates competitive performance while maintaining favourable computational efficiency, particularly for long RNA transcripts and lncRNA-related tasks.Systematic tokenization analysis shows that frequency-based BPE tokenization generally outperformed probabilistic Unigram segmentation across the evaluated RNA tasks, including transcript-level classification and splice-site prediction. These results highlight the influence of subword construction strategy on downstream performance.Vocabulary size influences both sequence compression and downstream performance, highlighting a trade-off between preserving local nucleotide information and modelling broader contextual dependencies.Interpretability analysis using Integrated Gradients identified motif-enriched regions associated with model predictions, providing candidates for independent testing of predictive and biological relevance.Subword tokenization reduces the number of tokens required to represent RNA sequences. LUNA-FM achieved competitive predictive performance and generally shorter recorded fine-tuning times under the selected configurations, while some nucleotide-level baseline runs encountered GPU memory constraints. LUNA-FM combines a long-context transformer architecture with subword tokenization for RNA modelling, incorporating alternating attention, improved activation functions, and padding-aware computation to support efficient representation learning across diverse RNA tasks. Across multiple downstream evaluations, LUNA-FM demonstrates competitive performance while maintaining favourable computational efficiency, particularly for long RNA transcripts and lncRNA-related tasks. Systematic tokenization analysis shows that frequency-based BPE tokenization generally outperformed probabilistic Unigram segmentation across the evaluated RNA tasks, including transcript-level classification and splice-site prediction. These results highlight the influence of subword construction strategy on downstream performance. Vocabulary size influences both sequence compression and downstream performance, highlighting a trade-off between preserving local nucleotide information and modelling broader contextual dependencies. Interpretability analysis using Integrated Gradients identified motif-enriched regions associated with model predictions, providing candidates for independent testing of predictive and biological relevance. Subword tokenization reduces the number of tokens required to represent RNA sequences. LUNA-FM achieved competitive predictive performance and generally shorter recorded fine-tuning times under the selected configurations, while some nucleotide-level baseline runs encountered GPU memory constraints.

Authors

Publication Details

Journal
Figshare
Published
2026-11-13
DOI
https://doi.org/10.6084/m9.figshare.35002882
Primary Topic
RNA and protein synthesis mechanisms
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

LUNA-FM: a subword foundation model for long non-coding RNAs for functional inference

Naima Vahab, Tabinda Sarwar, Sonika Tyagi, Estrid He
7 links
Figshare
RNA and protein synthesis mechanisms
article

LUNA-FM: a subword foundation model for long non-coding RNAs for functional inference

Naima Vahab, Tabinda Sarwar, Sonika Tyagi, Estrid He
article en

Abstract

RNA foundation models (FMs) have shown promise for RNA identification and functional annotation. However, many use single-nucleotide tokenization, which produces long input sequences and can increase computational costs when modelling long RNAs, including long non-coding RNAs (lncRNAs). To address these limitations, we introduce LUNA-FM (Length-efficient sUbword-based Nucleic Acid Foundation Model), an RNA language model pretrained on approximately 28 million redundancy-filtered ncRNA sequences. LUNA-FM combines subword tokenization with a long context transformer, enabling efficient modelling of long RNA transcripts. We systematically evaluate tokenization strategies and vocabulary sizes to investigate the trade-offs between sequence compression, representation granularity, computational efficiency, and downstream performance. Across multiple RNA prediction tasks, including transcript classification, subcellular localization, and RNA-protein interaction prediction, LUNA-FM achieved competitive performance compared with existing RNA FMs while reducing effective input length and fine-tuning cost. We also found that frequency-based subword tokenization generally produced more stable and effective representations than likelihood-based approaches, although the optimal vocabulary size remained task dependent. Training-set motif discovery identified candidate motifs enriched relative to matched low-attribution and dinucleotide-shuffled controls of preeclampsia-associated lncRNAs. Overall, our findings highlight the task-dependent effects of subword tokenization and vocabulary size, extend RNA FM evaluation to longer transcript inputs, and establish LUNA-FM as a competitive model with favourable recorded fine-tuning times. LUNA-FM combines a long-context transformer architecture with subword tokenization for RNA modelling, incorporating alternating attention, improved activation functions, and padding-aware computation to support efficient representation learning across diverse RNA tasks.Across multiple downstream evaluations, LUNA-FM demonstrates competitive performance while maintaining favourable computational efficiency, particularly for long RNA transcripts and lncRNA-related tasks.Systematic tokenization analysis shows that frequency-based BPE tokenization generally outperformed probabilistic Unigram segmentation across the evaluated RNA tasks, including transcript-level classification and splice-site prediction. These results highlight the influence of subword construction strategy on downstream performance.Vocabulary size influences both sequence compression and downstream performance, highlighting a trade-off between preserving local nucleotide information and modelling broader contextual dependencies.Interpretability analysis using Integrated Gradients identified motif-enriched regions associated with model predictions, providing candidates for independent testing of predictive and biological relevance.Subword tokenization reduces the number of tokens required to represent RNA sequences. LUNA-FM achieved competitive predictive performance and generally shorter recorded fine-tuning times under the selected configurations, while some nucleotide-level baseline runs encountered GPU memory constraints. LUNA-FM combines a long-context transformer architecture with subword tokenization for RNA modelling, incorporating alternating attention, improved activation functions, and padding-aware computation to support efficient representation learning across diverse RNA tasks. Across multiple downstream evaluations, LUNA-FM demonstrates competitive performance while maintaining favourable computational efficiency, particularly for long RNA transcripts and lncRNA-related tasks. Systematic tokenization analysis shows that frequency-based BPE tokenization generally outperformed probabilistic Unigram segmentation across the evaluated RNA tasks, including transcript-level classification and splice-site prediction. These results highlight the influence of subword construction strategy on downstream performance. Vocabulary size influences both sequence compression and downstream performance, highlighting a trade-off between preserving local nucleotide information and modelling broader contextual dependencies. Interpretability analysis using Integrated Gradients identified motif-enriched regions associated with model predictions, providing candidates for independent testing of predictive and biological relevance. Subword tokenization reduces the number of tokens required to represent RNA sequences. LUNA-FM achieved competitive predictive performance and generally shorter recorded fine-tuning times under the selected configurations, while some nucleotide-level baseline runs encountered GPU memory constraints.

Figshare
RNA and protein synthesis mechanisms
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon