AtlasBERT: a modernized encoder for Turkish general and clinical language inference

Decoder-only language models have advanced rapidly through architectural innovations such as new normalization methods, gated activations, and efficient attention mechanisms. However, encoder-only architectures, which are excellent at capturing contextual information for tasks like sequence classification and Natural Language Inference (NLI), have largely retained the original BERT family’s design principles. This gap creates particular challenges for low-resource languages like Turkish, where the scarcity of domain-specific pretrained encoders limits progress in specialized areas such as clinical natural language processing (NLP). Although English biomedical and clinical encoders have demonstrated that domain-adaptive pretraining improves biomedical and clinical NLP performance, comparable resources for Turkish remain underdeveloped. To address this, we present Atlas bidirectional encoder representations from transformers (BERT), a modernized BERT-style encoder that integrates recent innovations from both encoder and decoder models for Turkish general and clinical language inference. Inspired by models like ModernBERT, NeoBERT, and EuroBERT, our architecture employs pre-RMSNorm, SwiGLU feed-forward layers, and PyTorch FlexAttention with document-aware masking to extend the context window to 2,048 tokens while preserving document boundaries. Additionally, to address Turkish’s complex agglutinative morphology, we employ a SentencePiece Unigram tokenizer for subword segmentation. AtlasBERT is trained in two stages, first on 35.68 billion Turkish tokens from FineWeb-2, then with continued domain-adaptive pretraining on 791 million tokens of Turkish biomedical text. AtlasBERT achieves 87.82% mean test accuracy on Turkish Stanford natural language inference (SNLI), compared with BERTurk’s 87.03%, while in the clinical domain BioAtlasBERT reaches 77.80% on TurkMedNLI against a BioBERTurk baseline of 75.28%. These results show that combining modern encoder components with Turkish-tailored tokenization and two-stage pretraining provides a competitive foundation for Turkish general and clinical NLI.

Authors

Institutions

Publication Details

Journal
PeerJ Computer Science
Published
2026-11-19
DOI
https://doi.org/10.7717/peerj-cs.4173
Primary Topic
Topic Modeling
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

AtlasBERT: a modernized encoder for Turkish general and clinical language inference

Belgin Ergenç Bostanoğlu, Fatih Soygazi, Ískender Ülgen Oğul
PeerJ Computer Science
Topic Modeling
article

AtlasBERT: a modernized encoder for Turkish general and clinical language inference

Belgin Ergenç Bostanoğlu, Fatih Soygazi, Ískender Ülgen Oğul
article en

Abstract

Decoder-only language models have advanced rapidly through architectural innovations such as new normalization methods, gated activations, and efficient attention mechanisms. However, encoder-only architectures, which are excellent at capturing contextual information for tasks like sequence classification and Natural Language Inference (NLI), have largely retained the original BERT family’s design principles. This gap creates particular challenges for low-resource languages like Turkish, where the scarcity of domain-specific pretrained encoders limits progress in specialized areas such as clinical natural language processing (NLP). Although English biomedical and clinical encoders have demonstrated that domain-adaptive pretraining improves biomedical and clinical NLP performance, comparable resources for Turkish remain underdeveloped. To address this, we present Atlas bidirectional encoder representations from transformers (BERT), a modernized BERT-style encoder that integrates recent innovations from both encoder and decoder models for Turkish general and clinical language inference. Inspired by models like ModernBERT, NeoBERT, and EuroBERT, our architecture employs pre-RMSNorm, SwiGLU feed-forward layers, and PyTorch FlexAttention with document-aware masking to extend the context window to 2,048 tokens while preserving document boundaries. Additionally, to address Turkish’s complex agglutinative morphology, we employ a SentencePiece Unigram tokenizer for subword segmentation. AtlasBERT is trained in two stages, first on 35.68 billion Turkish tokens from FineWeb-2, then with continued domain-adaptive pretraining on 791 million tokens of Turkish biomedical text. AtlasBERT achieves 87.82% mean test accuracy on Turkish Stanford natural language inference (SNLI), compared with BERTurk’s 87.03%, while in the clinical domain BioAtlasBERT reaches 77.80% on TurkMedNLI against a BioBERTurk baseline of 75.28%. These results show that combining modern encoder components with Turkish-tailored tokenization and two-stage pretraining provides a competitive foundation for Turkish general and clinical NLI.

PeerJ Computer ScienceVol. 12
Izmir Institute of Technology (TR), Adnan Menderes University (TR)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon