The impact of tokenizer selection in genomic language models
Detailed results of all benchmarking experiments are available in https://github.com/leannmlindsey/DNAtokenization. Training datasets and pretrained models are available at https://huggingface.co/datasets/leannmlindsey. Datasets and processing scripts are available at doi: 10.5281/zenodo.16287401 and doi: 10.5281/zenodo.16287130.
Authors
- LeAnn M. Lindsey (ORCID: https://orcid.org/0000-0002-0515-0118)
- Hari Sundar (ORCID: https://orcid.org/0000-0001-9001-5107)
- Keith Dufault‐Thompson (ORCID: https://orcid.org/0000-0002-0991-2255)
- Xiaofang Jiang (ORCID: https://orcid.org/0000-0002-0955-8284)
- Anne Jeannette Blaschke (ORCID: https://orcid.org/0000-0002-2968-3007)
- W. Zac Stephens (ORCID: https://orcid.org/0000-0002-8072-9023)
- Nicole L. Pershing (ORCID: https://orcid.org/0000-0002-5399-5668)
- A K M Rubaiyat Reza Habib
Institutions
- Tufts University (US)
- National Institutes of Health (US)
- University of Utah (US)
- United States National Library of Medicine (US)
Publication Details
- Journal
- Bioinformatics
- Published
- 2025-08-18
- DOI
- https://doi.org/10.1093/bioinformatics/btaf456
- Citations
- 3
- Primary Topic
- Topic Modeling
- Type
- article
- Field-Weighted Citation Impact
- 7.44
Funders
- National Science Foundation
- University of Utah
- National Centre for Supercomputing Applications
- National Institutes of Health
- U.S. National Library of Medicine
- Office of Advanced Cyberinfrastructure
- National Center for Advancing Translational Sciences
- NHLBI Division of Intramural Research