Proportionate in Volume, Distinctive in Composition: Persian in the Open Model and Dataset Ecosystem of Hugging Face

Purpose. Open model hubs have become the main infrastructure through which language technologies are shared, yet there is no quantitative audit of how well Persian is served relative to comparable languages. We ask whether Persian resources are scarce relative to the language's digital footprint, how concentrated their production and use are, and how their composition differs from peers. Design and methods. We took a full census of all models and datasets on Hugging Face tagged with each of 20 languages (Persian and 19 comparison languages; 5,052 Persian-tagged models and 1,452 datasets), retrieved on 1 October 2026 through the public API, and combined it with Wikipedia size as a measure of digital footprint. We separate language-dedicated resources (a single language tag) from multilingual ones, estimate how many dedicated models a language's footprint predicts using log–log regression on the other 19 languages (with prediction intervals), and compare concentration of authors and downloads, task mix, licensing, base-model dependence and creation timing. Findings. Persian had 913 dedicated models (18.1% of Persian-tagged models) and 569 dedicated datasets. Relative to the footprint-based prediction (1,593 models) Persian was at 0.57 of the expected volume (95% prediction interval 0.13–2.53), and ratios under alternative footprint measures and definitions ranged from 0.42 to 0.75, always with intervals that include parity; the data therefore do not show a statistically distinguishable shortfall. Production and use were more concentrated than for peers: five authors made 29.6% of the dedicated models (peer median 19.3%) and three authors accounted for 94.5% of dedicated-model downloads (peer median 77.8%). Multilingual models held 81.7% of Persian-tagged models and 99.3% of downloads, which is typical (peer medians 86.8% and 98.5%). Composition was distinctive: speech recognition models were 29.6% of dedicated models (peers 14.7%), question answering 8.4% (peers 1.7%), and text generation 12.4% (peers 21.8%), and generative models were the 17th of 20 languages in share. Fewer dedicated Persian models than peers were created since 2023 (85.2% versus 89.6%), and the share of text generation among them was 13.5% versus 18.6%. Originality. This is, to our knowledge, the first language-specific quantitative audit of Persian on a major model hub; it shows that the practical problem is concentration, generative under-provision and dependence on a few maintainers rather than a raw shortage of artefacts, and it documents measurement pitfalls of hub metadata.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-18
DOI
https://doi.org/10.5281/zenodo.23448550
Primary Topic
Natural Language Processing Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Proportionate in Volume, Distinctive in Composition: Persian in the Open Model and Dataset Ecosystem of Hugging Face

P. Ghorbani
Zenodo (CERN European Organization for Nuclear Research)
Natural Language Processing Techniques
preprint

Proportionate in Volume, Distinctive in Composition: Persian in the Open Model and Dataset Ecosystem of Hugging Face

P. Ghorbani
preprint en

Abstract

Purpose. Open model hubs have become the main infrastructure through which language technologies are shared, yet there is no quantitative audit of how well Persian is served relative to comparable languages. We ask whether Persian resources are scarce relative to the language's digital footprint, how concentrated their production and use are, and how their composition differs from peers. Design and methods. We took a full census of all models and datasets on Hugging Face tagged with each of 20 languages (Persian and 19 comparison languages; 5,052 Persian-tagged models and 1,452 datasets), retrieved on 1 October 2026 through the public API, and combined it with Wikipedia size as a measure of digital footprint. We separate language-dedicated resources (a single language tag) from multilingual ones, estimate how many dedicated models a language's footprint predicts using log–log regression on the other 19 languages (with prediction intervals), and compare concentration of authors and downloads, task mix, licensing, base-model dependence and creation timing. Findings. Persian had 913 dedicated models (18.1% of Persian-tagged models) and 569 dedicated datasets. Relative to the footprint-based prediction (1,593 models) Persian was at 0.57 of the expected volume (95% prediction interval 0.13–2.53), and ratios under alternative footprint measures and definitions ranged from 0.42 to 0.75, always with intervals that include parity; the data therefore do not show a statistically distinguishable shortfall. Production and use were more concentrated than for peers: five authors made 29.6% of the dedicated models (peer median 19.3%) and three authors accounted for 94.5% of dedicated-model downloads (peer median 77.8%). Multilingual models held 81.7% of Persian-tagged models and 99.3% of downloads, which is typical (peer medians 86.8% and 98.5%). Composition was distinctive: speech recognition models were 29.6% of dedicated models (peers 14.7%), question answering 8.4% (peers 1.7%), and text generation 12.4% (peers 21.8%), and generative models were the 17th of 20 languages in share. Fewer dedicated Persian models than peers were created since 2023 (85.2% versus 89.6%), and the share of text generation among them was 13.5% versus 18.6%. Originality. This is, to our knowledge, the first language-specific quantitative audit of Persian on a major model hub; it shows that the practical problem is concentration, generative under-provision and dependence on a few maintainers rather than a raw shortage of artefacts, and it documents measurement pitfalls of hub metadata.

Zenodo (CERN European Organization for Nuclear Research)
Shahid Beheshti University (IR)
Natural Language Processing Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.