K2-Horizon: A Frontier Fleet of Open Source Language Models
We present K2-Horizon, a fleet of six fully open language models, from a 0.9B dense model to a 375B-A23B MoE, built for general-purpose reasoning and agentic work across math and science, software engineering, deep research, office automation, and beyond. The models lead or are competitive with strong open models of their size on agentic and coding benchmarks. On the modeling side, to strengthen reasoning, we introduce Mixture-of-Value Attention (MoVA), applying MoE to attention value projections, and softplus attention gating, a more flexible, theoretically motivated gate. Our 36B-A4B model uses both and nearly matches a 32B dense model trained on the same data, with one-seventh the active parameters. We also introduce Uno, lossless diffusion adapters that speed up decoding by up to 2.2x without a separate draft model. Our release follows the LLM360 definition of open source for transparency and reproducibility: the full development history, including intermediate checkpoints from every training stage, pre-training data, training logs, and all training, evaluation, and infrastructure code, notably our production-scale xLLM framework for dense and MoE training and the RL360 reinforcement-learning stack. Beyond the data itself, we release and document in detail every pipeline that built it, the part of model development that open releases most rarely disclose. These pipelines produced roughly 10T synthetic pre-training tokens, showing that synthetic data can be a major ingredient of pre-training at scale. Four of the models share one 22T-token pre-training corpus and recipe, revealing consistent scaling behavior and enabling controlled studies of scale and sparsity. All artifacts are available at https://ifm.ai/k2/
Authors
- Evan Dufraisse (ORCID: https://orcid.org/0000-0002-1513-914X)
- Khiem Pham
- Rifo Ahmad Genadi
- Zenan Zhai (ORCID: https://orcid.org/0000-0003-1391-6950)
- Hadi Abdine (ORCID: https://orcid.org/0009-0001-3944-4747)
- Anze Xie
- Tingxu Han (ORCID: https://orcid.org/0000-0003-1821-611X)
- Mikhail Yurochkin (ORCID: https://orcid.org/0000-0003-0153-6811)
- Charles Guille-Escuret
- Ryandito Diandaru
- Aldo Pareja
- Jalaj Bhandari (ORCID: https://orcid.org/0000-0002-7115-8986)
- Guokan Shang
- Yash Akhauri
- Andrej Leban (ORCID: https://orcid.org/0000-0003-0617-6843)
- Junu Kim (ORCID: https://orcid.org/0009-0001-9522-0756)
- Matthew Yang (ORCID: https://orcid.org/0000-0002-2117-1226)
- Yingquan Wu (ORCID: https://orcid.org/0000-0002-0211-8550)
- Shaurya Rohatgi (ORCID: https://orcid.org/0000-0001-7426-4576)
- Ming Shan Hee (ORCID: https://orcid.org/0000-0002-6328-5889)
- Chaitanya Dwivedi
- Nikhil Ranjan (ORCID: https://orcid.org/0000-0002-4982-820X)
- Shibo Hao
- Dhawal Gupta (ORCID: https://orcid.org/0000-0002-2486-866X)
- Elizaveta Rebrova (ORCID: https://orcid.org/0000-0002-4041-4238)
- Daria Soboleva (ORCID: https://orcid.org/0009-0003-2654-3767)
- Alexander Moreno (ORCID: https://orcid.org/0000-0003-3651-3222)
- Kerem Zaman
- Nurkhan Laiyk
- Varad Pimpalkhute (ORCID: https://orcid.org/0000-0002-0932-6010)
- Joel Hestness (ORCID: https://orcid.org/0000-0001-6920-0906)
- Shicheng Wen
- Omkar Pangarkar
- Subham Sekhar Sahoo
- Suqi Sun
- Bowen Tan (ORCID: https://orcid.org/0000-0003-3036-2881)
- Guowei He
- Lingjing Kong
- Weili Cao
- Rupesh Kumar Srivastava (ORCID: https://orcid.org/0000-0002-4032-4267)
- Hongli Zhan
- Adharsh Babu
- Besher Hassan
- Eric P. Xing
- Mallikarjuna Tupakula (ORCID: https://orcid.org/0009-0005-1801-9847)
- Ahmad Chamma
- Anshun Asher Zheng
- Jianshu She
- Micah Villmow
- Muzi Tao
- Jonathan Geuter
- Seungwook Han
- Aaryamonvikram Singh
- Kartik Sreenivasan
- Dani Bouch
- Po-Chun Chang (ORCID: https://orcid.org/0009-0002-9371-8582)
- Richard Fan
- Wail Bouhedja
- Kyle Chickering
- Bilal El Bouardi
- Chengqian Gao
- Haonan Li
- Xuezhou Zhang
- Yu Huang
- Richard Zhuang
- Yuqi Wang
- K2 Team
- Xuezhe Ma
- John Maggs
- Desai Fan
- Yuan Huang
- Liping Tang
- Renxi Wang
- Zhoujun Cheng
- Willis Guo
- Andrew Xing
- Huijuan Wang
- Junlin Chen
- Mukhesh Narra
- Liqun Ma
- Haolong Jia
- Chufan Shi
- Hongyi Wang
- Linghao Jin
- David Bellamy
- Amr Mohamed
- Lingjie Chen
- Hector Ren
- Zhe Zhou
- Yuekai Sun
- Shengkun Tang
- Zhengzhong Liu
- Zhenting Wang
- Yongxin Wang
- Shuanghai Yu
Publication Details
- Journal
- arXiv (Cornell University)
- Published
- 2026-11-16
- DOI
- https://doi.org/10.48550/arxiv.2611.21003
- Primary Topic
- Topic Modeling
- Type
- preprint