Deterministic Logical Time for Fault-Tolerant Distributed LLM Training

Physical clock synchronization is an under-examined infrastructure constraint in large-scale distributed LLM training. Both synchronous and asynchronous training paradigms rely on physical wall-clock alignment to decide gradient aggregation timing, parameter-version acceptance, and failed-node replacement. This creates an inherent tension: synchronous training stalls on the slowest worker, while asynchronous training trades convergence guarantees for throughput and lacks a cluster-wide, independently verifiable timing baseline. This working paper proposes a deterministic logical time axis decoupled from physical clocks, in two layers. The single-domain design anchors a training step to the logical tick boundary g_step_start mod N_train = 0, defines the aggregation deadline as a fixed logical offset g_step_start + Δn_deadline, derives a parameter-version window from the logical staleness bound to filter stale gradients, and achieves fault recovery via deterministic election and candidate-window realignment. The multi-domain extension, presented as a design draft, maintains an independent local logical axis and DSF/PSSM instance for each heterogeneous training domain, aligns domains to a global reference axis through mapping rules, and provides cross-domain aggregation anchors, domain-weight scaling, placeholder updates, QoS-driven window configuration, cross-domain quorum, and degradation mechanisms. Scope and evidence. The contribution is problem formalization, single-domain timing-correctness simulation, and an open validation protocol, not established training throughput or convergence advantages. Single-domain results come from a single-machine discrete-event simulator and cryptographic microbenchmarks. The simulation verifies timing logic only; it does not include GPU training or neural-network convergence. The multi-domain part is a design draft, argued for self-consistency only and not simulation-verified. Open call for validation. The paper defines six validation challenges, each with a failure criterion so that negative results are equally informative. Challenge A: placeholder-induced convergence deviation, executable on 8 to 16 GPUs. Challenge B: recovery bound under real fault injection, executable on 8 to 16 nodes. Challenge C: behavior-chain auditability, with third-party verification. Challenge D: cross-region beacon overhead, using two regions with measured RTT. Challenge E: cross-domain mapping and Token closure, executable on a single machine with multiple processes. Challenge F: cross-domain aggregation semantics, requiring real WAN and multi-party cooperation. Independent replication, falsification, and critique are welcome, including negative results. A suggested validation report template is included in the appendix. Licensing and patents. The document is released under CC BY 4.0. This license applies to the document copyright only and grants no patent license; patent application rights, patent rights, and all related rights are reserved by the rights holders. Academic reproduction, critique, and open-source implementation are permitted for research and validation purposes and do not constitute a patent license. Two pending patent applications cover mechanisms described in this work; the authors are primary inventors. Companion documents. Zenodo 23293794, Cryptographic-Driven Deterministic Logical Timeline: A Temporal Baseline for Machine Collaboration and AI Auditing, v2.0. Zenodo 23271917, Dual-timestamp binding and trigger-mode baseline, v3.0. This version adds a Chinese-language PDF of the same working paper. Both language versions share the same concept DOI. Correspondence: [[email protected]]

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-15
DOI
https://doi.org/10.5281/zenodo.24238249
Primary Topic
Distributed systems and fault tolerance
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Deterministic Logical Time for Fault-Tolerant Distributed LLM Training

Zhehao Wang, Mingyang Zhang, Yong Qiu
Zenodo (CERN European Organization for Nuclear Research)
Distributed systems and fault tolerance
article

Deterministic Logical Time for Fault-Tolerant Distributed LLM Training

Zhehao Wang, Mingyang Zhang, Yong Qiu
article en

Abstract

Physical clock synchronization is an under-examined infrastructure constraint in large-scale distributed LLM training. Both synchronous and asynchronous training paradigms rely on physical wall-clock alignment to decide gradient aggregation timing, parameter-version acceptance, and failed-node replacement. This creates an inherent tension: synchronous training stalls on the slowest worker, while asynchronous training trades convergence guarantees for throughput and lacks a cluster-wide, independently verifiable timing baseline. This working paper proposes a deterministic logical time axis decoupled from physical clocks, in two layers. The single-domain design anchors a training step to the logical tick boundary g_step_start mod N_train = 0, defines the aggregation deadline as a fixed logical offset g_step_start + Δn_deadline, derives a parameter-version window from the logical staleness bound to filter stale gradients, and achieves fault recovery via deterministic election and candidate-window realignment. The multi-domain extension, presented as a design draft, maintains an independent local logical axis and DSF/PSSM instance for each heterogeneous training domain, aligns domains to a global reference axis through mapping rules, and provides cross-domain aggregation anchors, domain-weight scaling, placeholder updates, QoS-driven window configuration, cross-domain quorum, and degradation mechanisms. Scope and evidence. The contribution is problem formalization, single-domain timing-correctness simulation, and an open validation protocol, not established training throughput or convergence advantages. Single-domain results come from a single-machine discrete-event simulator and cryptographic microbenchmarks. The simulation verifies timing logic only; it does not include GPU training or neural-network convergence. The multi-domain part is a design draft, argued for self-consistency only and not simulation-verified. Open call for validation. The paper defines six validation challenges, each with a failure criterion so that negative results are equally informative. Challenge A: placeholder-induced convergence deviation, executable on 8 to 16 GPUs. Challenge B: recovery bound under real fault injection, executable on 8 to 16 nodes. Challenge C: behavior-chain auditability, with third-party verification. Challenge D: cross-region beacon overhead, using two regions with measured RTT. Challenge E: cross-domain mapping and Token closure, executable on a single machine with multiple processes. Challenge F: cross-domain aggregation semantics, requiring real WAN and multi-party cooperation. Independent replication, falsification, and critique are welcome, including negative results. A suggested validation report template is included in the appendix. Licensing and patents. The document is released under CC BY 4.0. This license applies to the document copyright only and grants no patent license; patent application rights, patent rights, and all related rights are reserved by the rights holders. Academic reproduction, critique, and open-source implementation are permitted for research and validation purposes and do not constitute a patent license. Two pending patent applications cover mechanisms described in this work; the authors are primary inventors. Companion documents. Zenodo 23293794, Cryptographic-Driven Deterministic Logical Timeline: A Temporal Baseline for Machine Collaboration and AI Auditing, v2.0. Zenodo 23271917, Dual-timestamp binding and trigger-mode baseline, v3.0. This version adds a Chinese-language PDF of the same working paper. Both language versions share the same concept DOI. Correspondence: [[email protected]]

Zenodo (CERN European Organization for Nuclear Research)
Distributed systems and fault tolerance
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon