Block-Based Historical Curriculum for Small Language Model Training (CMF-180)

This paper presents an experiment on the influence of exposure order on the formation of internal representations in a small language model trained from scratch on a Greco-Latin and documentary corpus temporally bounded by 180 AD. The experiment tests whether learning is determined not only by the content of the data, but also by the temporal and structural regime in which those data are presented to the model. Three training regimes were compared under controlled conditions: global random shuffling, rigid chronology, and a block-based historical curriculum with internal shuffling. The corpus, tokenizer, model architecture, validation set, hyperparameters, and number of optimization steps were held constant. The only experimental variable was the order of exposure. Preliminary results show that rigid chronological ordering degrades performance and produces high anisotropy in the embedding space, while the block-based historical curriculum achieves the best validation loss among the three regimes. Additional analyses of anisotropy, cross-model similarity, semantic neighbor overlap, neighbor fragmentation, and language consistency suggest that historical order becomes useful only when applied at the level of epistemic blocks, not as a rigid sequence of documents or chunks. The paper proposes the term “block-based historical curriculum” for this exposure regime and interprets it as a metastable form of learning: ordered enough to preserve the historical direction of knowledge, but locally diverse enough to avoid representational collapse. The result is preliminary and limited to the CMF-180 setup, but it suggests that, in historically structured domains, model training should be viewed not only as data accumulation, but as the design of an exposure regime.