Beyond Pairwise Task-Order Models: A Seed-Aware Spectral Audit of Continual-Learning Landscapes

Accepted at the workshop "Continual Learning in the Era of Foundation Models and Embodied Agents" (CL4FMAgents), NeurIPS 2026. Task order can change catastrophic forgetting, yet a good position-effect or transition model does not establish that pairwise structure is exactly sufficient. We introduce a seed-aware spectral audit for complete task-order experiments. The audit treats a scalar response over all task permutations as a function on the symmetric group, decomposes it by assignment degree, removes the diagonal seed-variation term when estimating reproducible signal, and jointly tests the entire higher-degree complement. In ten-seed MLP experiments containing every order, degree greater than two accounts for 16.1% (approximate 95% CI [12.1%, 20.1%]) of estimated nonconstant signal at K = 5 and 19.4% ([11.4%, 27.4%]) at K = 6. Conditional on independent seed-level central symmetry, both whole-seed randomization tests reach the ten-seed resolution limit, p = 1/512. A separate, explicitly post-selected analysis uses a complete K = 7 landscape to illustrate how the audit localizes the residual within degree three. The result is a reusable distinction between low-degree approximation strength and exact structural sufficiency in continual-learning order effects. Code, complete task-order landscapes (K = 5, 6, 7) and claim-verification scripts: https://github.com/szymonghost/Task-order-spectral-auditOpenReview: https://openreview.net/forum?id=DFlWJgpxal