Hierarchical JEPA for Long Scientific Documents: Predicting Section and Abstract Embeddings

Long documents contain information at several time scales, from sentences to sections. A single-level encoder over the full text is expensive and loses high-level structure. We propose a hierarchical joint-embedding predictive architecture that predicts section embeddings from earlier sections and abstract embeddings from the full body. We evaluate it on a simulated hierarchical process with known latent scales and on scientific articles, where the abstract serves as the target. On a synthetic process, hierarchical pooling beats flat pooling; the JEPA results are pending. Preprint. Results in this version come from small-scale pilot experiments; the full experimental protocol is described in the text and is pending.