A Simple but Effective Context Retrieval for Sequential Sentence Classification in Long Legal Documents

Sequential sentence classification extends traditional classification, especially useful when dealing with long documents.However, stateof-the-art approaches face two major challenges: pre-trained language models struggle with input-length constraints, while proposed hierarchical models often introduce irrelevant content.To address these limitations, we propose a simple and effective document-level retrieval approach that extracts only the most relevant context.Specifically, we introduce two heuristic strategies: Sequential, which captures local information, and Selective, which retrieves the semantically similar sentences.Experiments on legal domain datasets show that both heuristics lead to consistent improvements over the baseline, with an average increase of 5.5 weighted-F1 points.Sequential heuristics outperform hierarchical models on two out of three datasets, with gains of up to 1.5, demonstrating the benefits of targeted context.