Tackling the Length Barrier: Dynamic Context Browsing for Knowledge-Intensive Task

Knowledge-intensive tasks often require complex reasoning and contextual understanding over long contexts. However, the learning and deployment of long-LLMs remains a challenging problem despite recent progresses. In this work, we propose that the short LLMs have great potentiality for solving knowledge-intensive tasks that have long context, i.e. they can be solved by purely working with oracle short-contexts within the input long-context. On top of this argument, we propose a framework called DCISO DynamiC knowledge-Intensive task S>Olver), which enables a short-LLM to address the knowledge-intensive tasks with long context via dynamic context browsing. In our framework, the short-LLM prompts itself to reason for two critical decisions: 1) how to access to the appropriate part of context within the input, 2) how to make effective use of the accessed context. By adaptively accessing and utilizing the context based on the presented tasks, DCISO can serve as a general framework to handle diversified knowledge-intensive long-context problems. We comprehensively evaluate different types of tasks from popular long-context benchmarks, where DCISO is able to achieve a substantially improved performance. Our codes will be released at this repository.