AILQA: Evaluating AI-driven legal question answering systems for the indian legal system

Abstract This comprehensive study introduces an advanced A rtificial Intelligence for I ndian L egal Q uestion A nswering or system tailored for the Indian legal context. leverages a variety of embedding and generative models, including the latest Large Language Models (LLMs), to address the unique challenges posed by the intricate and diverse nature of Indian legal texts, to enhance the accuracy and reliability of legal question responses. We conducted rigorous evaluations using both lexical and semantic metrics that are enriched by expert legal feedback to ensure relevance and accuracy. Our findings underscore the effectiveness of the Retrieval-Augmented Generation (RAG) paradigm in improving answer quality, particularly in complex legal domains. Additionally, we explored the performance on standardized tests such as the All India Bar Exam (AIBE), thus providing a robust benchmark for a practical application. Under the study’s evaluation protocol, some AI-generated responses received higher ratings than the available reference answers, particularly when they contained accurate and relevant supporting detail. This finding is specific to the evaluated dataset and rating criteria and should not be interpreted as evidence that the models generally outperform qualified legal professionals. We also discuss the challenges encountered, such as the need for precise context and the risks of model hallucination, and propose directions for future research to further refine AI capabilities in the legal field. This study aims to pave the way for enhanced legal decision-making support systems, making them more accessible and effective for legal professionals and the public alike.