Evaluating large language models for philosophical reasoning on ancient Indian texts

Despite the rapid advances in large language models (LLMs), understanding highly philosophical and culturally sophisticated writings remains underexplored. Ancient Indian works, in particular, pose unique challenges; they are rich in metaphor, moral argument, and contextual richness surpassing literal understanding or surface-level recall. In this experiment, we assessed if model type, retrieval strategy, and decoding parameters impact the accuracy and generalizability of LLMs when responding to multiple-choice questions (MCQs) from an ancient Indian philosophical work. We incorporated seven MCQs within a retrieval-augmented generation (RAG) setting and evaluated two leading LLMs ChatGPT-3.5 Turbo and Llama-2 in sixteen different settings varying in retrieval strategy (MPNet vs TF-IDF), temperature (1.0 vs 2.0), and top-p (0.1 vs 0.9). A total of 2,800 responses investigated two three-way ANOVAs to test the main and interaction effects of the variables. We find Llama-2 to systematically dominate ChatGPT-3.5 Turbo in both accuracy and generalizability. For decoding parameters, top-p emerged with a significant impact, with increasing values yielding improved performance, whereas temperature had a minor impact. Strikingly, no measurable difference was observed between the two retrieval strategies under the present experimental conditions. These findings suggest that, under the present experimental conditions, model architecture plays a more prominent role than retrieval strategy in driving performance. This study presents a new test framework for philosophical reasoning with LLMs and provides avenues for applying AI constructively in education, understanding of different cultures, and transmission of moral traditions.