An Experimental Evaluation of the Capability of Large Language Models to Reason About Value-Added Tax Cases in Austrian Tax Law

This paper explores the application of large language models (LLMs) and retrieval-augmented generation (RAG) systems in creating AI-based assistants for value-added tax (VAT) law consulting. Focusing on Austrian and EU tax law, the study aims to investigate the potential of LLMs as a legal reasoning tool for the automation of the identification of the country where VAT has to be levied in cross-border transactions. Experiments using a compiled dataset of textbook cases achieved over 70% accuracy in identifying the country of supply of goods or provisioning of services, with over 80% of the justifications deemed correct or at least partially correct by an expert evaluation. Despite these promising results, challenges remain, particularly in document retrieval and handling complex cases. The paper contributes a prototype RAG system, a curated case set, and insights into the reliability of LLMs for legal reasoning in VAT law.