Accountancy Capabilities of Large Language Models

Large language models (LLMs) have shown promise in various domains, but their applicability to specialized fields like accounting remains mostly unexplored. This research investigates the performance of five leading LLMs (Gemini [1], Claude, Mistral [2], ChatGPT, and Llama [3]) on a set of 50 accounting problems derived from cost variance analysis and financial accounting principles. The rest of this paper will help evaluate the extent to which the considered type of models can generate the correct solutions with the right intermediate steps. I will perform two evaluations to identify the first differences between the performance of LLMs and a current state of the art, and the last performance difference. Two evaluations are performed on a task drawn from a university examination and are five months apart. The author’s current state of the art is based only on the implementation considered in the previous section. Both the current state of the art and the author are implementations that were available on the websites mentioned earlier. Both evaluations will use all methods considered for evaluation.