Zero-Shot Text Classification Using Large Language Models for Key Audit Matters in Japanese Audit Reports

Japanese-listed companies are required to submit audit reports to the Prime Minister of Japan.In principle, these reports must include "Key Audit Matters" (KAMs), which are matters that the auditors, as professional experts, have judged as particularly important when auditing financial statements.A previous study proposed an automatic classification method called zero-shot text classification for KAMs.We examine whether zero-shot text classification with large language models (LLMs) such as ChatGPT can automatically classify KAMs.We also examine how the following three approaches contribute to the accuracy of zero-shot text classification by LLMs: definition refinement, majority decision-making based on LLM outputs, and use of state-of-theart models.The experimental results confirm that definition refinement and majority decisionmaking based on more than three results are useful to some extent.Furthermore, the latest ChatGPT model, gpt-4-1106-preview of the Generative Pre-trained Transformer 4 (GPT-4) model, achieved a classification accuracy of up to 87.2%.