Evaluation of Large Language Models for Extracting Financial Data from Tables in Annual Reports

This study evaluates the performance of extracting data from tables using three large language models (LLMs), namely ChatGPT 4, Custom GPT based on ChatGPT 4, and ChatPDF, in extracting and interpreting quantitative data from tables in financial reports. The models were tested on six questions regarding financial data with varying levels of difficulty using three financial reports from different industries and provided in different formats. The results are compared in terms of accuracy, precision, error rates, and qualitative analysis of the output quality. The results indicate that LLMs have a very limited ability to correctly read and interpret data from tables using annual reports. The study also showed that the same reports including the text yielded better results than the tables alone. The results also indicated that a more specific query can lead to slightly better results. However, the study shows that the current LLM technology is still unsuitable for practical applications in similar use cases related to table extraction, in particular where a high reliability of results is required. Thus, the study suggests that future research should focus on improving the capabilities of LLMs in financial data analysis, including the development of more advanced techniques for data extraction and interpretation.