Financial Language Model Evaluation (FLaME)

Language Models (LMs) have demonstrated impressive capabilities with core Natural Language Processing (NLP) tasks.The effectiveness of LMs for highly specialized knowledgeintensive tasks in finance remains difficult to assess due to major gaps in the methodologies of existing evaluation frameworks.These gaps have caused an erroneous belief in a far lower bound of LMs' performance on common Finance NLP (FinNLP) tasks.To accurately assess LM capabilities and demonstrate their potential for FinNLP tasks, we present the first holistic benchmarking suite for Financial Language Model Evaluation (FLAME).Our work includes the first comprehensive empirical study comparing standard LMs with 'reasoning-reinforced' LMs with 23 foundation LMs over 20 core financial NLP tasks.We open-source our framework software along with all data and results.