STEM-POM: Evaluating Language Models Math-Symbol Reasoning in Document Parsing

Advances in large language models (LLMs) have spurred research into enhancing their reasoning capabilities, particularly in mathrich STEM (Science, Technology, Engineering, and Mathematics) documents.While LLMs can generate equations or solve mathrelated queries, their ability to fully understand and interpret abstract mathematical symbols in long, math-rich documents remains limited.In this paper, we introduce STEM-POM, a comprehensive benchmark dataset designed to evaluate LLMs' reasoning abilities on math symbols within contextual scientific text.The dataset, sourced from realworld ArXiv documents, contains over 2K math symbols classified as main attributes of variables, constants, operators, and unit descriptors, with additional sub-attributes including scalar/vector/matrix for variables and local/global/discipline-specific labels for both constants and operators.Our extensive experiments demonstrate that state-of-the-art LLMs achieve an average accuracy of 20-60% under in-context learning and 50-60% with finetuning, highlighting a substantial gap in their ability to classify mathematical symbols.By improving LLMs' mathematical symbol classification, STEM-POM further enhances models' downstream mathematical reasoning capabilities.