Prompt-based Masked Language Modeling for Numerical Reasoning

Headline generation, a crucial task in summarization, aims to summarize an entire article into a concise, single line. Despite the proficiency of sequence-to-sequence encoder-decoder models and transformer-based large language models in text generation and summarization, generating headlines that include numerals representing the numerals in the news body remains a significant challenge. Generating a numeral-aware headline requires the ability of models to solve numerical and mathematical reasoning capabilities to infer relationships between numerals in the news body. Given the challenges in numeral-aware headline generation and numerical reasoning over numerals in news bodies, this study conducts an empirical investigation of Large Language Models (LLMs) using various strategies, including Pretrained , Few-Shot Prompting , and Chain-of-Thought Prompting , for numeral aware headline generation and numerical reasoning for headline generation. Building upon the insights gained from our empirical study on LLMs for numeral-aware headline generation and numerical reasoning, we propose two novel approaches: instruction tuning with Large Language Models for numeral-aware headline generation and prompt-based masked language modelling for numerical reasoning. We conducted our experiments on the NumHG dataset. We observed that our proposed method outperforms the Pretrained , Few-Shot Prompting , and Chain-of-Thought Prompting setups of Large Language Models (LLMs), as well as baseline models from the literature, on both numeral-aware headline generation and numerical reasoning tasks. Observations from the experimental results reveal that our proposed Prompt-Based Masked Language Modeling significantly improves the performance of small and medium-sized language models on numerical reasoning tasks. We also study the robustness of our proposed models by evaluating their performance in fact-checking numerical claims and performing numerical reasoning for numeral-aware text summarization. Our findings suggest that the proposed Prompt-Based Masked Language Modeling approach is also effective for numerical claim verification and numerical reasoning for numeral-aware text summarization.