Training Objectives and Evaluation Metrics for Counterfactual Story Rewriting

Counterfactual story rewriting is an important task of natural language processing that expects a model to comprehend a story and a statement contradicting a part of the story, and suitably rewrite the ending of the story in a way that well reflects the counterfactual statement. This task is particularly challenging due to the fact that the model is not expected to rewrite the original ending from scratch, but only make the minimal, selective changes required to incorporate the countering elements. As such, conventional training objectives that train the models to predict whole reference sentences may fail to capture the nuances of this task. Analogously, standard evaluation metrics that weigh all tokens equally may not be able to discriminate effectively between more correct and less correct predictions. For these reasons, in this article we propose novel training objectives and evaluation metrics that mirror this task more closely, and train and evaluate two Flan-T5 transformer models accordingly. Experiments carried out over a popular counterfactual story rewriting dataset show that the proposed T5 models have achieved significant performance improvements over their respective baselines, and have also outperformed configurations of GPT-3.5, GPT-4o and Gemini 2.0 in most metrics. In addition, a qualitative analysis of the predictions and a heatmap visualization suggest that the modified training objective has been able to focus the model over the required counterfactual elements.