Semantic Evaluation of Multilingual Data-to-Text Generation via NLI Fine-Tuning: Precision, Recall and F1 scores

Performance in the Knowledge Graph-to-Text generation has improved over time, particularly in English.However, models are still prone to mistakes like Additions and Omissions.Furthermore, few languages are taken into account since both train and test data are not readily available.In this paper, we hope to facilitate the development and improvement of multilingual KG-to-Text models by providing a multilingual evaluation framework that is referenceless and permits estimating how much a KGto-Text Model under-(omission) or over-(addition) generates.We focus on two high (English, Russian) and five low (Breton, Irish, Maltese, Welsh, Xhosa) resource languages and show that our metric has fair to moderate correlation with reference-based metrics, positioning it as a consistent alternative when no references are available.We also show that our metric outperforms prior reference-less metrics in correlation with existing human judgments.Additional human evaluation shows moderate to strong correlation with human annotators in assessing precision and recall at a higher granularity level than shown in previous studies.Since our metric provides scores for precision and recall, it helps better assess the level of over-or undergeneration of multilingual KG-to-Text models.We make our data, code and models available 1 .