Semantic evaluation metric conforming to AMR theory (SEMCAT): a new similarity metric for abstract meaning representation

In this paper, we propose SEMCAT (Semantic Evaluation Metric Conforms to AMR Theory), a novel similarity measuring method for Abstract Meaning Representation (AMR). AMR is a semantic structure used to explicitly express the truth-conditional meaning aspect of a natural language sentence. Our evaluation strategy is mainly designed to reflect the theoretical basis of AMR. Specifically, based on the concept of semantic similarity derived from neo-Davidsonian semantics, we enhance the $$\textsc {Smatch}^\sharp $$ metric with the Weisfeiler-Leman algorithm to reflect both the local and global structure of an AMR graph. In this work, we conducted two experiments to verify the usefulness of the SEMCAT metric. In Experiment 1, we conducted a series of statistical tests to confirm whether SEMCAT conforms to AMR guidelines using our simulated data, which were based on AMR 3.0. The experimental result showed that SEMCAT exhibited clear conformance to AMR guidelines compared to existing metrics. Also, we re-evaluate current state-of-the-art AMR parsers using SEMCAT to report new baselines. In Experiment 2, we examine the usefulness of SEMCAT by evaluating the correlation between AMR metrics and human evaluation scores for the paraphrase task, which is one of the main downstream tasks for AMR. In four benchmark data sets for the evaluation, SEMCAT achieved the highest average correlation among all existing AMR metrics with graph standardization.