Evaluating Capabilities of MT Systems in Translating Idiomatic Expressions Using a Specialized Dataset

Multiword expressions (MWEs) are generally problematic for machine-translation systems. In this paper, we (i) describe a set, available on CLARIN-IS, of appr. 1,000 idiomatic MWEs which have been translated into English; (ii) use the set as a template for a hidden evaluation set, to be used in a new leaderboard for Icelandic language technology, and (iii) evaluate -- using both automatic and manual approaches -- four MT systems' abilities to translate MWEs from Icelandic to English using both datasets. We find that traditional transformer-based MT systems evaluated commonly fail when translating idiomatic expressions, while LLMs do much better.