GENDEROUS: Machine Translation and Cross-Linguistic Evaluation of a Gender-Ambiguous Dataset

Contributing to research on gender beyond the binary, this work introduces GENDEROUS, a dataset of gender-ambiguous sentences containing gender-marked occupations and adjectives, and sentences with the ambiguous or nonbinary pronoun their.We cross-linguistically evaluate how machine translation (MT) systems and large language models (LLMs) translate these sentences from English into four grammatical gender languages: Greek, German, Spanish and Dutch.We show the systems' continued default to male-gendered translations, with exceptions (particularly for Dutch).Prompting for alternatives, however, shows potential in attaining more diverse and neutral translations across all languages.An LLMas-a-judge approach was implemented, where benchmarking against gold standards emphasises the continued need for human annotations.