AskQE: Question Answering as Automatic Evaluation for Machine Translation

How can a monolingual English speaker determine whether an automatic translation in French is good enough to be shared?Existing MT error detection and quality estimation (QE) techniques do not address this practical scenario.We introduce ASKQE, a question generation and answering framework designed to detect critical MT errors and provide actionable feedback, helping users decide whether to accept or reject MT outputs even without the knowledge of the target language.Using CON-TRATICO, a dataset of contrastive synthetic MT errors in the COVID-19 domain, we explore design choices for ASKQE and develop an optimized version relying on LLAMA-3 70B and entailed facts to guide question generation.We evaluate the resulting system on the BIOMQM dataset of naturally occurring MT errors, where ASKQE has higher Kendall's correlation and decision accuracy with human ratings compared to other QE metrics. 1 1 We release our code and dataset at https://github. com/dayeonki/askqe.COMET: 0.76 The SARS-CoV-2 virus may last on cardboard for up to 24 hours and on plastic for up to 3 days.Le virus SARS-Cov-2 peut survivre jusqu' 24 minutes sur du carton et jusqu' 3 jours sur du plastique.Q1.How long does the SARS CoV-2 virus last on cardboard?A1 (Source).24 hours A1 (MT).24 minutes Q2.How long does the SARS CoV-2 virus last on plastic?A2 (Source).3 days A2 (MT). 3 days Source.MT..The SARS-Cov-2 virus can survive up to 24 minutes on cardboard and up to 3 days on plastic.Backtranslation AskQE Le virus SARS-Cov-2 peut survivre jusqu' 24 minutes sur du carton et jusqu' 3 jours sur du plastique.Accept or Reject MT? Error Annotations QE Score F1.They are enveloped viruses.F2.They have a positive-sense single-stranded RNA genome.F3.They have a double-stranded RNA genome.F4.They have a nucleocapsid of helical symmetry.