META: Text Detoxification by leveraging METAmorphic Relations and Deep Learning Methods

In the world of online interactions, social communities face a significant challenge: the spread of offensive content and hate speech through toxic languages. Such issues led to growing research on text detoxification systems that can automatically rewrite toxic content. A systematic evaluation is required to ensure these systems produce high-quality detoxified text that modifies the original text to be non-toxic while preserving its content. However, this often relies on large amounts of labelled data and human judgement, which may not always be feasible. This limitation is typically known as the oracle problem. Metamorphic testing (MT) has conventionally been used to solve the oracle problem by deriving metamorphic relations (MRs) to test a program’s functionality. A new MT approach focused on data validation showed that MRs incorporated with tools can be used to identify defects in machine translation services. This paper draws inspiration from this new MT perspective by presenting four metamorphic relations incorporated with tools to evaluate style transfer accuracy, content preservation, fluency, and a joint of these three. Our proposed approach effectively identifies defective behaviour in state-of-the-art text detoxification systems.