Retrieval-Augmented Sign Language Translation

In this work, we present a framework named Retrieval-Augmented Sign Language Translation (RASLT). Since human translators can provide more accurate answers when they have access to similar translation samples proofread by experts, it is generally believed that similar references should be beneficial for the translation process. To augment existing approaches with extra references beyond input sign language video, our RASLT utilizes a cross-modal query expansion mechanism to enhance the input of existing sign language translation systems. Technically, our RASLT performs sign language translation in two stages, i.e., video retrieval and text generation. The video retriever first searches for the extra text descriptions from a sign language database based on the similarities between the sign language videos. Then, the retrieval-augmented translator takes the retrieved text descriptions as additional cues to generate the final translation, which leverages the attention mechanism to enhance the semantic representations of the input sign language video. Moreover, we further propose the augmented-pair loss to encourage the retrieved-augmented translator to explore the cross-modality relatedness and narrow the modality gap. Our framework is ready to be combined with different sign language translation systems to enhance their performance with acceptable overhead. Extensive experiments on public benchmarks demonstrate the effectiveness of our framework.