CSRef: Contrastive Semantic Alignment for Speech Referring Expression Comprehension

Referring expression comprehension (REC) aims to locate target objects based on referring expression. It is worth noting that using speech as input on mobile devices offers advantages in convenience and accessibility, particularly for individuals with disabilities. However, current REC methods are limited to textual input, and no established task, methods, datasets, or evaluation criteria exist for speech-based REC. In this paper, we introduce a new task, Speech Referring Expression Comprehension (SREC), along with novel methods, corresponding datasets, and evaluation criteria. Specifically, we propose a new SREC method called CSRef, which utilizes global Contrastive Semantic Alignment (CSA) to learn direct speech semantic extraction. This approach leads to a reduction in complexity in speech comprehension for SREC when compared to the transcription-based baseline, which involves speech recognition followed by REC. Through experiments conducted on three constructed SREC datasets, we have demonstrated the effectiveness and superiority of the proposed CSRef. Furthermore, the proposed method was also evaluated in a downstream application, specifically language-guided face pixelation. A comparison was made between the proposed method and MLLM-Guided Image Editing (MGIE) in terms of both the precision of the modified region and the overall effect of language-guided face pixelation. The results indicate that our method is highly effective.