Sep‐NMS: Unlocking the Aptitude of Two‐Stage Referring Expression Comprehension

ABSTRACT Referring expression comprehension (REC) aims to locate a specific region in an image described by a natural language. Existing two‐stage methods generate multiple candidate proposals in the first stage, followed by selecting one of these proposals as the grounding result in the second stage. Nevertheless, the number of candidate proposals generated in the first stage significantly exceeds ground truth and the recall of critical objects is inadequate, thereby enormously limiting the overall network performance. To address the above issues, the authors propose an innovative method termed Separate Non‐Maximum Suppression (Sep‐NMS) for two‐stage REC. Particularly, Sep‐NMS models information from the two stages independently and collaboratively, ultimately achieving an overall improvement in comprehension and identification of the target objects. Specifically, the authors propose a Ref‐Relatedness module for filtering referent proposals rigorously, decreasing the redundancy of referent proposals. A Relatedness module based on robust multimodal pre‐trained encoders is built to precisely assess the relevance between language and proposals to improve the recall of critical objects. It is worth mentioning that the authors are the pioneers in utilising a multimodal pre‐training model for proposal filtering in the first stage. Moreover, an Information Fusion module is designed to effectively amalgamate the multimodal information across two stages, ensuring maximum utilisation of the available information. Extensive experiments demonstrate that the approach achieves competitive performance with previous state‐of‐the‐art methods. The datasets used are publicly available: RefCOCO, RefCOCO+: https://doi.org/10.1007/978‐3‐319‐46475‐6_5 and RefCOCOg: https://doi.org/10.1109/CVPR.2016.9 .