A Multimodal Hierarchical Attentional Ordering Network

Sequence coherence modeling refers to how to coherently organize a given set of elements within a sequence, which is a fundamental aspect in comprehension, generation, and reasoning tasks. Although making great progress, existing deep neural network-based ordering approaches suffer from critical limitations: (1) Most of the previous coherence modeling approaches focus on the ordering of pure text sequences, while real-world data are often expressed in multimodal form; (2) The pairwise ranking method, widely used for ordering single/multimodal sequences, faces challenges in effectively modeling overall contextual relationships; (3) The redundancy problem would exist in multimodal ordering and can influence the encoding performance, which has been ignored when performing ordering. In this article, we propose a novel Multimodal Hierarchical Attentional Ordering Network (referred to as MHAONet) for multimodal sequence ordering, which effectively captures the required global dependencies in multimodal sequences to enhance the coherence modeling. In particular, a multimodal encoder is introduced to encode the whole sequence for fully exploiting global contexts, where a multimodal selectively visible mask module is elaborately designed for controlled information transmission between different modalities. This enables the model to capture multimodal semantic and contextual information across the sequence. Further, the token-level and paragraph-level hierarchical attention layers are employed to encourage the model to concentrate on features of different granularities. In addition, a transformer-based pointer network is used for performing order prediction. Comprehensive experiments and analysis on benchmark datasets demonstrate the effectiveness and superiority of our proposed model.