Learning to Assemble Compositional Neural Modules for Visual Reasoning

Visual reasoning is one of the fundamental abilities of human intelligence. For multi-hop reasoning problems, Neural Module Network (NMN) explicitly incorporates the logical structure of the problems and reveals better flexibility and generalization ability compared with other monolithic models. However, applying each module to a routine limits the scalability of conventional NMNs when processing unseen question semantics. We notice that each routine has mostly three sub-routines: base routine, attribute, and module argument, and make each model concentrate on a sub-routine to make the execution of all routines in a disentangled way. Then, to better achieve the module functionality, we propose language feature filtering to extract the routine-specific features from the self-encoded language token features. The filtered language features are then fused with the hidden token features with cross-attention and MLP and the fused token features are the outputs of each module. Experimental results on CLOSURE and CLEVR-CoGenT reveal that the proposed Compositional NMN (CNMN) can achieve better performances compared with other NMNs with less number of modules, which further validates the effectiveness of learning compositional modules for multi-hop visual reasoning problems.