GDVT: geometrically aware dual transformer encoding visual and textual features for image captioning

When describing pictures from the point of view of human observers, the tendency is to prioritize eye-catching objects, link them to corresponding labels, and then integrate the results with background information (i.e., nearby objects or locations) to provide context. Most caption generation schemes consider the visual information of objects, while ignoring the corresponding labels, the setting, and/or the spatial relationship between the object and setting. This fails to exploit most of the useful information that the image might otherwise provide. In the current study, we develop a model using object’s tags to supplement insufficient information of visual object features as well as establishing relationship between objects and background features based on relative and absolute coordinate information. In addition, we propose an attention architecture to account for all of the features in generating an image description. The effectiveness of the proposed geometrically aware dual transformer encoding visual and textual features (GDVT) is demonstrated in experiment settings with and without pre-training. The effectiveness of our added features is verified in the case studies.