CODNet: Context-based object detection network for multimodal image captioning and virtual question answering

Present MLL (Multimodal Large Language) models do exceptionally well in computer vision tasks, such as answering virtual questions and captioning images. However, they are not up to par in important perceptual tasks like object detection. To overcome this constraint, an entirely novel research work is presented for contextual object detection, which aims to comprehend observable items in various human-AI interaction scenarios. This study investigates two widely used scenarios: captioning the images and virtual question answering. For human-machine interaction, a brand-new CODNet (Context-based Object Detection Network) is presented that can locate, recognize, and connect visual objects with spoken inputs. The proposed network is capable of end-to-end modeling of visual language contexts. This network unifies language and vision functions by interpreting images in a foreign language and coordinating language functions with vision-centric jobs that may be managed and created flexibly using language instructions. The network comprises three sub-modules: an encoder that extracts the required features from the context, a pre-trained large language model meta AI (Llama) for decoding the context, and a decoder for predicting anchor boxes around the contextually recognized objects in images. The features are extracted with the help of a convolutional neural network merged with multiplicative LSTM. A fine-tuned YOLOv6 model detects and locates the required objects in the image provided. The network is trained on the MS-COCO dataset, evaluated on real-time images, and achieved over 95% average weighted accuracy. • Context-based object detection refers to identifying objects within an image or a sequence of images by considering the surrounding context in which these objects appear. For example, a keyboard is usually found near a computer. Some objects are more likely to appear together. For example, a fork and a knife are often found together on a dining table. • This study presents a new context-based object detection network that defines a new direction in visual object detection and improves LLMs. • A novel CODNet network is proposed that helps in first generating the object words and then classifying the objects in the images. The network comprises three sub-modules: encoder, Llama, and decoder. The network combines CNN-mLSTM (Convolutional Neural Network merged with multiplicative LSTM) to help extract features from provided images. Llama is a pre-trained large language model for decoding the context and finetuned YOLOv6 for object detection. • A new set of object words, namely, ERCODE (Easy Readable Context-Based Object DEtection), is also proposed in this work to help classify required objects in images.