A Two-Stage Vision-Language Model for Image Captioning and Future Event Prediction

Image captioning has traditionally focused on generating descriptions for individual static images. However, predicting future events from visual information is a fundamental challenge in this domain. While existing methods primarily describe current visual content, the ability to anticipate and generate captions for future events remains largely unexplored. We propose a novel approach for future caption prediction by leveraging the capabilities of Vision Transformer (ViT), Generative Pre-trained Transformer 2 (GPT-2), and Text-to-Text Transfer Transformer Model (T5) architectures. Our method employs a two-stage pipeline: ViT-GPT2 generates captions for current images, and T5 analyzes these captions to identify patterns and predict future captions. We evaluate our approach on a custom Cooking Dataset focused on sequential cooking activities and compare its performance with baseline approaches. The results demonstrate that our model outperforms baselines in standard caption-generation metrics, while also offering more contextually rich and anticipatory insights relevant to sequential tasks. Our work opens new possibilities for applications requiring anticipatory understanding, such as robotic assistance, automated surveillance, and predictive visual analysis.