Supervised Learning and SCST Image Captioning on Flickr30k: Analyzing Metric-Driven Trade-offs in Semantic Performance

Image captioning plays a pivotal role in enhancing accessibility, searchability, and human-computer interaction by automatically generating natural language descriptions for images.While standard encoder-decoder models trained using Supervised Learning (SL) with cross-entropy loss optimize word-level accuracy, they often fail to align with sequence-level evaluation metrics such as BLEU, CIDEr, and SPICE.Additionally, these models suffer from exposure bias.This paper explores augmenting SL with Reinforcement Learning (RL), specifically Self-Critical Sequence Training (SCST), to optimize the CIDEr metric, which better correlates with human consensus judgments.Using the Flickr30k dataset, we employed an attention-based ResNet-101 encoder and an LSTM decoder.In the SL phase, the model achieved a minimum validation loss of 3.08 and a token prediction accuracy of 0.404.Fine-tuning for 8 epochs using SCST, targeting the CIDEr metric, resulted in a significant improvement in n-gram metrics (e.g., a +7.3% relative gain in BLEU-3) and a modest +1.4% improvement in CIDEr.However, a -4.6% relative change was observed in the SPICE score, which measures semantic propositional accuracy.This study provides a detailed implementation, quantitative analysis, and qualitative examples, offering insights into the trade-offs of metric-specific optimization in RL for image captioning.It highlights the challenge of balancing improvements in n-gram consensus (CIDEr) with preserving semantic fidelity (SPICE) and underscores the need for diverse evaluation metrics, as optimizing one may negatively impact others.