Entity-Guided Contrastive Learning for Multimodal Abstractive Sentence Summarization

Multimodal abstractive sentence summarization condenses a news sentence into a headline with the help of its accompanying photograph. Vision-guided models inject a global visual representation into the encoder, yet they treat the entities named in the source sentence as ordinary tokens, so the decoder receives little fine-grained grounding and the visual signal is easily swamped by the stronger linguistic signal. This paper presents an entity-guided contrastive framework built on BART. Knowledge-graph entities recognised in the source sentence are represented with pretrained TransE embeddings and encoded jointly with the text and with the image by two encoders that share parameters. A gated fusion module combines the two encoder outputs into a single memory for the decoder, and contrastive objectives align the resulting representations with each other and with the target-summary representation. On the 2,000 instances of the MMSS test set, and against a text-only variant trained from the same pretrained weights under the same recipe, the framework improves ROUGE-1 from 55.13 to 56.38, ROUGE-2 from 31.60 to 32.89 and ROUGE-L from 51.38 to 52.47, and a paired bootstrap places the ROUGE-1 gain outside the zero-difference interval. A diagnostic section locates the remaining headroom in the decoder rather than in the alignment: the contrastive representations do separate the two branches, but the decoder grants the visual block under two per cent of its cross-attention mass, which bounds what the alignment can contribute to the headline.