VLMs struggle to extract definitions of structural segments from MSA images: toward the design of a human-in-the-loop annotation pipeline for the PDB-Descriptome project

We investigate the ability of vision-language models (VLMs) to extract structural region definitions from rasterized multiple sequence alignment (MSA) images, as part of the PDB-Descriptome project. Using synthetic MSAs with annotated structural biological entities (SBEs) and structural biological Referring Expressions (SBREs), we evaluate two VLMs, gemini-2.5-flash and gemini-2.5-pro, under naïve and strict prompts. While VLMs perform well in SBRE extraction, they show poor accuracy in defining SBE boundaries. In contrast, a human annotator achieves high boundary precision with slightly lower textual accuracy. These results support a human-in-the-loop pipeline for reliable structure-text annotation.