A Pathway to General-Purpose Scientific AI: Multimodal Comprehension of Scientific Images

Modern scientific communication is highly visual, yet current digital libraries and AI systems remain largely textcentric.While vision-language models (VLMs) such as GPT-4V and Gemini achieve impressive results on general benchmarks,recent evaluations show that they falter on scientific figures, struggling with multi-step reasoning, spatial inference, and domain-specific interpretation. This gap limits the ability of digital libraries to index, retrieve, and analyze the rich knowledge embedded in charts, plots, and other scientific images. In this poster, we present a position on advancing multimodal scientific AI and introduce our ongoing project ALD/E-ImageMiner. The project develops a gold-standard dataset from atomic layer depositionand etching literature, annotated for four complementary tasks: chart type classification, data table extraction, chart-to-textsummarization, and question answering. By enabling multi-task evaluation on domain-specific images, ALD/E-ImageMiner provides a rigorous benchmark for assessing and improving VLMs in specialized scientific contexts. We further argue for a longterm vision of “scientific conceptual understanding from images” as a new benchmark for AI in digital libraries, moving beyond surface-level perception to genuine comprehension of the scientific insights conveyed by visual artifacts. This work contributes a concrete dataset and a broader agenda for transforming digital libraries into truly multimodal knowledge infrastructures.