Bridging the Markush Gap in Optical Chemical Structure Recognition via a CLIP-Derived Visual Backbone and Synthetic Data Generation

Abstract The rapidly growing chemical literature contains invaluable information for pharmaceutical research and molecular biology, yet manual extraction on a large scale is increasingly impractical. Optical chemical structure recognition (OCSR), which converts molecular depictions into machine-readable representations, provides a key route toward automated chemical information extraction. However, existing OCSR methods remain limited in recognizing Markush structures, which are widely used in chemical documents, particularly pharmaceutical patents, because of their structural complexity and the scarcity of annotated data. Here, we introduce CLIP-OCSR, an encoder-decoder framework that integrates a domain-adaptively pretrained CLIP-RN50 vision encoder with a transformer decoder for chemical structure recognition, with a focus on Markush representations. To address data scarcity, we developed a synthetic data generation strategy that produces diverse Markush images paired with pseudo-SMILES representations encoding substituent, frequency, and position variations. Benchmarking demonstrates that CLIP-OCSR achieves competitive performance on standard OCSR benchmarks and shows strong capability in recognizing Markush structures involving substituents and frequency variations. For structures with position variations, CLIP-OCSR generates pseudo-SMILES predictions that retain the corresponding positional information, enabling the downstream enumeration of possible positional isomers. Beyond image-only OCSR, we extend CLIP-OCSR toward multimodal Markush structure extraction by integrating graphical recognition with LLM-based textual variable-definition extraction, enabling downstream structure instantiation. Overall, this work advances automated recognition of complex chemical structures and provides a framework for the structured extraction of Markush information from chemical documents.