Two-Stage Machine Learning Framework for Accurate Discrimination of Isomers and Very-Similar Molecules on Surfaces

The accurate detection and discrimination of on-surface organic isomers and very similar molecules are crucial for monitoring chemical reaction processes and analyzing various reaction mechanisms and molecular properties. Despite its importance, nano- and surface science communities still lack an efficient, robust, precise, and automated detection approach for on-surface isomers and highly similar molecules. Here, we present ReSTOLO, a convolution neural network (CNN)-based framework for precise detection and identification of multiple types of sparsely distributed molecules on surfaces, particularly designed for scanning tunneling microscopy (STM) images containing numerous molecules with analogous features. To address challenges arising from molecular shape and size similarities, we implemented a two-stage framework comprising two CNN models: YOLO v5.m was used for molecular localization, and ResNet-101 for classification. The framework optimally harnesses the advantages of both models by applying a box normalization connection. We demonstrated the framework's effectiveness by applying it to analyze a surface reaction process involving six molecules with nearly identical STM signatures. The training process employed an STM image database of single molecules augmented with physical and experimental tools constructed using standardized image boxes. This two-stage approach achieved approximately ∼20% improvements in performance metrics, including precision, recall, and accuracy, compared to conventional frameworks. The framework exhibits robust capabilities in automatically and efficiently pinpointing and discriminating between molecular species with similar configurations in complex surface reactions. This automated molecular discriminator represents a significant advance in facilitating STM tip-manipulated chemical reactions on surfaces.