LLMicroscopilot: An LLM assistant for scanning tunneling and transmission electron microscopy measurements

A key challenge in experimental high-resolution microscopy is the real-time interpretation of the observed images in conjunction with the parameters adjusted by the experimenter during data acquisition, e.g. to obtain a certain contrast. The parameter space of candidate structures, experimental parameters, and resulting image contrast can be vast and complex, often requiring a scientist who is well-trained in performing image simulations and a lot of trial and error to come up with a plausible interpretation of the observation. To make such expert simulations accessible to anyone, we introduce LLMicroscopilot, a chatbot tool powered by a large language model (LLM), that searches materials databases for suitable structures and simulates various types of microscopy images from them, without requiring the user to know how to operate either the database search, or the simulation software. It recommends suitable simulation parameters for obtaining high-quality results, recovers from errors in code execution, and refines results in dialog with the user. We demonstrate the capability of Microscopilot to operate transmission electron microscopy (TEM) and scanning tunneling microscopy (STM) simulation software back-ends, and more. Additionally, we highlight the potential for mining the memory of microscopy parameters that this tool is capable of building up, e.g. by recommending parameters that have led to accepted results in previous uses. This work is funded by the NFDI consortium FAIRmat - Deutsche Forschungsgemeinschaft (DFG) - Project 460197019