Deep Learning for Localization Microscopy in 2D and 3D

ConspectusSuper-resolution microscopy has fundamentally transformed our ability to observe biological structures, allowing nanoscale observation into biological samples such as fixed and live cells and tissues, and has been notably recognized by the 2014 Nobel Prize in Chemistry. Single-molecule localization microscopy (SMLM) stands out as a highly successful and widely accessible method for super-resolution microscopy. However, because SMLM relies on computationally reconstructing a single image from thousands of sparse frames, it suffers from significant algorithmic bottlenecks, particularly when dealing with high emitter densities or three-dimensional data. Deep learning has emerged as an exceptionally effective solution to bypass these computational challenges, enabling fast, parameter-free reconstruction. The application of neural networks for SMLM analysis is uniquely powerful because such networks can be trained entirely on simulated data; since the optical point spread function is well-understood, and SMLM images are fundamentally simple, consisting of a collection of point-spread functions, it is relatively easy to numerically simulate the vast quantities of highly accurate training images required to train reconstruction nets.In this Account, we summarize our contributions to localization microscopy by applying neural nets to address its limitations and bottlenecks. Specific challenges and applications include dense molecule fitting in 2D and in 3D, single-channel multicolor imaging, super spatiotemporal resolution microscopy, optical genome mapping, large field-of-view (FOV) imaging, and accurate background simulation. The fact that the training of the neural net is based on simulated images enables powerful and interesting capabilities. For example, the optical setup itself can be algorithmically designed together with the decoding neural net; we have used this concept for tasks such as designing optimal phase masks for depth encoding, for spectral encoding, and even for both simultaneously.