LoFi: Neural Local Fields for Scalable Image Reconstruction

We introduce LoFi (Local Field)—a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">coordinate-based</i> framework for image reconstruction which combines advantages of convolutional neural networks (CNNs) and neural fields or implicit neural representations (INRs). Unlike conventional deep neural networks, LoFi reconstructs an image one coordinate at a time, by processing only adaptive local information from the input which is relevant for the target coordinate. Similar to INRs, LoFi can efficiently recover images at any continuous coordinate, enabling image reconstruction at multiple resolutions. LoFi achieves excellent generalization to out-of-distribution data with memory usage almost independent of image resolution, while performing as well or better than standard deep learning models like CNNs and vision transformers (ViTs). Remarkably, training on <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$1024 \times 1024$</tex-math></inline-formula> images requires less than 200MB of memory—much less than standard CNNs and ViTs. Our experiments show that Locality enables training on extremely small datasets with ten or fewer samples without overfitting and without explicit regularization or early stopping.