The Dynamics of Spectral Bias in Operator Learning

Abstract Spectral bias is the phenomenon where neural networks learn the low-wavenumber content of a target before its high-wavenumber content. We analyze the training dynamics of neural operators on the discrete Poisson problem and find that the bias is a property of the eigenbasis of the kernel that governs the training update, not of the wavenumber basis in which it is usually reported. A model learning a discretized operator retains its error in the eigendirections of this kernel with small eigenvalues, which are the rates at which training contracts the error along those directions. Under smooth forcing, low variance is placed at high wavenumbers, and any model, linear or nonlinear, has small eigenvalues along the eigenvectors with high-wavenumber content. Under white forcing, the linear models learn the operator, while the nonlinear models retain error in slow directions that depend on the coupling between training samples in the kernel; for the GELU dense network they disappear entirely when that coupling is removed. Modes that a model cannot represent at all contribute zero eigenvalues. The slow eigenvectors decide which wavenumbers carry this error, through their projection onto the sine modes. For the spectral operator and the linear Fourier neural operator these eigenvectors are close to sine modes, so each wavenumber has its own rate and the error collects at high wavenumbers. Delocalized models, such as the dense networks under either forcing, have error of nearly the same magnitude at every wavenumber. The spectrum of the true solution decays with $k$ and falls below this floor at some wavenumber, past which the relative error grows. No mode is learned more slowly than another in this case; the growth comes from the decay of the target spectrum. Spectral bias is usually reported by looking at this retained error in the sine basis, and we use its projection onto the kernel's eigendirections to explain the shape it has there.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-13
DOI
https://doi.org/10.21203/rs.3.rs-11713020/v1
Primary Topic
Model Reduction and Neural Networks
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

The Dynamics of Spectral Bias in Operator Learning

Ashesh Chattopadhyay, Ching-yao Lai, Conrad Ainslie
Research Square
Model Reduction and Neural Networks
preprint

The Dynamics of Spectral Bias in Operator Learning

Ashesh Chattopadhyay, Ching-yao Lai, Conrad Ainslie
preprint en

Abstract

Abstract Spectral bias is the phenomenon where neural networks learn the low-wavenumber content of a target before its high-wavenumber content. We analyze the training dynamics of neural operators on the discrete Poisson problem and find that the bias is a property of the eigenbasis of the kernel that governs the training update, not of the wavenumber basis in which it is usually reported. A model learning a discretized operator retains its error in the eigendirections of this kernel with small eigenvalues, which are the rates at which training contracts the error along those directions. Under smooth forcing, low variance is placed at high wavenumbers, and any model, linear or nonlinear, has small eigenvalues along the eigenvectors with high-wavenumber content. Under white forcing, the linear models learn the operator, while the nonlinear models retain error in slow directions that depend on the coupling between training samples in the kernel; for the GELU dense network they disappear entirely when that coupling is removed. Modes that a model cannot represent at all contribute zero eigenvalues. The slow eigenvectors decide which wavenumbers carry this error, through their projection onto the sine modes. For the spectral operator and the linear Fourier neural operator these eigenvectors are close to sine modes, so each wavenumber has its own rate and the error collects at high wavenumbers. Delocalized models, such as the dense networks under either forcing, have error of nearly the same magnitude at every wavenumber. The spectrum of the true solution decays with $k$ and falls below this floor at some wavenumber, past which the relative error grows. No mode is learned more slowly than another in this case; the growth comes from the decay of the target spectrum. Spectral bias is usually reported by looking at this retained error in the sine basis, and we use its projection onto the kernel's eigendirections to explain the shape it has there.

Research Square
University of California, Santa Cruz (US), Stanford University (US)
Model Reduction and Neural Networks
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon