Avoiding degenerate saddles of the homogeneous cost of deep linear networks

Jacot, Ged, Şimşek, Hongler and Gabriel described the small-initialization training of deep linear networks through the gradient flow of a homogeneous cost. This description needs the flow not to converge to a saddle, proved almost surely for depth at most three when the input and output dimensions agree, and with probability at least 1/2 for larger depth. Every critical point of the cost is a strict saddle or has vanishing Hessian. The conserved imbalances between layers almost surely exclude limits with two vanishing layers and, for hidden layers of equal width, limits with two vanishing products of adjacent layers, or a vanishing layer and such a product not containing it, unless both products contain one vanishing layer. This settles depth three for every gradient of full rank, and depth four when also the hidden layers have equal width. For larger depth we reduce the question to critical points with at most one vanishing layer and give a numerical example at depth eight that the conserved quantities cannot exclude.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-16
DOI
https://doi.org/10.5281/zenodo.24272918
Primary Topic
Stochastic Gradient Optimization Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Avoiding degenerate saddles of the homogeneous cost of deep linear networks

Yitzchak Shmalo
Zenodo (CERN European Organization for Nuclear Research)
Stochastic Gradient Optimization Techniques
preprint

Avoiding degenerate saddles of the homogeneous cost of deep linear networks

Yitzchak Shmalo
preprint en

Abstract

Jacot, Ged, Şimşek, Hongler and Gabriel described the small-initialization training of deep linear networks through the gradient flow of a homogeneous cost. This description needs the flow not to converge to a saddle, proved almost surely for depth at most three when the input and output dimensions agree, and with probability at least 1/2 for larger depth. Every critical point of the cost is a strict saddle or has vanishing Hessian. The conserved imbalances between layers almost surely exclude limits with two vanishing layers and, for hidden layers of equal width, limits with two vanishing products of adjacent layers, or a vanishing layer and such a product not containing it, unless both products contain one vanishing layer. This settles depth three for every gradient of full rank, and depth four when also the hidden layers have equal width. For larger depth we reduce the question to critical points with at most one vanishing layer and give a numerical example at depth eight that the conserved quantities cannot exclude.

Zenodo (CERN European Organization for Nuclear Research)
Hebrew University of Jerusalem (IL)
Stochastic Gradient Optimization Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon