Stationarity in nonsmooth optimization between geometrical motivation and topological relevance

Abstract The goal of this paper is to compare alternative stationarity notions in structured nonsmooth optimization (SNO). Here, nonsmoothness is caused by complementarity, vanishing, orthogonality type, switching, or disjunctive constraints. On one side, we consider geometrically motivated notions of $$\widehat{N}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mover> <mml:mi>N</mml:mi> <mml:mo>^</mml:mo> </mml:mover> </mml:math> -, N -, and $$\overline{N}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mover> <mml:mi>N</mml:mi> <mml:mo>¯</mml:mo> </mml:mover> </mml:math> -stationarity in terms of Fréchet, Mordukhovich, and Clarke normal cones to the feasible set, respectively. On the other side, we advocate the notion of topologically relevant T-stationarity, which adequately captures the global structure of SNO. Our main findings say that (a) $$\widehat{N}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mover> <mml:mi>N</mml:mi> <mml:mo>^</mml:mo> </mml:mover> </mml:math> -stationary points include all local minimizers; (b) N -stationary points, which are not $$\widehat{N}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mover> <mml:mi>N</mml:mi> <mml:mo>^</mml:mo> </mml:mover> </mml:math> -stationary, correspond to the singular saddle points of first order; (c) T-stationary points, which are not N -stationary, correspond to the regular saddle points of first order; (d) $$\overline{N}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mover> <mml:mi>N</mml:mi> <mml:mo>¯</mml:mo> </mml:mover> </mml:math> -stationary points, which are not T-stationary, are irrelevant for optimization purposes, at least from the topological point of view. Overall, a hierarchy of stationarity notions for SNO is established.