Optimization behavior in Physics-Informed Neural Networks: Diagnostic analysis of PDE formulations

In the development of Physics-Informed Neural Networks, introducing auxiliary variables or adopting variational formulations is often explored as a strategy to facilitate the modeling of coupled partial differential equations. While these architectural choices offer theoretical advantages, their interaction with the underlying optimization behavior can introduce specific complexities into the learning process. This study presents a comparative diagnostic analysis of primitive, mixed, and energy-based formulations across fluid dynamics, heterogeneous porous media, and solid mechanics. Utilizing Neural Tangent Kernel and Centered Kernel Alignment metrics, we evaluate the spectral properties and geometric adaptation of the feature space during training. The empirical evaluations indicate that applying heterogeneous constraints can create a structural disparity, where the optimization process may prioritize lower-complexity residuals and consequently attenuate the learning signals for globally coupled physical laws. In physical systems featuring inherent singularities, standard global residual minimization may encounter difficulties in fully resolving high-gradient regions, and unconstrained variational approaches can be susceptible to optimization instabilities. Furthermore, evaluating multi-task balancing algorithms suggests that while gradient projection methods can mitigate directional conflicts in convective systems, they may restrict feature adaptation in domains characterized by strong spatial heterogeneity or kinematic coupling. These results suggest that balancing algorithms can mitigate symptomatic optimization issues, but they do not resolve structural ill-conditioning induced by heterogeneous constraints. When constraints are strongly coupled or spatially heterogeneous, gradient-projection methods may restrict representation learning, indicating that redesigning the formulation can be necessary beyond mere reweighting. • Field-specific NTK and CKA diagnose PINN optimization across three physical systems. • Heterogeneous constraints can attenuate learning of governing conservation laws. • Loss balancing alone may not resolve structural ill-conditioning in PDE formulations.