Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networks
From MaRDI portal
Publication:6437380
arXiv2305.12467MaRDI QIDQ6437380
Author name not available (Why is that?)
Publication date: 21 May 2023
Abstract: The training process of ReLU neural networks often exhibits complicated nonlinear phenomena. The nonlinearity of models and non-convexity of loss pose significant challenges for theoretical analysis. Therefore, most previous theoretical works on the optimization dynamics of neural networks focus either on local analysis (like the end of training) or approximate linear models (like Neural Tangent Kernel). In this work, we conduct a complete theoretical characterization of the training process of a two-layer ReLU network trained by Gradient Flow on a linearly separable data. In this specific setting, our analysis captures the whole optimization process starting from random initialization to final convergence. Despite the relatively simple model and data that we studied, we reveal four different phases from the whole training process showing a general simplifying-to-complicating learning trend. Specific nonlinear behaviors can also be precisely identified and captured theoretically, such as initial condensation, saddle-to-plateau dynamics, plateau escape, changes of activation patterns, learning with increasing complexity, etc.
Has companion code repository: https://github.com/wmz9/understanding_multi-phase_optimization_neurips2023
This page was built for publication: Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networks
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6437380)
