Benign Overfitting for Two-layer ReLU Convolutional Neural Networks

Author name not available (Why is that?)

Publication date: 7 March 2023

Abstract: Modern deep learning models with great expressive power can be trained to overfit the training data but still generalize well. This phenomenon is referred to as benign overfitting. Recently, a few studies have attempted to theoretically understand benign overfitting in neural networks. However, these works are either limited to neural networks with smooth activation functions or to the neural tangent kernel regime. How and when benign overfitting can occur in ReLU neural networks remains an open problem. In this work, we seek to answer this question by establishing algorithm-dependent risk bounds for learning two-layer ReLU convolutional neural networks with label-flipping noise. We show that, under mild conditions, the neural network trained by gradient descent can achieve near-zero training loss and Bayes optimal test risk. Our result also reveals a sharp transition between benign and harmful overfitting under different conditions on data distribution in terms of test risk. Experiments on synthetic data back up our theory.

Has companion code repository: https://github.com/uclaml/benign_relu_cnn

This page was built for publication: Benign Overfitting for Two-layer ReLU Convolutional Neural Networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6428731)