Linear unlearning for cross-validation
type: Item
instance of: scholarly article
title: Linear unlearning for cross-validation
zbMATH Open document ID: 0870.65146
DOI: 10.1007/BF02124747
author: Lars Kai Hansen
author: Jan Larsen
published in: Advances in Computational Mathematics
publication date: 1997-09-02T00:00:00Z
review text: Consider nonlinear regression in which the output \(y\) is regressed nonlinearly on the input vector \(x\). The authors focus on a neural network implementation, in which the output is predicted by \(\widehat y=F(x,w)\), where \(F(\cdot)\) denotes the nonlinear mapping of the neural net and \(w\) is the vector of network parameters. The conditional input-output distribution, i.e., the probability distribution of the output conditioned on a test input, is a basic objective for neural net modeling. A main source of uncertainty, when estimating the parameters of the conditional distribution, is the random selection of training data. The idea of cross-validation in neural net learning is based on training and testing on disjunct subsets resampled from the database, forming the cross-validation ensemble of models. The leave-one-out (LOO) ensemble of networks trained on all subsets leaving out one training example is an attractive -- though computationally expensive -- vehicle for generalization assessment of a neural network model. For the conventional neural net approaches unlearning of examples is not possible, and one basically has to train the full ensemble of networks, making the approach computationally unfeasible. This paper suggests the use of linear unlearning of examples to approximate the computationally expensive LOO cross-validation technique. It is assumed that unlearning of a single example only affects the network weights slightly. Under this hypothesis the change in the network parameters within the quadratic approximation of the network cost function is estimated. Using the ensemble an estimator for the test error of a regularized network is derived. The possibility of employing the ensemble of networks produced by the cross-validation scheme for constructing an ensemble predictor is analyzed. Considering a linear combination of networks, it is shown that the generalization performance is identical to that of using a single network trained on the full s…
Mathematics Subject Classification ID: 65C99
Mathematics Subject Classification ID: 62J02
Mathematics Subject Classification ID: 62M10
Mathematics Subject Classification ID: 62M20
zbMATH DE Number: 934192
zbMATH Keywords: nonlinear regression
zbMATH Keywords: neural network
zbMATH Keywords: input-output distribution
zbMATH Keywords: cross-validation
zbMATH Keywords: linear unlearning
zbMATH Keywords: quadratic approximation
zbMATH Keywords: time series prediction
MaRDI profile type: Publication
cites work: Fitting autoregressive models for prediction
cites work: Q4031423
cites work: Jackknifing in Nonlinear Regression
cites work: Q4724470
cites work: Q4840562
cites work: Q4066417
cites work: Bibliography on estimation of misclassification
cites work: Q3996207
full work available at URL: https://doi.org/10.1007/bf02124747
OpenAlex ID: W1994567799
description: scientific article; zbMATH DE number 934192