Shrinkage structure of partial least squares (Q2742761)

The linear regression model \(y=X b +\varepsilon\) is considered, where \(b\) is an unknown \(p\times 1\)-parameter vector and \(\varepsilon\) is an \(n \times 1\)-vector of noise terms, \(X\) denotes the \(n\times p\) predictor matrix with rows \(x_1,\ldots, x_n\), and \(y\) denotes the vector of responses. Let \(K_m\) be the Krylov space NEWLINE\[NEWLINEK_m ={\text span}\{ X^T y, (X^T X) X^T y, \ldots, (X^T X)^{m-1} X^T y\}.NEWLINE\]NEWLINE Following estimate for \(b\) is considered. The partial least squares regression estimate (PLS) \(\hat b_{PLS}\) solves the constrained optimization problem: minimize \(\|y -Xb\|_2,\;b\in K_m.\) The authors present an explicit formula for the PLS, that is they give the following estimate NEWLINE\[NEWLINE\hat b_{PLS} = R_m (R_m^T X^T X R_m)^{-1} R_m^T X^T y, NEWLINE\]NEWLINE where \(R_m\) is a \(p\times m\) matrix with columns that span the subspace \(K_m.\) The authors characterize PLS in terms of the scaling (shrinkage) along each eigenvector of the predictor correlation matrix. This characterization is useful in providing a link between PLS and other shrinkage estimators such as principal components regression and ridge regression, thus facilitating a direct comparison of PLS with these methods. The authors give a detailed analysis of the shrinkage structure of PLS and several new results regarding the nature and extent of shrinkage.