![](https://csdnimg.cn/release/download_crawler_static/13056739/bg6.jpg)
STATISTICAL VIEW OF CHEMOMETRICS REGRESSION TOOLS
with
s = ave(yx)
(15)
and I being the p x p identity matrix. The inverse
of the (possibly) ill-conditioned predictor-variable
covariance matrix V is thus stabilized by adding to
V a multiple of I. The degree of stabilization is reg-
ulated by the value of the "ridge" parameter A > 0.
A value of A = oo results in the model being the
response mean y = 0, whereas A = 0 gives rise to
the unregularized OLS estimates. A value for A in
any particular situation is generally obtained by con-
sidering it to be a meta parameter of the procedure
and estimating it through some model-selection pro-
cedure such as CV. Since here the response values
{Yi}l do enter linearly in the model estimates {.9i}j,
any of the competing model-selection criteria can
also be straightforwardly applied (see Golub, Heath,
and Wahba 1979).
3. A COMPARISON OF PCR, PLS, AND RR
From their preceding algorithmic descriptions, it
might appear that PCR, PLS, and RR are very dif-
ferent procedures leading to quite different model
estimates. In this section we provide a heuristic com-
parison that suggests that they are, in fact, quite
similar, in that they are all attempting to achieve the
same operational goal in slightly different ways. That
goal is to bias the solution coefficient vector a (5)
away from directions for which the projected sample
predictor variables have small spread; that is,
var(aTx/lal) = ave(aTx/lal)2 = small, (16)
where the average is over the training sample.
This comparison consists of regarding the regres-
sion procedure as a two-step process as in VSS (Stone
and Brooks 1990); first a K-dimensional subspace of
p-dimensional Euclidean space is defined, and then
the regression is performed under the restriction that
the coefficient vector a lies in that subspace:
K
a= akck, (17)
k=l
where the unit vectors {ck}f span the prescribed
subspace with ckck = 1. The regression procedures
can be compared by the way in which they define
the subspace {ck}f and the manner in which the
(constrained) regression is performed.
First, consider OLS in this setup. Here the sub-
space is defined by the (single) unit vector that max-
imizes the sample correlation (squared) between the
response and the corresponding linear combination
of the predictor variables
COLS = argmax corr2(y, CTX); (18)
CTc= 1
the OLS solution is then a simple least squares
regression of y on COLSX,
YOLS = [ave(ycOLsx)/aveO(coLs))2] OLSX. (19)
RR can also be cast into this framework. As in
OLS, the subspace is defined by a single unit vector,
but the criterion that defines that vector is somewhat
different:
var(cTx)
CRR = argmax corr2(y, cTx) var(cTx)
cTc=l var(crx) + A'
(20)
where A is the ridge parameter [(13)-(14)]. The ridge
solution is then taken to be a (shrinking) ridge regres-
sion of y on CRRX with the same value for the ridge
parameter
ave (yCRX) T
YRR ae( RRX)
_ave(cTRX)2 + R
(21)
(See Appendix.)
PCR defines a sequence of K-dimensional sub-
spaces each spanned by the first K eigenvectors (9)
of V (8). Thus each ck (1 < k < R) is the solution
to
c
ck(PCR) argmax var(cTx).
{TVC = 0} -1
TC= 1
(22)
The first constraint in (22) (V orthogonality) ensures
that the linear combinations associated with the dif-
ferent solution vectors are uncorrelated over the
training sample
corr(ckx, cfx) = 0,
k : 1.
(23)
As a consequence of this and the criterion (22), they
also turn out to be orthogonal cTc, = 0, k = 1. The
Kth PCR model is given by a least squares regression
of the response on the K linear combinations
{ckx}f. Since they are uncorrelated (23), this reduces
to the sum of univariate regressions on each one (10).
PLS regression also produces a sequence of K-
dimensional subspaces spanned by successive unit
vectors, and then the Kth PLS solution is obtained
by a least squares fit of the response onto the cor-
responding K-linear combinations in a strategy sim-
ilar to PCR. The only difference from PCR is in the
criterion used to define the vectors that span the K-
dimensional subspace and hence the corresponding
linear combinations. The criterion that gives rise to
PLS (Stone and Brooks 1990) is
ck(PLS) = argman corr2(y, cTx)var(cTx). (24)
{cTVc = 0}- 1
crc=1
As with PCR the vectors ck(PLS) are constrained to
be mutually V orthogonal so that the corresponding
linear combinations are uncorrelated over the train-
TECHNOMETRICS, MAY 1993, VOL. 35, NO. 2
113
This content downloaded from
106.37.205.239 on Mon, 26 Oct 2020 13:05:23 UTC
All use subject to https://about.jstor.org/terms