![](https://csdnimg.cn/release/download_crawler_static/4292209/bg5.jpg)
AN INTRODUCTION TO VARIABLE AND FEATURE SELECTION
where the bar notation stands for an average over the index k. This coefficient is also the cosine
between vectors x
i
and y, after they have been centered (their mean subtracted). Although the R(i)
is derived from R (i) it may be used without assuming that the input values are realizations of a
random variable.
In linear regression, the coefficient of determination, which is the square of R(i), represents the
fraction of the total variance around the mean value ¯y that is explained by the linear relation between
x
i
and y. Therefore, using R(i)
2
as a variable ranking criterion enforces a ranking according to
goodness of linear fit of individual variables.
6
The use of R(i)
2
can be extended to the case of two-class classification, for which each class
label is mapped to a given value of y, e.g., ±1. R(i)
2
can then be shown to be closely related to
Fisher’s criterion (Furey et al., 2000), to the T-test criterion, and other similar criteria (see, e.g.,
et al., 1999, Tusher et al., 2001, Hastie et al., 2001). As further developed in Section 6, the link
to the T-test shows that the score R(i) may be used as a test statistic to assess the significance of a
variable.
Correlation criteria such as R(i) can only detect linear dependencies between variable and tar-
get. A simple way of lifting this restriction is to make a non-linear fit of the target with single
variables and rank according to the goodness of fit. Because of the risk of overfitting, one can alter-
natively consider using non-linear preprocessing (e.g., squaring, taking the square root, the log, the
inverse, etc.) and then using a simple correlation coefficient. Correlation criteria are often used for
microarray data analysis, as illustrated in this issue by Weston et al. (2003).
2.3 Single Variable Classifiers
As already mentioned, using R(i)
2
as a ranking criterion for regression enforces a ranking according
to goodness of linear fit of individual variables. One can extend to the classification case the idea of
selecting variables according to their individual predictive power, using as criterion the performance
of a classifier built with a single variable. For example, the value of the variable itself (or its negative,
to account for class polarity) can be used as discriminant function. A classifier is obtained by setting
a threshold θ on the value of the variable (e.g., at the mid-point between the center of gravity of the
two classes).
The predictive power of the variable can be measured in terms of error rate. But, various other
criteria can be defined that involve false positive classification rate fpr and false negative classifi-
cation rate fnr. The tradeoff between fpr and fnr is monitored in our simple example by varying
the threshold θ. ROC curves that plot “hit” rate (1-fpr) as a function of “false alarm” rate fnr are
instrumental in defining criteria such as: The “Break Even Point” (the hit rate for a threshold value
corresponding to fpr=fnr) and the “Area Under Curve” (the area under the ROC curve).
In the case where there is a large number of variables that separate the data perfectly, ranking
criteria based on classification success rate cannot distinguish between the top ranking variables.
One will then prefer to use a correlation coefficient or another statistic like the margin (the distance
between the examples of opposite classes that are closest to one another for a given variable).
6. A variant of this idea is to use the mean-squared-error, but, if the variables are not on comparable scales, a comparison
between mean-squared-errors is meaningless. Another variant is to use R(i) to rank variables, not R(i)
2
. Positively
correlated variables are then top ranked and negatively correlated variables bottom ranked. With this method, one
can choose a subset of variables with a given proportion of positively and negatively correlated variables.
1161