变量与特征选择综述：过滤器与包装器方法比较

需积分: 7 93 浏览量更新于2024-07-28 收藏 644KB PDF 举报

身份认证购VIP最低享 7 折!

领优惠券(最高得80元）

本文档由Isabelle Guyon 和 André Elisseeff 合著，标题为《变量和特征选择：一个介绍》（An Introduction to Variable and Feature Selection），发表于2003年的《机器学习研究》(Journal of Machine Learning Research) 第3期，1157-1182页。随着数据集规模的爆炸性增长，尤其是在互联网文档文本处理、基因表达阵列分析以及组合化学等领域，特征选择成为了研究的热点。这些领域的数据往往包含数百万甚至数十亿的变量，因此有效的特征选择至关重要。特征选择的目标主要有三个：首先，通过减少冗余或无关的特征，提升预测模型的性能。这有助于提高模型的准确性和泛化能力，特别是在高维数据中，过多的特征可能导致过拟合问题。其次，特征选择可以简化模型，降低计算复杂度和成本，使得在资源有限的情况下也能构建和应用高效预测器。最后，通过选择对数据生成过程有解释力的特征，有助于深入理解问题的内在机制。文章作者总结了两种常见的特征选择方法：过滤法（Filter）和包裹法（Wrapper）。过滤法是先独立评估每个特征与目标变量之间的关联性，然后根据预设的评价指标（如卡方检验、互信息等）筛选出最相关的特征。这种方法简单快速，但可能会忽略特征间的交互效应。相比之下，包裹法则是通过构建和评估多个包含不同特征子集的模型来选择最佳特征组合，这种方法更全面，但计算代价较高。文中提到的这个专题论文集涵盖了特征选择的多个方面，包括但不限于特征选择的理论基础、评估策略、特征重要性的衡量方法、以及特征选择方法在实际应用中的挑战和改进策略。它为研究人员提供了一个全面的框架，帮助他们在面对大规模数据时，如何有效地进行特征选择，从而优化模型性能并促进领域知识的理解。通过阅读这篇综述，读者可以了解到如何在具体问题中权衡速度、精度和可解释性，以找到最适合的特征子集。

资源详情

资源推荐

AN INTRODUCTION TO VARIABLE AND FEATURE SELECTION

where the bar notation stands for an average over the index k. This coefﬁcient is also the cosine

between vectors x

and y, after they have been centered (their mean subtracted). Although the R(i)

is derived from R (i) it may be used without assuming that the input values are realizations of a

random variable.

In linear regression, the coefﬁcient of determination, which is the square of R(i), represents the

fraction of the total variance around the mean value ¯y that is explained by the linear relation between

and y. Therefore, using R(i)

as a variable ranking criterion enforces a ranking according to

goodness of linear ﬁt of individual variables.

The use of R(i)

can be extended to the case of two-class classiﬁcation, for which each class

label is mapped to a given value of y, e.g., ±1. R(i)

can then be shown to be closely related to

Fisher’s criterion (Furey et al., 2000), to the T-test criterion, and other similar criteria (see, e.g.,

et al., 1999, Tusher et al., 2001, Hastie et al., 2001). As further developed in Section 6, the link

to the T-test shows that the score R(i) may be used as a test statistic to assess the signiﬁcance of a

variable.

Correlation criteria such as R(i) can only detect linear dependencies between variable and tar-

get. A simple way of lifting this restriction is to make a non-linear ﬁt of the target with single

variables and rank according to the goodness of ﬁt. Because of the risk of overﬁtting, one can alter-

natively consider using non-linear preprocessing (e.g., squaring, taking the square root, the log, the

inverse, etc.) and then using a simple correlation coefﬁcient. Correlation criteria are often used for

microarray data analysis, as illustrated in this issue by Weston et al. (2003).

2.3 Single Variable Classiﬁers

As already mentioned, using R(i)

as a ranking criterion for regression enforces a ranking according

to goodness of linear ﬁt of individual variables. One can extend to the classiﬁcation case the idea of

selecting variables according to their individual predictive power, using as criterion the performance

of a classiﬁer built with a single variable. For example, the value of the variable itself (or its negative,

to account for class polarity) can be used as discriminant function. A classiﬁer is obtained by setting

a threshold θ on the value of the variable (e.g., at the mid-point between the center of gravity of the

two classes).

The predictive power of the variable can be measured in terms of error rate. But, various other

criteria can be deﬁned that involve false positive classiﬁcation rate fpr and false negative classiﬁ-

cation rate fnr. The tradeoff between fpr and fnr is monitored in our simple example by varying

the threshold θ. ROC curves that plot “hit” rate (1-fpr) as a function of “false alarm” rate fnr are

instrumental in deﬁning criteria such as: The “Break Even Point” (the hit rate for a threshold value

corresponding to fpr=fnr) and the “Area Under Curve” (the area under the ROC curve).

In the case where there is a large number of variables that separate the data perfectly, ranking

criteria based on classiﬁcation success rate cannot distinguish between the top ranking variables.

One will then prefer to use a correlation coefﬁcient or another statistic like the margin (the distance

between the examples of opposite classes that are closest to one another for a given variable).

6. A variant of this idea is to use the mean-squared-error, but, if the variables are not on comparable scales, a comparison

between mean-squared-errors is meaningless. Another variant is to use R(i) to rank variables, not R(i)

. Positively

correlated variables are then top ranked and negatively correlated variables bottom ranked. With this method, one

can choose a subset of variables with a given proportion of positively and negatively correlated variables.

1161

GUYON AND ELISSEEFF

The criteria described in this section extend to the case of binary variables. Forman (2003)

presents in this issue an extensive study of such criteria for binary variables with applications in text

classiﬁcation.

2.4 Information Theoretic Ranking Criteria

Several approaches to the variable selection problem using information theoretic criteria have been

proposed (as reviewed in this issue by Bekkerman et al., 2003, Dhillon et al., 2003, Forman, 2003,

Torkkola, 2003). Many rely on empirical estimates of the mutual information between each variable

and the target:

I (i) =

p(x

,y)log

p(x

,y)

p(x

)p(y)

dxdy , (3)

where p(x

) and p(y) are the probability densities of x

and y, and p(x

,y) is the joint density. The

criterion I (i) is a measure of dependency between the density of variable x

and the density of the

target y.

The difﬁculty is that the densities p(x

), p(y) and p(x

,y) are all unknown and are hard to

estimate from data. The case of discrete or nominal variables is probably easiest because the integral

becomes a sum:

I(i) =

∑

P(X = x

,Y = y)log

P(X = x

,Y = y)

P(X = x

)P(Y = y)

. (4)

The probabilities are then estimated from frequency counts. For example, in a three-class

problem, if a variable takes 4 values, P(Y = y) represents the class prior probabilities (3 fre-

quency counts), P(X = x

) represents the distribution of the input variable (4 frequency counts),

and P(X = x

,Y = y) is the probability of the joint observations (12 frequency counts). The estima-

tion obviously becomes harder with larger numbers of classes and variable values.

The case of continuous variables (and possibly continuous targets) is the hardest. One can

consider discretizing the variables or approximating their densities with a non-parametric method

such as Parzen windows (see, e.g., Torkkola, 2003). Using the normal distribution to estimate

densities would bring us back to estimating the covariance between X

and Y, thus giving us a

criterion similar to a correlation coefﬁcient.

3 Small but Revealing Examples

We present a series of small examples that outline the usefulness and the limitations of variable

ranking techniques and present several situations in which the variable dependencies cannot be

ignored.

3.1 Can Presumably Redundant Variables Help Each Other?

One common criticism of variable ranking is that it leads to the selection of a redundant subset. The

same performance could possibly be achieved with a smaller subset of complementary variables.

Still, one may wonder whether adding presumably redundant variables can result in a performance

gain.

1162

剩余25页未读，继续阅读

wangdelp

粉丝: 0
资源: 10

变量与特征选择综述：过滤器与包装器方法比较

SVMs with Various Feature Selection

An Introduction to Variable and Feature Selection

在C#中，要将通过属性表勾选的高亮feature添加到featureSelection.SelectionSet

from sklearn. feature_selection import SelectKBest

featureselection.selectionset 怎么获取已经被选择的feature

from sklearn.feature_selection import SelectKBest

feature_importances_和feature_selection 的区别

feature_selection_ga

sklearn.feature_selection怎么安装

from sklearn.feature_selection import SelectKBest from sklearn.feature_selection import chi2

AttributeError: module 'statsmodels.api' has no attribute 'feature_selection'

用python写一段回归特征消去方法（Recursive feature selection based on support vector machine)优化的SVM代码

mv_lasso_feature_selection(mvdat2) #LASSO Performing feature selection # A tibble: 0 × 2 # … with 2 variables: exposure <chr>, b <dbl> # ℹ Use `colnames()` to see all variable names

sklearn.feature_selection

sklearn.feature_selection f_regression

ModuleNotFoundError: No module named 'sklearn.feature_selection'

sklearn.feature_selection.rfe

AttributeError: module 'sklearn' has no attribute 'feature_selection'

Sandcastle.addToggleButton('Per-feature selection', false, function (checked) { picking = checked; if (!picking) { unselectFeature(selectedFeature); } }); 这一段代码怎么在自己的项目中使用

sklearn.feature_selection.RFECV参数

最新资源