Python实现adaboost

AdaBoost算法是一种集成学习方法，它能将多个弱分类器组合成一个强分类器。Python实现AdaBoost算法，可以使用第三方库scikit-learn，也可以手动实现。

使用scikit-learn的实现方法如下：

1.导入数据集和AdaBoost分类器：

from sklearn.datasets import make_classification
from sklearn.ensemble import AdaBoostClassifier

2.生成数据集：

X, y = make_classification(n_samples=1000, n_features=10, n_informative=4, n_redundant=0, random_state=0)

3.定义AdaBoost分类器：

clf = AdaBoostClassifier(n_estimators=100, random_state=0)

4.训练分类器：

clf.fit(X, y)

5.预测：

y_pred = clf.predict(X)

手动实现的方法可以参考以下步骤：

1.定义弱分类器：

def stump(X, y, weights):
    n_samples, n_features = X.shape
    feature_idx = 0
    threshold = 0
    polarity = 1
    min_error = float('inf')
    for i in range(n_features):
        feature_values = np.expand_dims(X[:, i], axis=1)
        unique_values = np.unique(feature_values)
        for j in range(len(unique_values)):
            threshold_tmp = unique_values[j]
            for k in [-1, 1]:
                yhat = np.ones(y.shape)
                yhat[X[:, i] < threshold_tmp] = -1
                error = weights.dot(yhat != y)
                if error < min_error:
                    min_error = error
                    feature_idx = i
                    threshold = threshold_tmp
                    polarity = k
    return feature_idx, threshold, polarity

2.定义AdaBoost算法：

class AdaBoost:
    def __init__(self, n_clf=5):
        self.n_clf = n_clf
        self.clfs = []

    def fit(self, X, y):
        n_samples, _ = X.shape
        weights = np.full(n_samples, 1 / n_samples)
        for _ in range(self.n_clf):
            clf = stump(X, y, weights)
            yhat = np.ones(y.shape)
            feature_idx, threshold, polarity = clf
            yhat[X[:, feature_idx] < threshold] = -1
            error = weights.dot(yhat != y)
            alpha = 0.5 * np.log((1 - error) / error + 1e-10)
            weights *= np.exp(-alpha * y * yhat)
            weights /= np.sum(weights)
            self.clfs.append((clf, alpha))

    def predict(self, X):
        clf_preds = []
        for clf, alpha in self.clfs:
            feature_idx, threshold, polarity = clf
            clf_pred = np.ones(X.shape[0])
            clf_pred[X[:, feature_idx] < threshold] = -1
            clf_preds.append(polarity * clf_pred * alpha)
        y_pred = np.sum(clf_preds, axis=0)
        return np.sign(y_pred)

3.训练和预测：

clf = AdaBoost(n_clf=100)
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)