精通Python的XGBoost实战指南

5星 · 超过95%的资源需积分: 10 198 浏览量更新于2024-07-18 1 收藏 2.07MB PDF 举报

"XGBoost与Python实战.pdf" 本书《XGBoost与Python》是机器学习进阶系列的一部分，旨在引导读者快速掌握Python中的梯度提升技术。XGBoost是一个高效且灵活的梯度提升库，它专为解决各种机器学习问题而设计。书中将详细介绍如何使用XGBoost库构建和评估梯度提升模型。尽管梯度提升背后有深厚的数学原理，但作为工具使用者，你无需深入了解这些理论即可在实际项目中发挥其威力。对于应用开发者来说，梯度提升领域相对较浅，通过努力学习，你可以迅速掌握并开始做出具有实质性影响的贡献。本书的目标就是帮助你实现这一目标。书中的免责声明指出，本书内容仅供教育目的，如需应用书中理念，读者需自行承担相应责任。作者已尽最大努力确保书中信息的准确性，但对因错误或遗漏造成的任何损失、损害或中断不承担责任。未经作者书面许可，本书的任何部分不得复制或以任何形式传播。版权信息显示，本书由Jason Brownlee于2017年创作，所有权利保留，并指定了版本号v1.5。目录预览了书中的主要内容，包括介绍、书的组织结构、阅读本书的要求、预期学习成果以及本书的非定位等章节。通过阅读本书，你将了解： 1. 书的组织结构：介绍每一章的主题和内容安排。 2. 阅读本书的要求：可能需要的基础知识和软件环境。 3. 你的学习成果：预期通过阅读本书能掌握的技能和理解的概念。 4. 本书不是什么：明确本书不涉及的领域，以便读者有清晰的期待。内容涵盖梯度提升的基本概念、XGBoost库的安装和使用、数据预处理、模型训练、调优技术、特征重要性分析、模型评估以及在实际项目中的应用示例。这些内容将帮助你深入理解XGBoost的工作原理，并在实践中有效地利用这个强大的工具。这本书是机器学习从业者和爱好者深入学习XGBoost和Python结合使用的理想指南，无论你是初学者还是经验丰富的开发者，都能从中受益。通过本书的学习，你将能够运用梯度提升技术解决复杂的数据问题，并在实际的机器学习项目中创造价值。

2.4. How Gradient Boosting Works 10

Note that thi s stagewise strategy is di↵erent from stepwise approaches that

readjust previously entered terms when new ones are added.

– Greedy Function Approximation: A Gradient Boosting Machine,1999.

The g ener a l i za ti o n all owed arbitrary di↵erentiable loss functions to be used, expanding the

technique beyond binary classiﬁcation problems to support regression, multiclass classiﬁcation

and more.

2.4 How Gradient Boosting Works

Gradient boosting involves three elements:

1. Alossfunctiontobeoptimized.

2. Aweaklearnertomakepredictions.

3. An additive model to add weak learners to minimize the loss fu n ct i o n .

2.4.1 Loss Function

The loss function used depends on the type of problem being solved. It must be di↵erentiable,

but many standard loss functions are supported and you can deﬁne your own. For example,

regression may use a squared error and classiﬁcation may use logarithmic loss. A beneﬁt of the

gradient boosting framework is that a new boosting algorithm does not have to be derived for

each loss function that may want to be used , instead , it is a generic enough framework that any

di↵erentiable loss function can be used.

2.4.2 Weak Learner

Decision trees are used as the weak learner in gradient boosting. Speciﬁcally regression trees

are used that output real values for splits and whose output can be added together, allowing

subsequent models outputs to be added and correct the residuals in the predictions. Trees are

constructed in a greedy manner, choosing the best split points based on purity scores like Gini

or to minimize the loss.

Initially, such as in the case of AdaBoost, very short decision trees were used that only had

asinglesplit,calledadecisionstump. Largertreescanbeusedgenerallywith4-to-8levels. Itis

common t o constrain the weak learners in speciﬁc ways, such as a maximum number of layers,

nodes, splits or leaf nodes. This is to ensure that the learners remain weak, but can still be

constructed in a greedy manner.

2.4.3 Additive Model

Trees are added one at a time, and existing trees in the model are not changed. A gradient

descent procedure is used to minimize the loss when adding trees. Traditionally, gradient descent

is used to minimize a set of parameters, such as the coeﬃcients in a regression equation or

2.5. Imp r ovements to Basic Gradient Boosting 11

weights in a neural network. After calculating error or loss, the weights are updated to minimize

that error.

Instead of par am et er s, we have weak learner sub-models or more speciﬁ call y decisi o n trees.

After calculating the loss, to perform the gradient descent procedure, we must add a tree to

the model that reduces the loss (i.e. follow the gradient). We do this by parameterizing the

tree, then modify the par a met er s of the tr ee and move in the right direction by (reducing the

residual loss. Gen er al l y this approach is called funct i on a l gradient descent or gradi ent descent

with fun ct i ons.

One way to produce a weighted combination of classiﬁers which optimizes [the

cost] is by gradient descent in function space

– Boosting Algorithms as G radient Descent in Function Space,1999.

The output for the new tree is then added to the output of the existing sequence of trees in

an e↵ort to correct or improve the ﬁnal output of the model. A ﬁxed number of trees are added

or training st op s once loss reaches an acceptable level or no lon g er improves on an externa l

validation dataset.

2.5 Improvements to Basic Gradient Boosting

Gradient boosting is a greedy algorithm and can over ﬁt a tr ai ni n g dataset qu i ckly. It can beneﬁt

from regularization methods that penalize various parts of the algori t h m an d gen er al l y i m p rove

the performa n ce of the algorithm by reducing overﬁtting. In this this section we will look at 4

enhancements to basic gradient boosting:

1. Tree Constraints.

2. Shrinkage.

3. Random Sampling.

4. Penalized Learning.

2.5.1 Tree Constraints

It is important th at t h e weak learners have skil l b u t rem ai n weak. Ther e are a number of ways

that the trees c an be co n st r ai n ed . A good general heuristic is that the more constr ai n ed tree

creation is, the more trees you will need in the model, and the reverse, where less constrained

individual trees, t h e fewer trees t h a t will be required. Below are some constraints th a t can be

imposed on the construction of decision trees:

Number of trees, generally adding more trees to the model can be very slow to overﬁt.

The advice is to keep adding tr ees until no further improvement is observed.

Tree depth, deep er trees are more complex trees and shorter trees are preferred. Generally,

better results are seen with 4-8 levels.

2.5. Imp r ovements to Basic Gradient Boosting 12

Number of nodes or number of leaves, like depth, this can constrain the size of the tree,

but is not constrained to a symmetrical structure if other constraints are used.

Number of observations per spl i t imposes a minimum constrai nt on the amount of training

data at a training node before a split can be considered

Minimum improvement to loss is a constraint on the improvement of any split added to a

tree.

2.5.2 Weighted Upda te s

The prediction s of each tree are a d d ed together sequentially. The contribution of each tree t o

this sum can be weighted to slow down the learning by the algorithm. This weighting is called

ashrinkageoralearningrate.

Each update is simply scaled by the value of the “learning rate paramet er v”

– Greedy Function Approximation: A Gradient Boosting Machine,1999.

The e↵ect is that learnin g is slowed down, in turn require more trees t o be added to the

model, in turn taking longer to train, providing a conﬁgur at i on trade-o↵ between the number of

trees and learni ng rate.

Decreasing the value of v [the learning rate] increases the best value for M [the

number of trees].

– Greedy Function Approximation: A Gradient Boosting Machine,1999.

It is common to have small values in the range of 0.1 to 0.3, as well as value s less than 0.1.

Similar to a learning rate in stochastic optimization, shrinkage reduces the

inﬂuence of each individual tree and leaves space for future trees to improve the

model.

– Stochastic Gradient Boosting,1999.

2.5.3 Stochastic Gradient Boo st ing

A big insight into bagging ensembles and random forest was allowing trees to be gre ed i l y created

from subsamples of the train i n g dataset. This same beneﬁt can be used to reduce the co rrel a t i on

between the trees in the seq u en ce i n gradient boosting models. This variation of boosting is

called stochastic gradient boosting.

at each iteration a subsample of the training data is drawn at random (without

replacement) from the full training dataset. The randomly selected subsample is

then used, instead of the full sample, to ﬁt the base learner.

– Stochastic Gradient Boosting,1999.

剩余114页未读，继续阅读

zwxeye

粉丝: 12
资源: 46

精通Python的XGBoost实战指南

machine_learning_mastery_with_python.zip

Python数据分析与机器学习-Xgboost集成算法

[machine_learning_mastery系列]Machine_Learning_Mastery_With_Python.pdf

[machine_learning_mastery系列]deep_learning_with_python.pdf(with code)

machine_learning_mastery_with_python_python_machinelearning_

Machine_Learning_Mastery_With_Python_－_Understand_Your_Data，_Create_Accurate_Models_and_Work_Projects_End－To－End.pdf ....pdf

machine_learning_mastery_with_python

machine_learning_mastery_with_python_sample_slightlybcn_machinel

Machine_Learning_Mastery_With_Python _Understand_Your_Data

[machine_learning_mastery系列]long-short-term-memory-networks-with-python.pdf

最新资源