SGB-ELM：一种改进的随机梯度提升集成极端学习机方法
61 浏览量
更新于2024-07-14
收藏 1.96MB PDF 举报
SGB-ELM是一项针对极端学习机（ELM）的创新研究，它提出了一种先进的集成策略，是随机梯度提升技术在ELM框架中的应用。该论文由华国、姬奎王、魏傲和于林何四位作者共同完成，他们分别来自兰州财经大学信息工程学院和深圳大学计算机科学与软件工程学院。
传统的ELM是一种单层前馈神经网络，通过随机矩阵和固定权重的方式进行训练，具有快速训练速度和高预测性能的特点。然而，为了提高其泛化能力和稳健性，SGB-ELM并没有直接将随机梯度提升算法融入到ELM的基本流程中，而是采取了一种递进式的方法。在这个新型集成方法中，SGB-ELM构建了一系列弱ELM模型，每个个体模型都是通过迭代的方式，利用随机梯度下降来调整参数，以逐步减少剩余误差。
具体来说，SGB-ELM的流程包括以下几个关键步骤：
1. 初始化：首先创建一个基础的ELM模型作为初始成员。
2. 弱学习：每次迭代中，选择一部分训练样本并计算残差，这些残差反映了当前模型在特定数据集上的性能不足。
3. 更新：根据残差，训练一个新的弱ELM模型，目标是减小剩余的预测误差。这个新模型通常会在特征空间中找到一个更好的决策边界。
4. 整合：将新的弱ELM模型加入到原始模型的集成中，通过加权平均或投票的方式整合所有模型的预测结果，以提高整体性能。
5. 重复：重复上述过程直到满足停止条件，如预设的迭代次数或者剩余误差达到可接受的阈值。
SGB-ELM的优势在于它结合了ELM的简单性和随机梯度提升的灵活性，能够在保持快速训练的同时，通过集成多个弱模型来提升预测精度和鲁棒性。此外，由于采用的是随机采样策略，它对大规模数据集有较好的适应性，并且在处理非线性和复杂问题时显示出优越性。
SGB-ELM是一种创新的集成学习方法，它在保持ELM高效性的基础上，通过引入随机梯度提升技术，为解决实际问题提供了强大的工具，对于提升机器学习模型在实际场景中的表现具有重要意义。在未来的研究中，SGB-ELM可能会激发更多的研究者探索更多基于随机梯度提升的ELM变体，进一步推动机器学习领域的理论和实践发展。
Computational Intelligence and Neuroscience
learning algorithms []. At the meantime, ELM also produces
good generalization performance. It has been veried that
ELM can achieve the equal generalization performance with
the typical Support Vector Machine algorithm [].
2.2. Stochastic Gradient Boosting. Stochastic gradient boost-
ing scheme was proposed by Friedman in [], and it is
a variant of the gradient boosting method presented in
[]. Given a training set {(x
,y
)}
=1
,thegoalistolearna
hypothesis
(x)that maps x to y and minimizes the training
loss as follows:
(
x
)
=arg min
𝐾
(x)
=1
y
,
x
,
()
where (⋅,⋅)isthelossfunctionwhichevaluatesthedierence
between the predicted value and the target and K denotes
the number of iterations. In boosting mechanism, K additive
individual learners are trained sequentially by
(
x
)
=arg min
𝑘
(x)
=1
y
,
−1
x
+
x
()
and
(
x
)
=
−1
(
x
)
+
(
x
)
,
()
where = 1,2,⋅⋅⋅,. It is shown that the optimization
problem depends much on the loss function and becomes
unsolvable when (⋅,⋅)is complex. Creatively, gradient boost-
ing constructs the weak individuals based on the pseudo
residuals, which are the gradient of loss function with respect
to the model values predicted at the current learning step. For
instance, let 𝜖
()
be the pseudo residual of the th sample at the
th iteration written as
𝜖
()
=−
y
,y
y
y
=
𝑘−1
(x
𝑖
)
,
()
and thus the th weak learner
(x)is trained by
(
x
)
=arg min
𝑘
(x)
=1
𝜖
()
,
x
.
()
As gradient boosting constructs additive ensemble model
by sequentially tting a weak individual learner to the current
pseudo-residuals of whole training dataset at each iteration,
it costs much training time and may suer from overtting
problem. In view of that, a minor modication named
stochastic gradient boosting is proposed to incorporate some
randomization to the procedure. Specically, at each iteration
a randomly selected subset instead of the full training dataset
isusedtottheindividuallearnerandcomputethemodel
updateforthecurrentiteration.Namely,let{()}
1
be a
random permutation of the integers {1,2,⋅⋅⋅,},andthe
subset with size
<of the entire training dataset can be
given by {(x
()
,y
()
)}
=1
.Furthermore,theth weak learner
using the stochastic gradient boosting ensemble scheme is
trained by solving the following optimization problem as
∗
(
x
)
=arg min
∗
𝑘
(x)
=1
𝜖
()
()
,
∗
x
()
.
()
Given the base learner
0
(x)which is trained by the initial
training dataset, the nal ensemble learning model con-
structed by stochastic gradient boosting scheme predicts an
unknown testing instance x as follows:
(
x
)
=
0
(
x
)
+
=1
∗
(
x
)
.
()
Stochastic gradient boosting is also considered as a special
linear search optimization algorithm, which makes the newly
added individual learner t the fastest descent direction of
partial training loss at each learning step.
3. Stochastic Gradient Boosting-Based
Extreme Learning Machine (SGB-ELM)
SGB-ELM is a novel hybrid learning algorithm, which intro-
duces the stochastic gradient boosting method into ELM
ensemble procedure. As boosting mechanism focuses on
gradually reducing the training residuals at each iteration
and ELM is a special multiparameters network (for classi-
cation tasks particularly), instead of combining the ELM
and stochastic gradient boosting primitively, we design an
enhanced training scheme to alleviate possible overtting in
our proposed SGB-ELM algorithm. e detailed implemen-
tation of SGB-ELM is presented in Algorithm , where the
determination of optimal output weights for each individual
ELM learner is illustrated in Algorithm accordingly.
ere are many existing second-order approximation
methods including sequential quadratic programming (SQP)
[] and majorization-minimization algorithm (MM) [].
SQP is an eective method for nonlinearly constrained
optimization by solving quadratic subproblems. MM aims
to optimize the local alternative objective which is easier
to solve in comparison with the original cost function.
Instead of using second-order approximation directly, SGB-
ELM designs an optimization criterion for the output-layer
weights of each individual ELM. In view of that, quadratic
approximation is merely employed as an optimization tool in
SGB-ELM.
In SGB-ELM, the key issue is to determine the optimal
output-layer weights of each weak individual ELM, which is
expected to further decrease the training loss and meanwhile
keep a simple network structure. Consequently, we design a
learning objective considering not only the tting ability for
training instances but also the complexity of our ensemble
model as follows:
()
=
=1
y
,y
+
=1
,
()
where (⋅,⋅) is a dierentiable loss function that measures
the dierence between the predicted output y
and the target
剩余14页未读，继续阅读
2020-06-24 上传
2020-05-17 上传
点击了解资源详情
2021-04-29 上传
2021-04-29 上传
2021-04-29 上传
2021-03-02 上传
2021-07-09 上传
2021-06-28 上传
weixin_38687343
- 粉丝: 6
- 资源: 903
我的内容管理
展开
最新资源
- 前端协作项目：发布猜图游戏功能与待修复事项
- Spring框架REST服务开发实践指南
- ALU课设实现基础与高级运算功能
- 深入了解STK：C++音频信号处理综合工具套件
- 华中科技大学电信学院软件无线电实验资料汇总
- CGSN数据解析与集成验证工具集：Python和Shell脚本
- Java实现的远程视频会议系统开发教程
- Change-OEM: 用Java修改Windows OEM信息与Logo
- cmnd：文本到远程API的桥接平台开发
- 解决BIOS刷写错误28：PRR.exe的应用与效果
- 深度学习对抗攻击库：adversarial_robustness_toolbox 1.10.0
- Win7系统CP2102驱动下载与安装指南
- 深入理解Java中的函数式编程技巧
- GY-906 MLX90614ESF传感器模块温度采集应用资料
- Adversarial Robustness Toolbox 1.15.1 工具包安装教程
- GNU Radio的供应商中立SDR开发包：gr-sdr介绍
