自动机器学习与超参数优化：从黑盒优化到多精度策略

96 浏览量更新于2024-07-14 收藏 1.42MB PDF 举报

"AutoML - Chapter 01 - Hyperparameter Optimization (2018) - 计算机科学" 本章主要探讨了机器学习中的超参数优化（Hyperparameter Optimization，简称HPO）问题，特别是针对复杂且计算成本高昂的模型，如自动化机器学习框架和深度神经网络。随着这些技术的广泛应用，对HPO的研究再次受到了关注。作者Matthias Feurer和Frank Hutter首先介绍了基于模型自由方法的黑盒函数优化策略，这些方法在不考虑目标函数具体结构的情况下寻找最优解。黑盒优化通常包括随机搜索、网格搜索以及进化算法等，它们试图通过多次实验来找到最佳超参数组合。随后，章节重点转向了现代多精度方法（multi-fidelity methods）。由于许多现代机器学习应用的高计算需求，纯黑盒优化变得非常昂贵。因此，多精度方法利用低成本的近似版本（如小规模数据集或简化模型）来评估超参数设置的质量，从而在时间和性能之间取得平衡。这种方法能够显著减少优化过程所需的计算资源，提高效率。接着，章节讨论了贝叶斯优化（Bayesian optimization），这是一种高效且广泛应用的HPO方法。它通过构建概率模型来预测不同超参数设置下的性能，并利用这些模型指导接下来的实验选择，以最小化评估次数。最后，作者指出了HPO领域目前面临的开放问题和未来研究方向。这些问题可能包括如何处理大规模高维超参数空间、如何有效地利用计算资源、以及如何将先验知识纳入优化过程等。此外，随着模型复杂性的增加，如何设计适应性强、通用性好的自动调参策略也是研究者们需要关注的重点。该章节深入浅出地概述了HPO的主要方法和技术，并对未来的研究提出了挑战和展望，对于理解与实践AutoML和深度学习中的超参数优化具有重要价值。

1.3. BLACKBOX HYPERPARAMETER OPTIMIZATION 9

1.3.2 Bayesian Optimization

Bayesian optimization is a state-of-the-art optimization framework for the global

optimization of expensive blackbox functions, which recently gained traction in

HPO by obtaining new state-of-the-art results in tuning deep neural networks

for image classiﬁcation [140, 141], speech recognition [22] and neural language

modeling [105], and by demonstrating wide applicability to diﬀerent problem

settings. For an in-depth introduction to Bayesian optimization, we refer to the

excellent tutorials by Shahriari et al. [135] and Brochu et al. [18].

In this section we ﬁrst give a brief introduction to Bayesian optimization,

present alternative surrogate models used in it, describe extensions to condi-

tional and constrained conﬁguration spaces, and then discuss several important

applications to hyperparameter optimization.

Many recent advances in Bayesian optimization do not treat HPO as a black-

box any more, for example multi-ﬁdelity HPO (see Section 1.4), Bayesian opti-

mization with meta-learning (see Chapter 2), and Bayesian optimization taking

the pipeline structure into account [160, 159]. Furthermore, many recent devel-

opments in Bayesian optimization do not directly target HPO, but can often

be readily applied to HPO, such as new acquisition functions, new models and

kernels, and new parallelization schemes.

Bayesian Optimization in a Nutshell

Bayesian optimization is an iterative algorithm with two key ingredients: a prob-

abilistic surrogate model and an acquisition function to decide which point to

evaluate next. In each iteration, the surrogate model is ﬁtted to all observations

of the target function made so far. Then the acquisition function, which uses

the predictive distribution of the probabilistic model, determines the utility of

diﬀerent candidate points, trading oﬀ exploration and exploitation. Compared

to evaluating the expensive blackbox function, the acquisition function is cheap

to compute and can therefore be thoroughly optimized.

Although many acquisition functions exist, the expected improvement (EI) [72]:

E[I(λ)] = E[max(f

min

− Y, 0)] (1.2)

is common choice since it can be computed in closed form if the model prediction

Y at conﬁguration λ follow a normal distribution:

E[I(λ)] = (f

min

− µ(λ)) Φ



min

− µ(λ)



+ σφ



min

− µ(λ)



, (1.3)

where φ(·) and Φ(·) are the standard normal density and standard normal dis-

tribution function, and f

min

is the best observed value so far.

Figure 1.2 illustrates Bayesian optimization optimizing a toy function.

Surrogate Models

Traditionally, Bayesian optimization employs Gaussian processes [124] to model

the target function because of their expressiveness, smooth and well-calibrated

剩余35页未读，继续阅读

weixin_38514805

粉丝: 9
资源: 932

自动机器学习与超参数优化：从黑盒优化到多精度策略

Improving-Deep-Neural-Networks-Hyperparameter-tuning-Regularization-and-Optimization:我从不断完善的深度神经网络进行编程作业的解决方案

Hyperparameter Optimization using Hyperopt - Slides (2016)-计算机科学

Two-Phase Iteration for Value Function Approximation and Hyperparameter Optimization in Gaussian-Kernel-Based Adaptive Critic Design

scikit-nni：AutoML-使用Microsoft NNI的超参数搜索scikit-learn管道

awesome-automl-papers:自动化机器学习论文，文章，教程，幻灯片和项目的精选清单

ultraopt:Distributed Asynchronous Hyperparameter Optimization better than HyperOpt. 比HyperOpt更强的分布式异步超参优化库

Hyperparameter-Optimization-of-Machine-Learning-Algorithms:机器学习和深度学习模型的超参数优化调整方法的实现（简单明了）

hyperparameter-optimization:机器学习算法的贝叶斯超参数优化的实现

Masterarbeit-Hyperparameter-tuning-

XGboost-hyperparameter-tuning

最新资源