Figure 1: SGD fluctuation (Source: Wikipedia)
2.3 Mini-batch gradient descent
Mini-batch gradient descent finally takes the best of both worlds and performs an update for every
mini-batch of n training examples:
θ = θ − η · ∇
θ
J(θ; x
(i:i+n)
; y
(i:i+n)
) (3)
This way, it a) reduces the variance of the parameter updates, which can lead to more stable conver-
gence; and b) can make use of highly optimized matrix optimizations common to state-of-the-art
deep learning libraries that make computing the gradient w.r.t. a mini-batch very efficient. Common
mini-batch sizes range between
50
and
256
, but can vary for different applications. Mini-batch
gradient descent is typically the algorithm of choice when training a neural network and the term
SGD usually is employed also when mini-batches are used. Note: In modifications of SGD in the
rest of this post, we leave out the parameters x
(i:i+n)
; y
(i:i+n)
for simplicity.
In code, instead of iterating over examples, we now iterate over mini-batches of size 50:
for i in range ( n b _ epochs ):
np . random . shuffle ( data )
for batch in get_batches ( data , batch_size =50):
params_grad = evaluat e _ gradien t ( loss_function , batch , params )
params = params - l e a rning_rate * params_grad
3 Challenges
Vanilla mini-batch gradient descent, however, does not guarantee good convergence, but offers a few
challenges that need to be addressed:
•
Choosing a proper learning rate can be difficult. A learning rate that is too small leads to
painfully slow convergence, while a learning rate that is too large can hinder convergence
and cause the loss function to fluctuate around the minimum or even to diverge.
•
Learning rate schedules [
18
] try to adjust the learning rate during training by e.g. annealing,
i.e. reducing the learning rate according to a pre-defined schedule or when the change in
objective between epochs falls below a threshold. These schedules and thresholds, however,
have to be defined in advance and are thus unable to adapt to a dataset’s characteristics [4].
•
Additionally, the same learning rate applies to all parameter updates. If our data is sparse
and our features have very different frequencies, we might not want to update all of them to
the same extent, but perform a larger update for rarely occurring features.
•
Another key challenge of minimizing highly non-convex error functions common for neural
networks is avoiding getting trapped in their numerous suboptimal local minima. Dauphin
et al. [
5
] argue that the difficulty arises in fact not from local minima but from saddle points,
i.e. points where one dimension slopes up and another slopes down. These saddle points are
usually surrounded by a plateau of the same error, which makes it notoriously hard for SGD
to escape, as the gradient is close to zero in all dimensions.
3
剩余13页未读，继续阅读
NOWAY_EXPLORER
- 粉丝: 79
- 资源: 3
我的内容管理
展开
最新资源
- C++多态实现机制详解：虚函数与早期绑定
- Java多线程与异常处理详解
- 校园导游系统：无向图实现最短路径探索
- SQL2005彻底删除指南：避免重装失败
- GTD时间管理法：提升效率与组织生活的关键
- Python进制转换全攻略：从10进制到16进制
- 商丘物流业区位优势探究：发展战略与机遇
- C语言实训：简单计算器程序设计
- Oracle SQL命令大全：用户管理、权限操作与查询
- Struts2配置详解与示例
- C#编程规范与最佳实践
- C语言面试常见问题解析
- 超声波测距技术详解：电路与程序设计
- 反激开关电源设计：UC3844与TL431优化稳压
- Cisco路由器配置全攻略
- SQLServer 2005 CTE递归教程：创建员工层级结构
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
