Published as a conference paper at ICLR 2019
3.3 ANALYSIS ON NON-CONVERGENCE OF ADAM
As we have observed in the previous section, a common characteristic of these counterexamples
is that the net update factor for the gradient with large magnitude is smaller than these with small
magnitude. The above observation can also be interpreted as a direct consequence of inappropriate
correlation between v
t
and g
t
. Recall that v
t
= β
2
v
t−1
+(1−β
2
)g
2
t
. Assuming v
t−1
is independent
of g
t
, then: when a new gradient g
t
arrives, if g
t
is large, v
t
is likely to be larger; and if g
t
is small,
v
t
is also likely to be smaller. If β
1
= 0, then k(g
t
) = α
t
/
√
v
t
. As a result, a large gradient is likely
to have a small net update factor, while a small gradient is likely to have a large net update factor in
Adam.
When it comes to the scenario where β
1
> 0, the arguments are actually quite similar. Given v
t
=
β
2
v
t−1
+(1−β
2
)g
2
t
. Assuming v
t−1
and {g
t+i
}
∞
i=1
are independent from g
t
, then: not only does v
t
positively correlate with the magnitude of g
t
, but also the entire infinite sequence {v
i
}
∞
i=t
positively
correlates with the magnitude of g
t
. Since the net update factor k(g
t
) =
P
∞
i=t
α
i
/
√
v
i
(1−β
1
)β
i−t
1
negatively correlates with each v
i
in {v
i
}
∞
i=t
, it is thus negatively correlated with the magnitude of
g
t
. That is, k(g
t
) for a large gradient is likely to be smaller, while k(g
t
) for a small gradient is likely
to be larger.
The unbalanced net update factors cause the non-convergence problem of Adam as well as all other
adaptive learning rate methods where v
t
correlates with g
t
. To construct a counterexample, the same
pattern is that: the large gradient is along the “correct” direction, while the small gradient is along
the opposite direction. Due to the fact that the accumulated influence of a large gradient is small
while the accumulated influence of a small gradient is large, Adam may update parameters along
the wrong direction.
Finally, we would like to emphasize that even if Adam updates parameters along the right direc-
tion in general, the unbalanced net update factors are still unfavorable since they slow down the
convergence.
4 THE PROPOSED METHOD: DECORRELATION VIA TEMPORAL SHIFTING
According to the previous discussion, we conclude that the main cause of the non-convergence of
Adam is the inappropriate correlation between v
t
and g
t
. Currently we have two possible solutions:
(1) making v
t
act like a constant, which declines the correlation, e.g., using a large β
2
or keep v
t
non-
decreasing (Reddi et al., 2018); (2) using a large β
1
(Theorem 1), where the aggressive momentum
term helps to mitigate the impact of unbalanced net update factors. However, neither of them solves
the problem fundamentally.
The dilemma caused by v
t
enforces us to rethink its role. In adaptive learning rate methods, v
t
plays the role of estimating the second moments of gradients, which reflects the scale of gradient on
average. With the adaptive learning rate α
t
/
√
v
t
, the update step of g
t
is scaled down by
√
v
t
and
achieves rescaling invariance with respect to the scale of g
t
, which is practically useful to make the
training process easy to control and the training system robust. However, the current scheme of v
t
,
i.e., v
t
= β
2
v
t−1
+ (1 − β
2
)g
2
t
, brings a positive correlation between v
t
and g
t
, which results in
reducing the effect of large gradients and increasing the effect of small gradients, and finally causes
the non-convergence problem. Therefore, the key is to let v
t
be a quantity that reflects the scale of
the gradients, while at the same time, be decorrelated with current gradient g
t
. Formally, we have
the following theorem:
Theorem 5 (Decorrelation leads to convergence). For any fixed online optimization problem with
infinitely repeating of a finite set of cost functions {f
1
(θ), . . . , f
t
(θ), . . . f
n
(θ)}, assuming β
1
= 0
and α
t
is fixed, we have, if v
t
follows a fixed distribution and is independent of the current gradient
g
t
, then the expected net update factor for each gradient is identical.
Let P
v
denote the distribution of v
t
. In the infinitely repeating online optimization scheme, the
expectation of net update factor for each gradient g
t
is
E[k(g
t
)] =
∞
X
i=t
E
v
i
∼P
v
[
α
i
√
v
i
(1 − β
1
)β
i−t
1
]. (13)
5
剩余25页未读，继续阅读
小乐&小蓝的house
- 粉丝: 0
- 资源: 3
我的内容管理
展开
最新资源
- 中国微型数字传声器：技术革新与市场前景
- 智能安防：基于Hi3515的嵌入式云台控制系统设计
- 手机电量低时辐射真增千倍？解析手机使用谣言
- 56F803型DSP驱动的高精度大功率超声波电源控制策略研究
- ARM与GPRS结合的远程监测系统设计
- GPS与RFID技术结合的智能巡检系统设计
- CPLD驱动的低功耗爆炸场温度测试系统设计
- 基于FPGA的智能驱动控制系统：可扩展设计与工业网络协议
- 基于ATmega128和CH374的嵌入式USB接口设计
- 基于AT89C52的温度补偿超声波测距仪：高精度设计与应用
- MSP430F448单片机在交流数字电压表中的应用
- 提升变频器应用效率的12项实用技巧
- STM32F103在数字电镀电源并联均流系统中的应用
- PSpice仿真下的升压开关电源设计：拓扑分析与CCM稳定性提升
- 轻巧高效：MSP430主导的低成本无线传感器网络节点设计
- FPGA在EDA/PLD中实现LVDS接口的应用解析
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
