![](https://csdnimg.cn/release/download_crawler_static/85032805/bg1.jpg)
Cyclical Learning Rates for Training Neural Networks
Leslie N. Smith
U.S. Naval Research Laboratory, Code 5514
4555 Overlook Ave., SW., Washington, D.C. 20375
leslie.smith@nrl.navy.mil
Abstract
It is known that the learning rate is the most important
hyper-parameter to tune for training deep neural networks.
This paper describes a new method for setting the learning
rate, named cyclical learning rates, which practically elim-
inates the need to experimentally find the best values and
schedule for the global learning rates. Instead of mono-
tonically decreasing the learning rate, this method lets the
learning rate cyclically vary between reasonable bound-
ary values. Training with cyclical learning rates instead
of fixed values achieves improved classification accuracy
without a need to tune and often in fewer iterations. This
paper also describes a simple way to estimate “reasonable
bounds” – linearly increasing the learning rate of the net-
work for a few epochs. In addition, cyclical learning rates
are demonstrated on the CIFAR-10 and CIFAR-100 datasets
with ResNets, Stochastic Depth networks, and DenseNets,
and the ImageNet dataset with the AlexNet and GoogLeNet
architectures. These are practical tools for everyone who
trains neural networks.
1. Introduction
Deep neural networks are the basis of state-of-the-art re-
sults for image recognition [17, 23, 25], object detection
[7], face recognition [26], speech recognition [8], machine
translation [24], image caption generation [28], and driver-
less car technology [14]. However, training a deep neural
network is a difficult global optimization problem.
A deep neural network is typically updated by stochastic
gradient descent and the parameters θ (weights) are updated
by θ
t
= θ
t−1
−
t
∂L
∂θ
, where L is a loss function and
t
is
the learning rate. It is well known that too small a learning
rate will make a training algorithm converge slowly while
too large a learning rate will make the training algorithm
diverge [2]. Hence, one must experiment with a variety of
learning rates and schedules.
Conventional wisdom dictates that the learning rate
should be a single value that monotonically decreases dur-
Figure 1. Classification accuracy while training CIFAR-10. The
red curve shows the result of training with one of the new learning
rate policies.
ing training. This paper demonstrates the surprising phe-
nomenon that a varying learning rate during training is ben-
eficial overall and thus proposes to let the global learning
rate vary cyclically within a band of values instead of set-
ting it to a fixed value. In addition, this cyclical learning rate
(CLR) method practically eliminates the need to tune the
learning rate yet achieve near optimal classification accu-
racy. Furthermore, unlike adaptive learning rates, the CLR
methods require essentially no additional computation.
The potential benefits of CLR can be seen in Figure
1, which shows the test data classification accuracy of the
CIFAR-10 dataset during training
1
. The baseline (blue
curve) reaches a final accuracy of 81.4% after 70, 000 it-
erations. In contrast, it is possible to fully train the network
using the CLR method instead of tuning (red curve) within
25,000 iterations and attain the same accuracy.
The contributions of this paper are:
1. A methodology for setting the global learning rates for
training neural networks that eliminates the need to
perform numerous experiments to find the best values
and schedule with essentially no additional computa-
tion.
2. A surprising phenomenon is demonstrated - allowing
1
Hyper-parameters and architecture were obtained in April 2015 from
caffe.berkeleyvision.org/gathered/examples/cifar10.html
arXiv:1506.01186v6 [cs.CV] 4 Apr 2017