US Naval Research Laboratory Technical Report 5510-026
The second example is on the Imagenet dataset with two architectures: resnet-50, and inception-
resnet-v2. Here the cause of underfitting is due to the underlying complexities of the data distribu-
tions. In Figure 3b the test loss continues to decrease over the 100,000 iterations (about 3 epochs) but
the inception-resnet-v2 decreases more and becomes more horizontal, indicating that the inception-
resnet-v2 has less underfitting.
Increasing the learning rate helps reduce underfitting. An easy way to find a good learning rate is
the LR range test (Smith, 2017) (additional description of the LR range test is in Section 4.1).
(a) Cifar-10 dataset with a shallow 3 layer network. (b) Imagenet dataset with resnet-50 architecture.
Figure 4: Increasing validation/test loss indicates overfitting. Examples of overfitting are shown for
Cifar-10 and Imagenet. WD = weight decay, LR = learning rate, CLR = cyclical learning rate, CM
= cyclical momentum.
3.3 OVERFITTING
Overfitting tends to be more complicated than underfitting but clues are visible in the test loss. In
Figure 2 the test loss goes from underfitting (decreasing) to overfitting (increasing) but overfitting
in neural networks is often not so simple. In Figure 1 there are signs of overfitting in the test loss at
small learning rates (0.01 - 0.04) but then the test loss continues to decrease at higher learning rates
as though it is underfitting. This indicates that learning rates that are too small can exhibit some
overfitting behavior.
Figure 4a displays test loss for training a shallow, 3-layer network trained on Cifar-10. The figure
shows the results of a learning rate range test at three values for weight decay (WD). At WD = 10
−4
the loss reaches a minimum loss near a learning rate of 0.002, then begins to increase, displaying
overfitting. If WD = 4 × 10
−3
, the loss is stable over a larger range of learning rate values and the
loss attains a lower loss value, indicating that the latter WD value is superior to the former. Also
note that the sharp increase in loss for the yellow curve near the learning rate of 0.005 is not a sign
of overfitting but is caused by instabilities in the training due to the large learning rate. Similarly,
the red curve diverges above a learning rate of 0.008.
The next example is training the Imagenet dataset with a resnet-50 architecture. While the blue
curve in Figure 4b shows underfitting with weight decay of 10
−4
and a constant learning rate of
0.1, the red curve shows overfitting with too small a weight decay of 10
−7
. Both curves indicate
non-optimal settings of the hyper-parameters.
An additional example of overfitting is visible with the yellow curve in Figure 7a as it slowly rises
above LR=0.006 and until divergence near LR=0.01. The blue curves in Figure 9a and in Figure
11a show two additional forms of overfitting. The former illustrates a more gradual increase in the
error while the later shows a temporary initial increase (additional analysis of this Figure is given in
Section 5.5). All situations are examples of incorrectly set hyper-parameters.
The art of setting the network’s hyper-parameters amounts to ending up at the balance point between
underfitting and overfitting.
5
剩余20页未读，继续阅读
大饼博士X
- 粉丝: 4072
- 资源: 9
我的内容管理
展开
最新资源
- 多模态联合稀疏表示在视频目标跟踪中的应用
- Kubernetes资源管控与Gardener开源软件实践解析
- MPI集群监控与负载平衡策略
- 自动化PHP安全漏洞检测：静态代码分析与数据流方法
- 青苔数据CEO程永：技术生态与阿里云开放创新
- 制造业转型： HyperX引领企业上云策略
- 赵维五分享：航空工业电子采购上云实战与运维策略
- 单片机控制的LED点阵显示屏设计及其实现
- 驻云科技李俊涛：AI驱动的云上服务新趋势与挑战
- 6LoWPAN物联网边界路由器：设计与实现
- 猩便利工程师仲小玉：Terraform云资源管理最佳实践与团队协作
- 类差分度改进的互信息特征选择提升文本分类性能
- VERITAS与阿里云合作的混合云转型与数据保护方案
- 云制造中的生产线仿真模型设计与虚拟化研究
- 汪洋在PostgresChina2018分享：高可用 PostgreSQL 工具与架构设计
- 2018 PostgresChina大会：阿里云时空引擎Ganos在PostgreSQL中的创新应用与多模型存储
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
