combined with the image at time t, producing an output and a new state. Since
the last output benefits from learning from the whole sequence, it is natural to
place the frame that we want to evaluate at the end.
The first temporal copy is special, since it contains regular feed-forward
connections. This allows us to produce activations in each layer such that all
connection types can be used in the transition from t to t + 1.
Network Depth. When processing input at time t, we allow L −1 time steps for
the information to reach the top level of the network and the same amount for
propagating back to the bottom layer, where the output corresponding to time
t is evaluated. Note that the last temporal steps do not need all the hidden
layers, since their activation would no longer propagate to the output.
Our RNN is trained with backpropagation through time (BPTT), and can
be interpreted as a very deep non-recurrent net after unfolding in time. In
this non-recurrent network, multiple paths lead to the output, with the shortest
path — from input t = T to the final output — having only length 2L − 1, and
the longest 2L + t, which amounts to a depth of 14 layers for our L = 3, T = 8
network.
Weight Initialization and Optimization. We initialize the weights and biases
from a Gaussian distribution. It is important to ensure that the activations
do not explode or vanish early during training. Ideally, activations in the first
forward pass should have similar magnitudes. This is difficult to control, how-
ever. Instead, we choose the standard deviation of the weights for each layer l
according to the scheme proposed by He et al. (2015):
σ =
s
2
k
l
2
· d
l−1
, (1)
which takes into account the filter size k
l
and the number of filters of the last
layer d
l−1
. We determine the mean of the bias such that the average of the
activations in every point of our network is positive and slightly decreasing over
time. Liang and Hu (2015) use local contrast normalization at all layers to the
same effect, which requires more GPU memory for the hidden layer activations.
Due to our larger inputs and outputs and the increased number of time steps,
current GPU memory restrictions prevent us from doing the same.
We learn the parameters of our network with backpropagation through
time (BPTT) using RMSProp, which is a variant of resilient backpropagation
(RPROP, Riedmiller and Braun 1993) suitable for mini-batch learning (Dauphin
et al., 2015). RPROP and RMSProp to a large degree consider only the sign of
the gradient, thus being robust against vanishing and exploding gradients, both
common phenomena in RNN training.
During learning, we apply dropout (Srivastava et al., 2014). Combining
dropout with RNNs is delicate, however. If it affects recurrent connections,
their ability to learn long-range dependencies suffers (Pham et al., 2014). Thus,
we apply dropout only to the final convolution with non-shared weights that
5
剩余20页未读，继续阅读
Fun_He
- 粉丝: 18
- 资源: 104
我的内容管理
展开
最新资源
- 多模态联合稀疏表示在视频目标跟踪中的应用
- Kubernetes资源管控与Gardener开源软件实践解析
- MPI集群监控与负载平衡策略
- 自动化PHP安全漏洞检测：静态代码分析与数据流方法
- 青苔数据CEO程永：技术生态与阿里云开放创新
- 制造业转型： HyperX引领企业上云策略
- 赵维五分享：航空工业电子采购上云实战与运维策略
- 单片机控制的LED点阵显示屏设计及其实现
- 驻云科技李俊涛：AI驱动的云上服务新趋势与挑战
- 6LoWPAN物联网边界路由器：设计与实现
- 猩便利工程师仲小玉：Terraform云资源管理最佳实践与团队协作
- 类差分度改进的互信息特征选择提升文本分类性能
- VERITAS与阿里云合作的混合云转型与数据保护方案
- 云制造中的生产线仿真模型设计与虚拟化研究
- 汪洋在PostgresChina2018分享：高可用 PostgreSQL 工具与架构设计
- 2018 PostgresChina大会：阿里云时空引擎Ganos在PostgreSQL中的创新应用与多模型存储
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
