Published as a conference paper at ICLR 2020
memory buffer (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018). Another approach combines
experience replay algorithms with lifelong learning (Rolnick et al., 2018; Riemer et al., 2018).
In the second category, one increases model capacity as new tasks are added. For example, progressive
neural networks (PNN) copy the entire network for the previous task and add new hidden units
when adopting to a new task (Rusu et al., 2016). This prevents forgetting on previous tasks by
construction (the network on previous tasks remains the same). However, it leads to significant
memory consumption when faced with a large number of lifelong learning tasks. Some following
methods expand the model in a more parameter efficient way at the cost of introducing an extra
learning task and not entirely preventing forgetting. Yoon et al. (2017) applies group sparsity
regularization to efficiently expand model capacity; Xu & Zhu (2018) learns to search for the best
architectural changes by carefully designed reinforcement learning strategies.
3 METHODS
As described above, ensembles suffer from expensive memory and computational costs. In this
section, we introduce BatchEnsemble, an efficient way to ensemble deep neural networks.
3.1 BATCHENSEMBLE
In this section, we introduce how to ensemble neural networks in an efficient way. Let
W
be
the weights in a neural network layer. Denote the input dimension as
m
and the output dimen-
sion as
n
, i.e.
W ∈ R
m×n
. For ensemble, assuming the ensemble size is
M
and each ensem-
ble member has weight matrix
W
i
. Each ensemble member owns a tuple of trainable vectors
r
i
and
s
i
which share the same dimension as input and output (
m
and
n
) respectively, where
i
ranges from
1
to
M
. Our algorithm generates a family of ensemble weights
W
i
by the following:
One shared
weight matrix
(slow weight)...
...multiplied by
independent rank
one fast weights...
...yields ensemble
weight matrices for
each member.
0 1-1
W
Figure 2: An illustration on how to generate the
ensemble weights for two ensemble members.
W
i
= W ◦ F
i
, where F
i
= r
i
s
>
i
, (1)
For each training example in the mini-batch, it
receives an ensemble weight
W
i
by element-
wise multiplying
W
, which we refer to as “slow
weights”, with a rank-one matrix
F
i
, which we
refer to as “fast weights.” The subscript
i
repre-
sents the selection of ensemble member. Since
W
is shared across ensemble members, we term
it as "shared weight" in the following paper. Fig-
ure 2 visualizes BatchEnsemble. Rather than
modulating the weight matrices, one can also
modulate the neural networks’ intermediate fea-
tures, which achieves promising performance in
visual reasoning tasks (Perez et al., 2017).
Vectorization: We show how to make the above ensemble weight generation mechanism paralleliz-
able within a device, i.e., where one computes a forward pass with respect to multiple ensemble
members in parallel. This is achieved by manipulating the matrix computations for a mini-batch (Wen
et al., 2018). Let
x
denote the activations of the incoming neurons in a neural network layer. The
next layer’s activations are given by:
y
n
= φ
W
>
i
x
n
(2)
= φ
W ◦ r
i
s
>
i
>
x
n
(3)
= φ
W
>
(x
n
◦ r
i
)
◦ s
i
, (4)
where
φ
denotes the activation function and the subscript
n
represents the index in the mini-batch.
The output represents next layer’s activations from the
i
th
ensemble member. To vectorize these
computations, we define matrices
R
and
S
whose rows consist of the vectors
r
i
and
s
i
for all examples
4
剩余19页未读，继续阅读
潜夙
- 粉丝: 0
- 资源: 40
我的内容管理
展开
最新资源
- Vue实现iOS原生Picker组件：详细解析与实现思路
- Arduino蓝牙小车：参数调试与功能控制
- 百度Java面试精华：200页精选资源涵盖核心知识点
- Swift使用CoreData填坑指南：CoreData在Swift 3.0的变化
- 微距离无线充电器创新设计及其实验探索
- MTK Android平台开发全攻略：44步详解流程
- RecyclerView全面解析：替代ListView的新选择
- Android开发：自动适配中英文键盘解决方案
- Android调用WebService接口教程
- Android开发：BitmapUtil图片处理全解析与实例
- Android多线程断点续传实现详解
- PCA算法在人脸识别会议签到系统中的应用
- EventBus 3.0：Android事件总线详解与实战应用
- Android FileUtil：全面解析文件操作实用技巧与实例
- RecyclerView添加头部和尾部实战教程
- Android实现微博滑动固定顶部栏实战与优化
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
