Architecture.
Following the STM practice [
18
], we take
res4
features with stride 16 from the base
ResNets as our backbone features and discard
res5
. A
3×3
convolutional layer without non-linearity
is used as a projection head from the backbone feature to either the
key
space (
C
k
dimensional) or
the
value
space (
C
v
dimensional). We set
C
v
to be 512 following STM and discuss the choice of
C
k
in Section 4.1.
Feature reuse.
As seen from Figure 1, both the key encoder and the value encoder are processing
the same frame, albeit with different inputs. It is natural to reuse features from the key encoder (with
fewer inputs and a deeper network) at the value encoder. To avoid bloating the feature dimensions
and for simplicity, we concatenate the last layer features from both encoders (before the projection
head) and process them with two ResBlocks [
62
] and a CBAM block
3
[
64
] as the final value output.
3.2 Memory Reading and Decoding
Given
T
memory frames and a query frame, the feature extraction step would generate the followings:
memory key
k
M
∈ R
C
k
×T HW
, memory value
v
M
∈ R
C
v
×T HW
, and query key
k
Q
∈ R
C
k
×HW
,
where
H
and
W
are (stride 16) spatial dimensions. Then, for any similarity measure
c : R
C
k
×R
C
k
→
R
, we can compute the pairwise affinity matrix
S
and the softmax-normalized affinity matrix
W
,
where S, W ∈ R
T HW ×HW
with:
S
ij
= c(k
M
i
, k
Q
j
) W
ij
=
exp (S
ij
)
P
n
(exp (S
nj
))
, (1)
where
k
i
denotes the feature vector at the
i
-th position. The similarities are normalized by
√
C
k
as in
standard practice [
18
,
33
] and is not shown for brevity. In STM [
18
], the dot product is used as
c
.
Memory reading regularization like KMN [22] or top-k filtering [21] can be applied at this step.
With the normalized affinity matrix
W
, the aggregated readout feature
v
Q
∈ R
C
v
×HW
for the
query frame can be computed as a weighted sum of the memory features with an efficient matrix
multiplication:
v
Q
= v
M
W, (2)
which is then passed to the decoder for mask generation.
In the case of multi-object segmentation, only Equation 2 has to be repeated as
W
is defined between
image features only, and thus is the same for different objects. In the case of STM [
18
],
W
must be
recomputed instead. Detailed running time analysis can be found in Section 6.2.
Decoder.
Our decoder structure stays close to that of the STM [
18
] as it is not the focus of this paper.
Features are processed and upsampled at a scale of two gradually with higher-resolution features
from the key encoder incorporated using skip-connections. The final layer of the decoder produces a
stride 4 mask which is bilinearly upsampled to the original resolution. In the case of multiple objects,
soft aggregation [18] of the output masks is used.
3.3 Memory Management
So far we have assumed the existence of a memory bank of size
T
. Here, we will describe the
construction of the memory bank. For each memory frame, we store two items: memory key and
memory value. Note that all memory frames (except the first one) are once query frames. The memory
key is simply reused from the query key, as described in Section 3.1 without extra computation. The
memory value is computed after mask generation of that frame, independently for each object as the
value encoder takes both the image and the object mask as inputs.
STM [
18
] consider every fifth query frame as a memory frame, and the immediately previous frame
as a temporary memory frame to ensure accurate matching. In the case of STCN, we find that it is
unnecessary, and in fact harmful, to include the last frame as temporary memory. This is a direct
consequence of using shared key encoders – 1) key features are sufficiently robust to match well
without the need for close-range (temporal) propagation, and 2) the temporary memory key would
otherwise be too similar to that of the query, as the image context usually changes smoothly and we do
not have the encoding noises resultant from distinct encoders, leading to drifting.
4
This modification
also reduces the number of calls to the value encoder, contributing a significant speedup.
3
We find this block to be non-essential in a later experiment but it is kept for consistency.
4
This effect is amplified by the use of L2 similarity. See the supplementary material for a full comparison.
4
剩余18页未读，继续阅读
易小侠
- 粉丝: 6550
- 资源: 9万+
我的内容管理
展开
最新资源
- 最优条件下三次B样条小波边缘检测算子研究
- 深入解析：wav文件格式结构
- JIRA系统配置指南：代理与SSL设置
- 入门必备：电阻电容识别全解析
- U盘制作启动盘：详细教程解决无光驱装系统难题
- Eclipse快捷键大全：提升开发效率的必备秘籍
- C++ Primer Plus中文版：深入学习C++编程必备
- Eclipse常用快捷键汇总与操作指南
- JavaScript作用域解析与面向对象基础
- 软通动力Java笔试题解析
- 自定义标签配置与使用指南
- Android Intent深度解析：组件通信与广播机制
- 增强MyEclipse代码提示功能设置教程
- x86下VMware环境中Openwrt编译与LuCI集成指南
- S3C2440A嵌入式终端电源管理系统设计探讨
- Intel DTCP-IP技术在数字家庭中的内容保护
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
