![](https://csdnimg.cn/release/download_crawler_static/11994475/bg3.jpg)
Feature Extraction Loss Computation
C(0)
…
…
…
C(1) C(2)
C(D-1)
C
r
(0)
C
r
(1) C
r
(2) C
r
(D-1)
…
Recurrent Regularization
GT Depth Map
One-hot
Loss
…
Softmax
Conv + BN + ReLU, stride = 1
Conv, stride = 1
Conv + BN + ReLU, stride = 2
GRU unit
Cost Maps
Regularized Cost Maps
M
Differentiable Homography Warping
&Variance Cost Metric
M
M M
M
P
Q
Figure 2: The R-MVSNet architecture. Deep image features are extracted from input images and then warped to the fronto-
parallel planes of the reference camera frustum. The cost maps are computed at different depths and are sequentially regular-
ized by the convolutional GRU. The network is trained as a classification problem with the cross-entropy loss
the end of MVSNet to further enhance the depth map qual-
ity. As deep image features {F
i
}
N
i=1
are downsized during
the feature extraction, the output depth map size is 1/4 to
the original image size in each dimension.
MVSNet has shown state-of-the-art performance on
DTU dataset [1] and the intermediate set of Tanks and
Temples dataset [17], which contain scenes with outside-
looking-in camera trajectories and small depth ranges.
However, MVSNet can only handle a maximum reconstruc-
tion scale at H ×W ×D = 1600 ×1184 ×256 with the 16
GB large memory Tesla P100 GPU, and will fail at larger
scenes e.g., the advanced set of Tanks and Temples. To
resolve the scalability issue especially for the wide depth
range reconstructions, we will introduce the novel recurrent
cost volume regularization in the next section.
3.2. Recurrent Regularization
Sequential Processing An alternative to globally regu-
larize the cost volume C in one go is to sequentially pro-
cess the volume through the depth direction. The simplest
sequential approach is the winner-take-all plane sweeping
stereo [7], which crudely replaces the pixel-wise depth
value with the better one and thus suffers from noise (Fig. 1
(a)). To improve, cost aggregation methods [29, 31] filter
the matching cost C(d) at different depths (Fig. 1 (b)) so as
to gather spatial context information for each cost estima-
tion. In this work, we follow the idea of sequential process-
ing, and propose a more powerful recurrent regularization
scheme based on convolutional GRU. The proposed method
is able to gather spatial as well as the uni-directional con-
text information in the depth direction (Fig. 1 (c)), which
achieves regularization results comparable to the full-space
3D CNNs but is much more efficient in runtime memory.
Convolutional GRU Cost volume C could be viewed as
D cost maps {C(i)}
D
i=1
concatenated in the depth direc-
tion. If we denote the output of regularized cost maps
as {C
r
(i)}
D
i=1
, for the ideal sequential processing at the
t
th
step, C
r
(t) should be dependent on cost maps of the
current step C(t) as well as all previous steps {C(i)}
t−1
i=1
.
Specifically, in our network we apply a convolutional vari-
ant of GRU to aggregate such temporal context information
in depth direction, which corresponds to the time direction
in language processing. In the following, we denote ‘’ as
the element-wise multiplication, ‘[]’ the concatenation and
‘∗’ the convolution operation. Cost dependencies are for-
mulated as:
C
r
(t) = (1 − U(t)) C
r
(t − 1) + U(t) C
u
(t) (1)
where U(t) is the update gate map to decide whether to up-
date the output for current step, C
r
(t −1) is the regularized
cost map of late step, and C
u
(t) could be viewed as the
updated cost map in current step, which is defined as:
C
u
(t) = σ
c
(W
c
∗ [C(t), R(t) C
r
(t − 1)] + b
c
) (2)
R(t) here is the reset gate map to decide how much the pre-
vious C
r
(t − 1) should affect the current update. σ
c
(·) is
3