3.1. Photo-geometric autoencoding
An image
I
is a function
Ω → R
3
defined on a grid
Ω = {0, . . . , W − 1} × {0, . . . , H − 1}
, or, equivalently, a
tensor in
R
3×W ×H
. We assume that the image is roughly
centered on an instance of the object of interest. The goal
is to learn a function
Φ
, implemented as a neural network,
that maps the image
I
to four factors
(d, a, w, l)
comprising
a depth map
d : Ω → R
+
, an albedo image
a : Ω → R
3
,
a global light direction
l ∈ S
2
, and a viewpoint
w ∈ R
6
so
that the image can be reconstructed from them.
The image
I
is reconstructed from the four factors in two
steps, lighting Λ and reprojection Π, as follows:
ˆ
I = Π (Λ(a, d, l), d, w) . (1)
The lighting function
Λ
generates a version of the object
based on the depth map
d
, the light direction
l
and the albedo
a
as seen from a canonical viewpoint
w = 0
. The viewpoint
w
represents the transformation between the canonical view
and the viewpoint of the actual input image
I
. Then, the
reprojection function
Π
simulates the effect of a viewpoint
change and generates the image
ˆ
I
given the canonical depth
d
and the shaded canonical image
Λ(a, d, l)
. Learning uses
a reconstruction loss which encourages I ≈
ˆ
I (Section 3.2).
Discussion.
The effect of lighting could be incorporated
in the albedo
a
by interpreting the latter as a texture rather
than as the object’s albedo. However, there are two good
reasons to avoid this. First, the albedo
a
is often symmetric
even if the illumination causes the corresponding appearance
to look asymmetric. Separating them allows us to more
effectively incorporate the symmetry constraint described
below. Second, shading provides an additional cue on the
underlying 3D shape [
24
,
3
]. In particular, unlike the recent
work of [
52
] where a shading map is predicted independently
from shape, our model computes the shading based on the
predicted depth, mutually constraining each other.
3.2. Probably symmetric objects
Leveraging symmetry for 3D reconstruction requires iden-
tifying symmetric object points in an image. Here we do so
implicitly, assuming that depth and albedo, which are recon-
structed in a canonical frame, are symmetric about a fixed
vertical plane. An important beneficial side effect of this
choice is that it helps the model discover a ‘canonical view’
for the object, which is important for reconstruction [44].
To do this, we consider the operator that flips a map
a ∈ R
C×W ×H
along the horizontal axis
2
:
[flip a]
c,u,v
=
a
c,W −1−u,v
.
We then require
d ≈ flip d
0
and
a ≈ flip a
0
.
While these constraints could be enforced by adding corre-
sponding loss terms to the learning objective, they would
be difficult to balance. Instead, we achieve the same effect
2
The choice of axis is arbitrary as long as it is fixed.
indirectly, by obtaining a second reconstruction
ˆ
I
0
from the
flipped depth and albedo:
ˆ
I
0
= Π (Λ(a
0
, d
0
, l), d
0
, w) , a
0
= flip a, d
0
= flip d. (2)
Then, we consider two reconstruction losses encouraging
I ≈
ˆ
I
and
I ≈
ˆ
I
0
. Since the two losses are commensurate,
they are easy to balance and train jointly. Most importantly,
this approach allows us to easily reason about symmetry
probabilistically, as explained next.
The source image
I
and the reconstruction
ˆ
I
are compared
via the loss:
L(
ˆ
I, I, σ) = −
1
|Ω|
X
uv ∈Ω
ln
1
√
2σ
uv
exp −
√
2`
1,uv
σ
uv
, (3)
where
`
1,uv
= |
ˆ
I
uv
− I
uv
|
is the
L
1
distance between the
intensity of pixels at location
uv
, and
σ ∈ R
W ×H
+
is a confi-
dence map, also estimated by the network
Φ
from the image
I
, which expresses the aleatoric uncertainty of the model.
The loss can be interpreted as the negative log-likelihood
of a factorized Laplacian distribution on the reconstruction
residuals. Optimizing likelihood causes the model to self-
calibrate, learning a meaningful confidence map [32].
Modelling uncertainty is generally useful, but in our case
is particularly important when we consider the “symmetric”
reconstruction
ˆ
I
0
, for which we use the same loss
L(
ˆ
I
0
, I, σ
0
)
.
Crucially, we use the network to estimate, also from the same
input image
I
, a second confidence map
σ
0
. This confidence
map allows the model to learn which portions of the input
image might not be symmetric. For instance, in some cases
hair on a human face is not symmetric as shown in Fig. 2, and
σ
0
can assign a higher reconstruction uncertainty to the hair
region where the symmetry assumption is not satisfied. Note
that this depends on the specific instance under consideration,
and is learned by the model itself.
Overall, the learning objective is given by the combina-
tion of the two reconstruction errors:
E(Φ; I) = L(
ˆ
I, I, σ) + λ
f
L(
ˆ
I
0
, I, σ
0
), (4)
where
λ
f
= 0.5
is a weighing factor,
(d, a, w, l, σ, σ
0
) =
Φ(I)
is the output of the neural network, and
ˆ
I
and
ˆ
I
0
are
obtained according to Eqs. (1) and (2).
3.3. Image formation model
We now describe the functions
Π
and
Λ
in Eq. (1) in more
detail. The image is formed by a camera looking at a 3D
object. If we denote with
P = (P
x
, P
y
, P
z
) ∈ R
3
a 3D
point expressed in the reference frame of the camera, this is
mapped to pixel p = (u, v, 1) by the following projection:
p ∝ KP, K =
f 0 c
u
0 f c
v
0 0 1
,
c
u
=
W −1
2
,
c
v
=
H−1
2
,
f =
W −1
2 tan
θ
FOV
2
.
(5)
4
剩余17页未读，继续阅读
TracelessLe
- 粉丝: 5w+
- 资源: 466
我的内容管理
展开
最新资源
- Vue实现iOS原生Picker组件：详细解析与实现思路
- Arduino蓝牙小车：参数调试与功能控制
- 百度Java面试精华：200页精选资源涵盖核心知识点
- Swift使用CoreData填坑指南：CoreData在Swift 3.0的变化
- 微距离无线充电器创新设计及其实验探索
- MTK Android平台开发全攻略：44步详解流程
- RecyclerView全面解析：替代ListView的新选择
- Android开发：自动适配中英文键盘解决方案
- Android调用WebService接口教程
- Android开发：BitmapUtil图片处理全解析与实例
- Android多线程断点续传实现详解
- PCA算法在人脸识别会议签到系统中的应用
- EventBus 3.0：Android事件总线详解与实战应用
- Android FileUtil：全面解析文件操作实用技巧与实例
- RecyclerView添加头部和尾部实战教程
- Android实现微博滑动固定顶部栏实战与优化
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
