单幅图像深度3D人脸重建学习方法
需积分: 10 6 浏览量
更新于2024-07-06
收藏 9.6MB PDF 举报
本文主要探讨了"Deep 3D Portrait from a Single Image"这一主题，它是一种基于深度学习的方法，旨在从单张肖像照片中恢复人类头部的三维几何结构。作者Sicheng Xu、Jiaolong Yang、Dong Chen、Fang Wen、YuDeng和Yunde Jia，以及来自北京理工大学、微软亚洲研究院和清华大学的研究者们，共同开发了这个无监督学习系统，无需依赖任何真实的三维数据作为训练样本。
该方法的核心是使用参数化的三维人脸模型（如3DMM）结合一个深度图来表示头部，包括头发和耳朵等区域。这种方法采用两步学习策略：首先，通过自我重建技术在单张图像上学习面部形状；然后，利用立体匹配的方式处理头发和耳朵部分，进一步提高几何精度并确保整体头部结构的一致性。这种分步骤学习不仅提高了重建的准确性，还确保了头像在不同姿势下的三维一致性。
为了评估方法的性能，研究人员不仅考察了三维几何结构的精确度，还通过在二维图像上的姿态变换任务来检验其实际应用效果。他们设计了一个后处理网络，通过对抗学习进行优化，以便在恢复的几何基础上进行精细调整，以生成更逼真的三维复原结果。
这项研究展示了深度学习在单张肖像照片中捕捉和重构复杂头部细节的能力，为人脸相关的3D建模、动画制作、虚拟现实等领域提供了新的可能性。同时，它的无监督学习框架也为未来的3D重建工作提供了有价值的技术路线。
constraint by minimizing the brightness error
l
color
=
Z
H
0
2
kI
0
2
(d
1
) − I
2
k
1
+
Z
H
0
1
kI
0
1
(d
2
) − I
1
k
1
(4)
where H
0
2
= H
0
2
(H
1
, d
1
) is the warped region from H
1
computed by head poses and d
1
in the transformation pro-
cess described above; similarly for H
0
1
= H
0
1
(H
2
, d
2
). We
also apply a gradient discrepancy loss which is robust to il-
lumination change thus widely adopted in stereo and optical
flow estimation [6, 5, 56]:
l
g rad
=
Z
H
0
2
k∇I
0
2
(d
1
) −∇I
2
k
1
+
Z
H
0
1
k∇I
0
1
(d
2
) −∇I
1
k
1
(5)
where ∇ denotes the gradient operator. To impose a spatial
smoothness prior, we add a second-order smoothness loss
l
smooth
=
Z
H
1
|∆d
1
| +
Z
H
2
|∆d
2
| (6)
where ∆ denotes the Laplace operator.
Face depth as condition and output. Instead of directly
estimating hair and ear depth from the input image I, we
project the reconstructed face shape F onto image plane to
get a face depth map d
f
. We make d
f
an extra conditional
input concatenated with I. Note d
f
provides beneficial in-
formation (e.g., head pose, camera distance) for hair and
ear depth estimation. In addition, it allows the known face
depth around the contour to be easily propagated to the ad-
jacent regions with unknown depth.
More importantly, we train the network to also predict
the depth of the facial region using d
f
as target:
l
face
==
Z
F
1
−S
h
1
T
F
1
|d
1
− d
f
1
|+
Z
F
2
−S
h
2
T
F
2
|d
2
− d
f
2
| (7)
where S
h
denotes the hair region defined by segmentation.
Note learning face depth via l
face
should not introduce
much extra burden for the network since d
f
is provided as
input. But crucially, we can now easily enforce the consis-
tency between the reconstructed 3D face and the estimated
3D geometry in other regions, as in this case we calculate
the smoothness loss across whole head regions S
1
, S
2
:
l
smooth
=
Z
S
1
|∆d
1
| +
Z
S
2
|∆d
2
| (8)
Figure 2 (2nd and 3rd columns) compares the results
with and without face depth. We also show quantitative
comparisons in Table 1 (2nd and 3rd columns). As can be
observed, using face depth significantly improves head ge-
ometry consistency and reconstruction accuracy.
Layer-order loss. Hair can often occlude a part of facial
region, leading to two depth layers. To ensure correct rela-
tive position between the hair and occluded face region (i.e.,
Input w/o face depth with face depth + l
layer
Figure 2: 3D head reconstruction result of our method with
different settings.
the former should be in front of the latter), we introduced a
layer-order loss defined as:
l
layer
=
Z
S
h
1
T
F
1
max(0, d
1
− d
f
1
)+
Z
S
h
2
T
F
2
max(0, d
2
− d
f
2
)
(9)
which penalizes incorrect layer order. As shown in Fig. 2,
the reconstructed shapes are more accurate with l
layer
.
Network structure. We apply a simple encoder-decoder
structure using a ResNet-18 [25] as backbone. We discard
its global average pooling and the last fc layers, and append
several transposed convolutional layers to upsample the fea-
ture maps to the full resolution. Skip connections are added
at 64×64, 32×32 and 16×16 resolutions. The input image
size is 256 × 256. More details of the network structure can
be found in the suppl. material.
5. Single-Image Head Pose Manipulation
Given the 3D head model reconstructed from the input
portrait image, we modify its pose and synthesize new por-
trait images, described as follows.
5.1. 3D Pose Manipulation and Projection
To change the head pose, one simply needs to apply a
rigid transformation in 3D for the 3DMM face F and hair-
ear mesh H given the target pose
¯
p or displacement δp. Af-
ter the pose is changed, we reproject the 3D model onto 2D
image plane to get coarse synthesis results. Two examples
are shown in Fig. 3.
5.2. Image Refinement with Adversarial Learning
The reprojected images suffer from several issues. No-
tably, due to pose and expression change, some holes may
appear, where the missing background and/or head region
should be hallucinated akin to an image inpainting process.
Besides, the reprojection procedure may also induce certain
artifacts due to imperfect rendering.
剩余15页未读，继续阅读
2021-02-08 上传
2018-08-06 上传
2021-03-16 上传
2021-02-21 上传
2015-01-28 上传
2021-05-10 上传
2021-06-26 上传
2021-03-25 上传
TracelessLe
- 粉丝: 6w+
- 资源: 468
我的内容管理
展开
最新资源
- Python库 | mtgpu-0.2.5-py3-none-any.whl
- endpoint-testing-afternoon:一个下午的项目，以帮助使用Postman巩固测试端点
- 经济中心
- z7-mybatis:针对mybatis框架的练习，目前主要技术栈包含springboot，mybatis，grpc，swgger2，redis，restful风格接口
- Cloudslides-Android:云幻灯同步演示应用-Android Client
- testingmk:做尼采河
- ecom-doc-static
- kindle-clippings-to-markdown:将Kindle的“剪贴”文件转换为Markdown文件，每本书一个
- 减去图像均值matlab代码-TVspecNET:深度学习的光谱总变异分解
- 自动绿色
- Alexa-Skills-DriveTime:该存储库旨在演示如何建立ALEXA技能，以帮助所有人了解当前流量中从源头到达目的地所花费的时间
- 灰色按钮克星易语言版.zip易语言项目例子源码下载
- HTML5:基本HTML5
- dubbadhar-light
- 使用Xamarin Forms创建离线移动密码管理器
- matlab对直接序列扩频和直接序列码分多址进行仿真实验源代码
