Keyframe-based Texture Mapping for RGBD Human Reconstruction
Yishu Heng
1
, Chen Wang
*
1
, and Yue Qi
1,2
1
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University , Beijing 100191, China
2
Beihang University Qingdao Research Institute , Qingdao 266100, China
ABSTRACT
Realistic human model has a wide range of requirements in 3D
content creation. A model with high-quality texture map can display
human body surface details in low facets which could be toughly rep-
resented by geometric mesh. Image-based texture mapping suffers
from discontinuities due to geometry inaccuracy, camera pose drifts,
and illumination changes. In this paper, we propose a keyframe-
based texture map generation method to obtain more desired tex-
ture mapping results. Our method firstly acquire the keyframes
by performing a spatio-temporal sampling strategy, rather than just
sampling keyframes according to time interval. Then, we apply
an efficient patch-based optimization to the keyframes to make the
texture data in different views alinged with each other. Finally, we
generate a texture atlas from the aligned texture and the simplified
mesh. Experimental results demonstrate that our method can get re-
alistic human models with low facets and competitive details within
short minutes.
Index Terms:
Texture-Mapping; Patch-Match; Alternating-
Optimization; Keyframe-Selection;
1 INTRODUCTI ON
Realistic human model has a wide range of requirements in AR/VR,
games and movie industry, especially when the entertainment appli-
cations on mobile device developed impressively in present day. The
wide availability of consumer RGB-D sensors, such as Microsoft
Kinect and Asus Xtion, has boosted research in 3D geometry re-
construction in recent years. State-of-the-art methods in 3D model
reconstruction yield impressively accurate geometric reconstruc-
tion in real-time, e.g. [24] [6].Moreover, ordinary users are now
able to produce geometric models of objects using techniques like
KinectFusion [15].
However, reproducing the full appearance of real-world objects
also requires reconstructing high-quality texture maps. Fast and
robust estimation of the model texture has been given less attention.
Image-based texture mapping is a common approach to produce
a view-independent texture map from a set of images taken from
different viewpoints. Naively projecting and combining the input
images produces blurring and ghosting artifacts since the geometry
and camera poses are usually estimated from noisy data, and thus,
are inaccurate.
In this paper, we proposed a novel texture mapping method
for RGBD human reconstruction. First, we adopt a simplification
method on the reconstructed model. Then, we perform a spatial-
tempo keyframe sampling during the scanning. This sampling guar-
antee the selected keyframe quality and reduce the redundant data.
In the optimization step, we perform a patch-based optimization
similar to Bi S. [3] on the selected keyframes to synthesize aligned
*
e-mail:vr wangchen@buaa.edu.cn
Figure 1: The pipeline of our texture map generation method. Input
color images are mapped and blended on a simplified version of the
3D reconstructed mesh to produce a global texture map.
frames. Finally, we parameterize the model to 2D image and blend-
ing all color frame to get the final texture atlas. We validate the
effectiveness of the proposed method in different model and show
competitive results. In summary, our contributions are:
•
We proposed a keyframe-based method to generate single
consistent texture atlas for RGBD human reconstruction.
•
We perform a tempo-spatial keyframes sampling strategy, in-
stead of sampling keyframes based fixed time interval, to pre-
serve the texture quality and accelerate the optimization.
•
We directly minimize the error of all keyframes to synthesize
the aligned texture data, rather optimize the voxels/vertices
color or the camera poses.
2 RELATED WORKS
Image-based texture mapping suffers from discontinuities due to
geometry inaccuracy, camera pose drifts, and illumination changes.
To produce a single consistent texture map from a set of images
captured from different view points, which can then be rendered
with different lightings, the main challenge is the accuracies in the
capturing process. Several methods have been presented to register
the images to the geometry [2, 4, 9]. While these methods addressed
the camera calibration inaccuracies effectively, they are not able to
handle inaccurate geometry, and optical distortions in RGB images
which are common problems of consumer depth cameras.
In order to address the accuracy issues that can not be solved by
optimizing the camera pose, some researchers propose to optimize