实时面部动作捕捉系统与 Blendshape 技术

需积分: 50 74 浏览量更新于2024-07-14 收藏 10.8MB PDF 举报

"Real-Time Facial Motion Capture System" 是一篇由Steven Twist撰写的硕士论文，主要研究实时面部动作捕捉系统，特别关注了表情捕捉中的blendshape技术。这篇论文深入探讨了在计算机动画和视觉效果领域中如何实现高效、精确且成本合理的实时面部运动捕捉。在论文的"Previous Work"部分，Steven Twist概述了现有的面部重建技术和其他相关技术。他讨论了标记位置的计算方法，分析了这些标记位置的重要性，并提到了头盔式摄像头作为捕捉设备的可能性。这些技术是理解面部运动捕捉的基础。 "Design"章节中，作者详细介绍了系统的构建。硬件部分包括了对摄像头和头盔式摄像头的讨论，这是捕捉和传输图像流的关键。跟踪技术用于识别和追踪面部特征，而跟踪平滑算法则确保了捕捉数据的连续性和稳定性。Blendshapes是本文的核心，这是一种通过混合不同基础形状来创建复杂面部表情的技术。Steven Twist详细解释了blendshape理论，并阐述了如何映射运动捕捉数据到blendshape权重，这部分理论涉及向量空间的概念，并且在解决权重计算时给出了详尽的步骤。在"Implementation"章节，作者展示了具体实现这个系统的过程，包括头盔式摄像头和其他硬件的配置，以及如何处理摄像头图像流、追踪技术的实现、跟踪平滑的细节，以及如何将运动捕捉数据映射到blendshapes上。这一部分提供了实现系统所需的工程细节。 "Results and Analysis"章节中，Steven Twist评估了系统的性能，包括时间性能（是否能实现实时处理）、成本性能（系统的经济性）以及视觉性能（捕捉到的面部表情的逼真度）。这些分析对于理解系统实际应用的可行性和效率至关重要。最后，"Conclusion"部分总结了研究工作，提出了未来可能的研究方向，比如技术的改进和优化。同时，Steven Twist强调了这项工作对实时面部动作捕捉领域的贡献。这篇论文为实时面部动作捕捉系统提供了一个全面的视角，特别是在使用blendshape技术进行表情驱动方面，它不仅涵盖了理论框架，还包含了实际系统的实现和性能评估，对于理解和开发这类系统具有很高的参考价值。

Chapter 3

Design

Figure 3.1: Diagram illustrating the architecture of our facial motion capture system.

Figure 3.1 illustrates the overall architecture of our facial motion capture system. Facial

motion capture can be thought of as a pipeline, beginning with some form of input data from

a hardware device, and ending with the generation of animation data in some form.

The facial motion capture pipeline can be subdivided into smaller modules. Each mod-

ule encapsulates a speciﬁc task, receiving certain data as input, processing that data, and

outputting new data for the consumption of other modules.

The architectural and algorithmic design of each module is discussed in its own section,

below. Chapter 4 contains similar sections for each module, detailing the techniques used to

implement each of the modules in our facial motion capture system.

3.1 Hardware

The entry point to any motion capture system is some form of hardware device which captures

relevant data from the actor, and streams it to the motion capture software (Pighin 2006).

In the case of our facial motion capture system, our hardware device is a video camera aimed

at the actor’s face. This video camera must be capable of outputting a continual live video

stream to a desktop PC. For our current discussion of system design, the format of the video

stream is unimp ortant.

Section 2.2.3 introduced the idea of helmet mounted cameras, and described the beneﬁts

that they bring to a facial motion capture system. Our facial motion capture system will

utilise such a helmet mounted cam era setup, where the video camera ﬁlming the actor’s face

is rigidly attached to the actor’s head, via a helmet.

This gains us all the beneﬁts mentioned in Section 2.2.3, namely: the removal of marker

occlusions, due to the proximity of the camera to the actor’s head; the removal of any rigid

transformations of the actor’s head from the relative transformations of the facial markers in

the video feed; and the subsequent ease of marker labelling.

CHAPTER 3. DESIGN Steven Twist

The hardware setup used by our facial motion capture system has a signiﬁcant impact

on the rest of the pipeline. By using a single video camera, the software component of the

pipeline receives only 2D image data of the face. Thus all other operations performed on

that data will occur in two dimensions. The decision to utilise a single camera, as opposed

to a multi-camera setup, was made for three reasons:

• Most facial motion occurs in a 2D plane, with relatively little facial motion occurring in

a forwards-backwards direction, in terms of the orientation of the actor’s head (Ekman

et al. 2002).

• To reduce overall hardware complexity, and thus costs, ensuring that the hardware

could be built on a tight budget (which is one of the goals of this project (Chapter 1)).

• To allow for real-time facial motion capture solving by reducing the amount of input

data (Firestone 2008), a key goal of this project (Chapter 1).

3.2 Webcam Image Stream

The Webcam Image Stream module receives video data from the video camera hardware as

a series of frames. The format of this data will be dependant on the hardware used. This

module has a simple purpose: to take each of the input video frames, and convert them to

a RGB bitmap data format for subsequent processing by the Tracking module (Sec tion 3.3).

In addition, the Webcam Image Stream module draws the RGB frames to the screen, such

that the video feed of the actor’s face is displayed as part of the user interface.

3.3 Tracking

The Tracking module receives an RGB bitmap frame of video data from the Webcam Image

Stream module (Section 3.2), and is responsible for detecting and tracking a set of facial

markers in that frame, outputting the 2D location of each facial marker. In addition, the

Tracking module draws the marker locations over the video feed of the actor’s face to allow

the end user to assess the quality of the tracking data.

The Tracking module was implemented for a previous project, and thus its design and

implementation are covered in detail in that project’s report (Twist 2010). That report has

been reprinted, for convenience, in Appendix A of this thesis.

3.4 Tracking Smoothing

High-frequency, low-amplitude noise (known as “shake”) in the marker data is introduced as

a result of both the tracking process (Twist 2010) and due to physical shake in the camera,

introduced by small reverberations in the hardware setup caused by the motion of the actor’s

head and jaw.

This shake was ignored until the rest of the pipeline was implemented, at which time it

was determined that the shake had a detrimental impact on the animation quality (as later

mentioned in Section 5.3).

Thus, it was decided to pass the tracked marker data through a Tracking Smoothing

module, which acts as a low-pass ﬁlter (Wikipedia 2010c), removing this high frequency

noise from the motion signal of the markers. For every frame of marker data received by

the Tracking Smoothing module, a frame of smoothed marker data is output to the Motion

Capture Data Mapping module (Section 3.6).

CHAPTER 3. DESIGN Steven Twist

3.5 Blendshapes

The next stage in the facial motion capture pipeline after Tracking Smoothing (Section 3.4)

is Motion Capture Data Mapping (Section 3.6). However, it is important to understand

exactly the type of facial rig that needs to be driven by the motion capture data, in order to

understand the requirements of the Motion Capture Data Mapping module. Therefore, we

will ﬁrst discuss the Blendshapes module.

A key goal of this facial motion capture system is to animate any CG face, not just one of

close resemblance to the actor (Chapter 1). As discussed extensively in Section 2.2.2, in order

to animate any CG face one needs to break the connection between the geometric nature of

the actor’s face, and the geometry of the animated model (Sagar 2006). As earlier discussed

(Section 2.2.2), there are two main techniques currently being used in production to achieve

such a detachment: muscle-based (Choe et al. 2001, Havaldar 2006, and Sagar 2006), and

blendshap e- based (Chuang et al. 2002, and Joshi et al. 2006).

Both techniques are heavily related. The muscle-based approach analyses the motion

capture data to determine which facial muscles are acting, and then drives the corresponding

muscles in the facial rig (Sagar 2006). The blendshape-based approach analyses the motion

capture data to determine which blendshapes are acting, and then drives the corresponding

blendshap es in the facial rig.

Blendshape-based facial rigs are arguably much more simplistic than muscle-based facial

rigs, given that they consist of a simple linear combination of poses (Section 3.5.1). Given

the scope of this project, it was decided to use a blendshape-based facial rig, as we have

previous experience working with blendshapes and have access to existing blendshape-based

facial rigs for use with this project. In addition, it is our understanding that blendshapes are

computationally less expensive than muscle simulations (again, due to the simplicity of the

underlying mathematics), and thus would be better suited to a real-time application.

3.5.1 Blendshape Theory

A blendshape-based facial rig consists of a rest pose, and a number of target poses (Ritchie

et al. 2005). All poses are provided by the artist as topologically identical models.

However, rather than simply storing the true locations of every vertex in the target pose

mesh, a target blendshape is constructed by calculating the vector oﬀset of each vertex in

the target p ose from the corresponding rest pose vertex (Lorach 2007):

= T

− R

(3.1)

where T

is the position of vertex i in the target pose, R

is the position of vertex i in the

rest pose, and B

is the vector oﬀset of vertex i.

In this way, if one took the rest pose and added to each vertex the corresponding oﬀset

vector of a particular target blendshape, the resultant model would be that of the target pose

initially used to create that blendshape.

Each target blendshape has an associated weight parameter. Given a set of weights, the

resultant model is calculated as the weighted sum of all the blendshap es added to the rest

pose (Joshi et al. 2006):

= R

j=1

(3.2)

where M

is the resultant position of vertex i, w

is the weight parameter for blendshape

j, B

is the vector oﬀset of vertex i in blendshape j, and n is the number of blendshapes.

剩余64页未读，继续阅读

风翼冰舟

粉丝: 2485
资源: 54

实时面部动作捕捉系统与 Blendshape 技术

Real-time-Facial-Expression-Recognition-and-Fast-Face-Detection-master.zip

Algorithm-Facial-Recognition-Attendance-System.zip

3D Shape Regression for Real-time Facial Animation

101-Ways-to-Teach-Children-Social-Skills.pdf

computer-of-facial-cartoon.zip_图形图像处理_matlab_

Node.js-Facial-Recognition-master.zip

Real-Time-Facial-Expression-Recognition-with-DeepLearning

latest-model-099-94.7200.pth

Facial-Expression-Recognition.Pytorch-master.zip_2B facial_pytor

Real-Time Facial Recognition Using HOG Features：这个例子主要是演示如何实时注册新人脸、标记新人脸、提取特征和识别人脸。-matlab开发

最新资源