多模态人机交互：方法综述与挑战探讨

版权申诉

7 浏览量更新于2024-06-26 1 收藏 523KB PDF 举报

身份认证购VIP最低享 7 折!

领优惠券(最高得80元）

多模态人机交互综述是一篇深入探讨了MMHCI（Multi-Modal Human-Computer Interaction）领域的重要文献，由Alejandro Jaimes和Nicu Sebe于2007年发表在《Computer Vision and Image Understanding》上。文章旨在全面回顾和分析该领域的关键方法，重点关注身体、手势、视线和情感交互，如人脸表情识别和语音中的情感表达。作者强调了在设计多模态交互系统时需要考虑的跨学科知识，如心理学、认知科学、社会学、工效学以及计算机科学，以提高交互的可用性和用户体验。文章的引入部分指出，MMHCI的研究目标是通过理解用户、交互系统和用户与系统之间的互动，提升计算机技术对人类的友好程度。由于涉及多个学科，因此研究者需要具备广泛的知识背景，以便设计出适应不同用户需求的交互系统。尽管技术进步推动了单模态技术（如语音和音频处理，以及计算机视觉）和硬件设备的发展，但多模态方法的融合已经成为MMHCI的核心挑战。 1.1 动机部分阐述了多模态交互研究的驱动力，即模拟人类自然的沟通方式，其中声音和视觉信息交织在一起。随着低成本摄像头和其他传感器的发展，研究者得以探索如何有效地结合语音和视觉输入，这为MMHCI带来了显著的进步。传统的单模态人机交互应用已经不足以满足复杂且真实的用户场景，因此，多模态交互成为未来发展的趋势，它涵盖了诸如增强现实、虚拟现实、智能家居等新兴应用。文章深入讨论了用户建模、多模态融合策略，以及在这个领域面临的挑战，例如数据集成、模式识别的准确性、用户隐私保护等。同时，它也指出了一些开放的研究课题，比如如何更好地理解和预测用户的情感状态，以及如何设计出能无缝适应各种情境的交互界面。这篇综述论文为读者提供了一个关于多模态人机交互的全面框架，强调了其跨学科性质，并预示了未来研究的重点方向和应用潜力。对于任何关注人机交互技术、特别是寻求在现实世界场景中提升用户体验的从业者来说，这篇文章具有重要的参考价值。

资源详情

资源推荐

performing a particular action. The parameters of the model are then a description of the hand pose or

trajectory and depend on the modeling approach used. Among the important problems involved in the

analysis are hand localization [187], hand tracking [194], and the selection of suitable features [83].

After the parameters are computed, the gestures represented by them need to be classified and

interpreted based on the accepted model and based on some grammar rules that reflect the internal

syntax of gestural commands. The grammar may also encode the interaction of gestures with other

communication modes such as speech, gaze, or facial expressions. As an alternative to modeling,

some authors have explored the use of combinations of simple 2D motion based detectors for gesture

recognition [71].

In any case, to fully exploit the potential of gestures for an MMHCI application, the class of

possible recognized gestures should be as broad as possible and ideally any gesture performed by

the user should be unambiguously interpretable by the interface. However, most of the gesture-based

HCI systems allow only symbolic commands based on hand posture or 3D pointing. This is due to the

complexity associated with gesture analysis and the desire to build real-time interfaces. Also, most of

the systems accommodate only single-hand gestures. Yet, human gestures, especially

communicative gestures, naturally employ actions of both hands. However, if the two-hand gestures

are to be allowed, several ambiguous situations may appear (e.g., occlusion of hands, intentional vs.

unintentional, etc.) and the processing time will likely increase. Another important aspect that is

increasingly considered is the use of other modalities (e.g., speech) to augment the MMHCI system

[127,162]. The use of such multimodal approaches can reduce the complexity and increase the

naturalness of the interface for MMHCI [126].

3.3. Gaze detection

Gaze, defined as the direction to which the eyes are pointing in space, is a strong indicator of

attention, and it has been studied extensively since as early as 1879 in psychology, and more recently

in neuroscience and in computing applications[41]. While early eye tracking research focused only on

systems for in-lab experiments, many commercial and experimental systems are available today for a

wide range of applications.

Eye tracking systems can be grouped into wearable or non-wearable, and infrared-based or

appearance-based. In infrared-based systems, a light shining on the subject whose gaze is to be

tracked creates a ‘‘red-eye effect:’’ the difference in reflection between the cornea and the pupil

is used to determine the direction of sight. In appearance based systems, computer vision techniques

are used to find the eyes in the image and then determine their orientation. While wearable systems

are the most accurate (approximate error rates below 1.4_ vs. errors below 1.7_ for nonwearable

infrared), they are also the most intrusive. Infrared systems are more accurate than

appearance-based, but there are concerns over the safety of prolonged exposure to infrared lights. In

addition, most non-wearable systems require (often cumbersome) calibration for each individual

[108,121].

Appearance-based systems usually capture both eyes using two cameras to predict gaze

direction. Due to the computational cost of processing two streams simultaneously, the resolution of

the image of each eye is often small. This makes such systems less accurate, although increasing

computational power and lower costs mean that more computationally intensive algorithms can be run

in real time. As an alternative, in [181], the authors propose using a single high-resolution image of

one eye to improve accuracy. On the other hand, infrared-based systems usually use only one

camera, but the use of two cameras has been proposed to further increase accuracy[152].

Although most research on non-wearable systems has focused on desktop users, the ubiquity of

computing devices has allowed for application in other domains in which the user is stationary (e.g.,

[168,152]). For example, the authors of [168] monitor driver visual attention using a single,

non-wearable camera placed on a car’s dashboard to track face features and for gaze detection.

Wearable eye trackers have also been investigated mostly for desktop applications (or for users

that do not walk wearing the device). Also, because of advances in hardware (e.g., reduction in size

and weight) and lower costs, researchers have been able to investigate uses in novel applications.

For example, in [193], eye tracking data are combined with video from the user’s perspective, head

directions, and hand motions to learn words from natural interactions with users; the authors of [137]

use a wearable eye tracker to understand hand–eye coordination in natural tasks, and the authors of

[38] use a wearable eye tracker to detect eye contact and record video for blogging.

The main issues in developing gaze tracking systems are intrusiveness, speed, robustness, and

accuracy. The type of hardware and algorithms necessary, however, depend highly on the level of

analysis desired. Gaze analysis can be performed at three different levels [23]: (a) highly detailed

low-level micro-events, (b) low-level intentional events, and (c) coarse-level goal-based events.

Micro-events include micro-saccades, jitter, nystagmus, and brief fixations, which are studied for their

physiological and psychological relevance by vision scientists and psychologists. Low-level intentional

events are the smallest coherent units of movement that the user is aware of during visual activity,

which include sustained fixations and revisits. Although most of the work on HCI has focused on

coarse-level goal based events (e.g., using gaze as a pointer [165]), it is easy to foresee the

importance of analysis at lower levels, particularly to infer the user’s cognitive state in affective

interfaces (e.g., [62]). Within this context, an important issue often overlooked is how to interpret

eye-tracking data. In other words, as the user moves his eyes during interaction, the system must

decide what the movements mean in order to react accordingly. We move our eyes 2 –3 times per

second, so a system may have to process large amounts of data within a short time, a task that is not

trivial even if processing does not occur in real-time. One way to interpret eye tracking data is to

cluster fixation points and assume, for instance, that clusters correspond to areas of interest.

Clustering of fixation points is only one option, however, and as the authors of[154] discuss, it can be

剩余30页未读，继续阅读

若♡

粉丝: 6228
资源: 1万+

会员权益专享

多模态人机交互：方法综述与挑战探讨

多模态人机交互综述(译文).docx

面向深度学习的多模态融合技术研究综述.pdf

多模态深度学习综述.pdf

python多模态人机交互

面向深度学习的多模态融合技术研究综述_何俊.pdf

多模态新闻推荐系统综述

给我推荐20个比流行的多模态模型

雷达多模态人体活动识别综述

多模态交互式AI案例分析

大模型技术进化论:多模态大模型综述 pdf

AIGC的多模态融合和交互

多模态图像融合算法综述

多模态假新闻检测综述

多模态特征交互的研究挑战是什么

多模态交互的国外研究现状

人机交互技术的研究2000字

给我推荐20个多模态模型

查找多模态情感识别的论文

多模态只指哪些多模态

会员权益专享

最新资源