基于LSTM-VAE的多模态机器人执行监控系统

需积分: 7 71 浏览量更新于2024-07-18 收藏 24.19MB PDF 举报

本研究论文标题为《多模态执行监控器 for 辅助机器人》(A Multimodal Execution Monitor for Assistive Robots)，着重于开发一种创新的监控系统，旨在提升辅助机器人在执行任务过程中的性能管理和故障预防。该系统的核心技术是结合了长短时记忆网络(LSTM)与变分自编码器(VAE)的异常检测算法，这在机器学习领域中被广泛应用，特别是在预测维护分析中。 LSTM是一种递归神经网络(RNN)的变种，特别适用于处理序列数据，如时间序列数据。它通过记忆单元和门控机制来解决传统RNN的梯度消失和爆炸问题，能够捕捉长期依赖关系。在这个项目中，LSTM被用来理解和解析多传感器收集的复杂数据，例如机器人在执行任务时产生的各种动态行为模式。变分自编码器(VAE)则是一种生成模型，通过学习数据的潜在分布来进行无监督学习和数据压缩。它不仅能够生成新的样本，还能进行异常检测，因为异常值在编码后的低维空间中通常会偏离正常数据的分布。通过集成LSTM和VAE，论文提出的方法能够实时监测机器人的执行状态，识别出可能的异常行为，从而提前预测并防止设备故障。本文的研究背景是当前对辅助机器人性能优化的需求日益增长，特别是对于那些在医疗、护理或日常生活协助中扮演重要角色的机器人。一个有效的执行监控器能够提高系统的可靠性和效率，减少突发故障对用户的影响。此外，论文还可能探讨了数据融合、多模态信息处理（如视觉、听觉和触觉）在异常检测中的作用，以及如何通过深度学习模型优化对复杂环境的适应性。在研究过程中，作者Daehyung Park经历了长达五年半的辛勤工作，最终完成了这项具有深远意义的课题。他的成果得到了多位导师的认可，包括来自生物医学工程、互动计算学部和哈佛大学的专家们，他们的指导和支持对该研究起到了关键作用。这篇论文不仅展示了机器学习在机器人领域的实际应用，也为未来在更智能、更自主的机器人设计中引入更高级的故障预警和自我修复能力奠定了基础。总结来说，该论文主要贡献在于开发了一种多模态执行监控器，通过LSTM-VAE的集成实现了对辅助机器人异常行为的高效检测和预防，对于提升机器人系统的整体性能和用户体验具有重要的实践价值。

5.8 Our previous system [8] estimated the location of the user’s mouth using a

Kinect V2 and an ARTag attached to the user’s forehead. . . . . . . . . . . 107

5.9 Estimated mouth poses from our vision-based mouth pose estimator. . . . . 107

5.10 A variety of sensors we used to monitor task executions. This ﬁgure shows

a feeding demonstration with an able-bodied person. . . . . . . . . . . . . . 109

5.11 Anomaly Detection Framework using hidden Markov models. . . . . . . . 109

5.12 Captures of the long-term evaluation by the author. Our feeding system fed

6 types of foods to the author for 22 days. . . . . . . . . . . . . . . . . . . 112

5.13 Examples of foods we used during the long-term evaluation. . . . . . . . . 112

5.14 The number of daily successful and failed feeding executions during the

long-term evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

5.15 Our system feeding yogurt to an able-bodied person. A person with quadriple-

gia commanded the system remotely. . . . . . . . . . . . . . . . . . . . . . 114

5.16 Left Henry Evans, a person with severe quadriplegia, successfully used

our robot-assisted feeding system in his home to feed himself while our

execution monitor was running. Right A capture of Henry’s head tracker

and one-button mouse. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

5.17 Evaluation with a person with disabilities in an experiment room in Georgia

Tech. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

5.18 Distributions of robot-assisted feeding durations from 3 able-bodied partic-

ipants and 3 people with disabilities (p-value = 0.0004 with a two-sided

Welch’s t-test). The participants ate yogurt with a silicone spoon. . . . . . . 118

5.19 9 able-bodied participants (left) and 3 people with disabilities (right) re-

ported the level of agreement with the statement ‘the system is easy-to-use

to performing self-feeding task.’ The two-sided Welch’s t-test resulted in

p-value = 0.2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

5.20 9 able-bodied participants (Left) and 3 people with disabilities (Right) re-

ported the level of agreement with the statement ‘the system is comfortable

to performing self-feeding task.’ The two-sided Welch’s t-test resulted in

p-value = 0.28. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

xvi

SUMMARY

Assistive robots have the potential to serve as caregivers, providing assistance with ac-

tivities of daily living (ADLs) and instrumental activities of daily living (IADLs) to people

with disabilities. Yet for many people with disabilities, including the people with upper

limb disabilities or quadriplegia, such tasks prove challenging without assistance from a

human caregiver. Monitoring when something has gone wrong could help assistive robots

operate more safely and effectively around people. However, the complexity of interacting

with people and objects in human environments can make challenges in monitoring op-

erations. By monitoring multimodal sensory signals, an execution monitor could perform

a variety of roles, such as detecting success, determining when to switch behaviors, and

otherwise exhibiting more common sense.

The purpose of this dissertation is to introduce a multimodal execution monitor to im-

prove safety and success of assistive manipulation services. To accomplish this goal, we

make three main contributions. First, we introduce a data-driven anomaly detector, a part

of the monitor, that reports anomalous task executions from multimodal sensory signals

online. The system’s anomaly detector models multimodal sensory signals from typical

non-anomalous task executions using classic or state-of-the-art machine-learning methods:

a hidden Markov model (HMM) and a long-short-term-memory based variational autoen-

coder (LSTM-VAE). It then determines an anomaly that is biased from the model using a

detection threshold that varies based on the state of the task execution.

Second, we introduce a data-driven anomaly classiﬁer that recognizes the type and

cause of common anomalies through an artiﬁcial neural network after fusing multimodal

features. Our method recognizes representative anomalies that are more likely to occur dur-

ing the direct task at hand. For robot-assisted feeding, we identiﬁed 12 anomalies through

fault tree analysis which is a deductive analysis approach for resolving system hazards into

their causes.

xviii

剩余163页未读，继续阅读

云中鹤2015

粉丝: 0
资源: 3

基于LSTM-VAE的多模态机器人执行监控系统

ios-AssistiveTouch效果-跟随手指移动的悬浮button.zip

类似于苹果AssistiveTouch.zip

ios-AssistiveTouch简仿.zip

multimodal token fusion for vision transformers

unimodal models and multimodal models

a hybrid method for traffic flow forecasting using multimodal deep learning

Knowledge-driven Egocentric Multimodal Activity Recognition

exploration of deep learning-based multimodal fusion for semantic road scene

多模态在线哈希的国外研究现状

cmu multimodal data sdk下载

最新资源