没有合适的资源?快使用搜索试试~ 我知道了~
首页政策依赖的人类反馈互动学习:COACH算法与机器人行为优化
"《基于策略依赖的人类反馈的交互式学习》 该论文探讨了人类教师通过积极和消极反馈指导机器人进行交互式学习的问题。以往的研究假设,人们提供的反馈是基于他们教授的行为,与学习者的当前策略无关。然而,研究者通过实证发现,这一假设并不成立,人类教练对决策的反馈(无论是正面还是负面)实际上会受到学习者当前策略的影响。 基于这一洞察,论文提出了一种名为Convergent Actor-Critic by Humans (COACH) 的算法。COACH 是一种针对策略依赖性反馈设计的学习方法,旨在收敛到局部最优解。它特别强调在实际环境中,如机器人编程,由于现实世界的复杂性和不确定性,这种对于策略敏感的反馈处理方式显得尤为重要。 在介绍中,作者指出机器人编程之所以困难,很大程度上是因为外部环境的复杂性,以及其内在的难以预测性。传统的假设无法充分捕捉到人类指导时的动态反馈模式,而COACH算法的引入则试图弥补这一空白,使得机器人能够更有效地从人类教师的策略相关反馈中学习和适应。 研究者通过在一个物理机器人上展示了COACH的实际应用,证明了它在学习多个行为模式方面的有效性。这不仅验证了理论模型的实用性,也为在人工智能领域,特别是聊天机器人(如ChatGPT)的设计中,如何更好地理解和利用人类教师的反馈提供了新的思路和方法论。 这篇论文深化了我们对人类与机器学习系统交互的理解,强调了策略依赖性反馈在交互式学习中的核心作用,并为未来开发出更加智能、适应性强的学习算法奠定了基础。"
资源详情
资源推荐
![](https://csdnimg.cn/release/download_crawler_static/87795269/bg1.jpg)
Interactive Learning from Policy-Dependent Human Feedback
James MacGlashan
1
Mark K Ho
2
Robert Loftin
3
Bei Peng
4
Guan Wang
2
David L. Roberts
3
Matthew E. Taylor
4
Michael L. Littman
2
Abstract
This paper investigates the problem of interac-
tively learning behaviors communicated by a hu-
man teacher using positive and negative feed-
back. Much previous work on this problem has
made the assumption that people provide feed-
back for decisions that is dependent on the be-
havior they are teaching and is independent from
the learner’s current policy. We present empirical
results that show this assumption to be false—
whether human trainers give a positive or neg-
ative feedback for a decision is influenced by
the learner’s current policy. Based on this in-
sight, we introduce Convergent Actor-Critic by
Humans (COACH), an algorithm for learning
from policy-dependent feedback that converges
to a local optimum. Finally, we demonstrate that
COACH can successfully learn multiple behav-
iors on a physical robot.
1. Introduction
Programming robots is very difficult, in part because
the real world is inherently rich and—to some degree—
unpredictable. In addition, our expectations for physical
agents are quite high and often difficult to articulate. Nev-
ertheless, for robots to have a significant impact on the lives
of individuals, even non-programmers need to be able to
specify and customize behavior. Because of these complex-
ities, relying on end-users to provide instructions to robots
programmatically seems destined to fail.
Reinforcement learning (RL) from human trainer feedback
provides a compelling alternative to programming because
agents can learn complex behavior from very simple posi-
tive and negative signals. Furthermore, real-world animal
training is an existence proof that people can train complex
*
Equal contribution
1
Cogitai
2
Brown University
3
North Car-
olina State University
4
Washington State University. Correspon-
dence to: James MacGlashan <james@cogitai.com>.
Proceedings of the 34
th
International Conference on Machine
Learning, Sydney, Australia, PMLR 70, 2017. Copyright 2017
by the author(s).
behavior using these simple signals. Indeed, animals have
been successfully trained to guide the blind, locate mines
in the ocean, detect cancer or explosives, and even solve
complex, multi-stage puzzles.
Despite success when learning from environmental reward,
traditional reinforcement-learning algorithms have yielded
limited success when the reward signal is provided by hu-
mans. This failure underscores the importance that algo-
rithms for learning from humans are based on appropriate
models of human-feedback. Indeed, much human-centered
RL work has investigated and employed different mod-
els of human-feedback (Knox & Stone, 2009b; Thomaz &
Breazeal, 2006; 2007; 2008; Griffith et al., 2013; Loftin
et al., 2015). Many of these algorithms leverage the ob-
servation that people tend to give feedback that is best in-
terpreted as guidance on the policy the agent should be fol-
lowing, rather than as a numeric value to be maximized
by the agent. However, these approaches assume models
of feedback that are independent of the policy the agent
is currently following. We present empirical results that
demonstrate that this assumption is incorrect and further
demonstrate cases in which policy-independent learning al-
gorithms suffer from this assumption. Following this result,
we present Convergent Actor-Critic by Humans (COACH),
an algorithm for learning from policy-dependent human
feedback. COACH is based on the insight that the ad-
vantage function (a value roughly corresponding to how
much better or worse an action is compared to the current
policy) provides a better model of human feedback, cap-
turing human-feedback properties like diminishing returns,
rewarding improvement, and giving 0-valued feedback a
semantic meaning that combats forgetting. We compare
COACH to other approaches in a simple domain with sim-
ulated feedback. Then, to validate that COACH scales to
complex problems, we train five different behaviors on a
TurtleBot robot.
2. Background
For modeling the underlying decision-making problem of
an agent being taught by a human, we adopt the Markov
Decision Process (MDP) formalism. An MDP is a 5-tuple:
hS, A, T, R, γi, where S is the set of possible states of the
下载后可阅读完整内容,剩余9页未读,立即下载
![.zip](https://img-home.csdnimg.cn/images/20210720083646.png)
![pdf](https://img-home.csdnimg.cn/images/20210720083512.png)
![pdf](https://img-home.csdnimg.cn/images/20210720083512.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://profile-avatar.csdnimg.cn/default.jpg!1)
IT徐师兄
- 粉丝: 1980
- 资源: 2689
上传资源 快速赚钱
我的内容管理 展开
我的资源 快来上传第一个资源
我的收益
登录查看自己的收益我的积分 登录查看自己的积分
我的C币 登录后查看C币余额
我的收藏
我的下载
下载帮助
![](https://csdnimg.cn/release/wenkucmsfe/public/img/voice.245cc511.png)
最新资源
- 解决Eclipse配置与导入Java工程常见问题
- 真空发生器:工作原理与抽吸性能分析
- 爱立信RBS6201开站流程详解
- 电脑开机声音解析:故障诊断指南
- JAVA实现贪吃蛇游戏
- 模糊神经网络实现与自学习能力探索
- PID型模糊神经网络控制器设计与学习算法
- 模糊神经网络在自适应PID控制器中的应用
- C++实现的学生成绩管理系统设计
- 802.1D STP 实现与优化:二层交换机中的生成树协议
- 解决Windows无法完成SD卡格式化的九种方法
- 软件测试方法:Beta与Alpha测试详解
- 软件测试周期详解:从需求分析到维护测试
- CMMI模型详解:软件企业能力提升的关键
- 移动Web开发框架选择:jQueryMobile、jQTouch、SenchaTouch对比
- Java程序设计试题与复习指南
资源上传下载、课程学习等过程中有任何疑问或建议,欢迎提出宝贵意见哦~我们会及时处理!
点击此处反馈
![](https://img-home.csdnimg.cn/images/20220527035711.png)
![](https://img-home.csdnimg.cn/images/20220527035711.png)
![](https://img-home.csdnimg.cn/images/20220527035111.png)
安全验证
文档复制为VIP权益,开通VIP直接复制
![](https://csdnimg.cn/release/wenkucmsfe/public/img/green-success.6a4acb44.png)