Interactive Learning from Policy-Dependent Human Feedback
James MacGlashan
1
Mark K Ho
2
Robert Loftin
3
Bei Peng
4
Guan Wang
2
David L. Roberts
3
Matthew E. Taylor
4
Michael L. Littman
2
Abstract
This paper investigates the problem of interac-
tively learning behaviors communicated by a hu-
man teacher using positive and negative feed-
back. Much previous work on this problem has
made the assumption that people provide feed-
back for decisions that is dependent on the be-
havior they are teaching and is independent from
the learner’s current policy. We present empirical
results that show this assumption to be false—
whether human trainers give a positive or neg-
ative feedback for a decision is influenced by
the learner’s current policy. Based on this in-
sight, we introduce Convergent Actor-Critic by
Humans (COACH), an algorithm for learning
from policy-dependent feedback that converges
to a local optimum. Finally, we demonstrate that
COACH can successfully learn multiple behav-
iors on a physical robot.
1. Introduction
Programming robots is very difficult, in part because
the real world is inherently rich and—to some degree—
unpredictable. In addition, our expectations for physical
agents are quite high and often difficult to articulate. Nev-
ertheless, for robots to have a significant impact on the lives
of individuals, even non-programmers need to be able to
specify and customize behavior. Because of these complex-
ities, relying on end-users to provide instructions to robots
programmatically seems destined to fail.
Reinforcement learning (RL) from human trainer feedback
provides a compelling alternative to programming because
agents can learn complex behavior from very simple posi-
tive and negative signals. Furthermore, real-world animal
training is an existence proof that people can train complex
*
Equal contribution
1
Cogitai
2
Brown University
3
North Car-
olina State University
4
Washington State University. Correspon-
dence to: James MacGlashan <james@cogitai.com>.
Proceedings of the 34
th
International Conference on Machine
Learning, Sydney, Australia, PMLR 70, 2017. Copyright 2017
by the author(s).
behavior using these simple signals. Indeed, animals have
been successfully trained to guide the blind, locate mines
in the ocean, detect cancer or explosives, and even solve
complex, multi-stage puzzles.
Despite success when learning from environmental reward,
traditional reinforcement-learning algorithms have yielded
limited success when the reward signal is provided by hu-
mans. This failure underscores the importance that algo-
rithms for learning from humans are based on appropriate
models of human-feedback. Indeed, much human-centered
RL work has investigated and employed different mod-
els of human-feedback (Knox & Stone, 2009b; Thomaz &
Breazeal, 2006; 2007; 2008; Griffith et al., 2013; Loftin
et al., 2015). Many of these algorithms leverage the ob-
servation that people tend to give feedback that is best in-
terpreted as guidance on the policy the agent should be fol-
lowing, rather than as a numeric value to be maximized
by the agent. However, these approaches assume models
of feedback that are independent of the policy the agent
is currently following. We present empirical results that
demonstrate that this assumption is incorrect and further
demonstrate cases in which policy-independent learning al-
gorithms suffer from this assumption. Following this result,
we present Convergent Actor-Critic by Humans (COACH),
an algorithm for learning from policy-dependent human
feedback. COACH is based on the insight that the ad-
vantage function (a value roughly corresponding to how
much better or worse an action is compared to the current
policy) provides a better model of human feedback, cap-
turing human-feedback properties like diminishing returns,
rewarding improvement, and giving 0-valued feedback a
semantic meaning that combats forgetting. We compare
COACH to other approaches in a simple domain with sim-
ulated feedback. Then, to validate that COACH scales to
complex problems, we train five different behaviors on a
TurtleBot robot.
2. Background
For modeling the underlying decision-making problem of
an agent being taught by a human, we adopt the Markov
Decision Process (MDP) formalism. An MDP is a 5-tuple:
hS, A, T, R, γi, where S is the set of possible states of the
下载后可阅读完整内容，剩余9页未读，立即下载
IT徐师兄
- 粉丝: 1980
- 资源: 2689
我的内容管理
展开
最新资源
- 解决Eclipse配置与导入Java工程常见问题
- 真空发生器：工作原理与抽吸性能分析
- 爱立信RBS6201开站流程详解
- 电脑开机声音解析：故障诊断指南
- JAVA实现贪吃蛇游戏
- 模糊神经网络实现与自学习能力探索
- PID型模糊神经网络控制器设计与学习算法
- 模糊神经网络在自适应PID控制器中的应用
- C++实现的学生成绩管理系统设计
- 802.1D STP 实现与优化：二层交换机中的生成树协议
- 解决Windows无法完成SD卡格式化的九种方法
- 软件测试方法：Beta与Alpha测试详解
- 软件测试周期详解：从需求分析到维护测试
- CMMI模型详解：软件企业能力提升的关键
- 移动Web开发框架选择：jQueryMobile、jQTouch、SenchaTouch对比
- Java程序设计试题与复习指南
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
