强化学习：2018年一月版完整草稿

需积分: 10 40 浏览量更新于2024-07-17 收藏 12.32MB PDF 举报

身份认证购VIP最低享 7 折!

30元优惠券

"《强化学习：入门》2018年一月版是Richard S. Sutton和Andrew G. Barto合著的英文书籍，该书是第二版的草稿，内容已经完成，可能还会在第16章增加一个案例研究。书中尚缺少索引，作者欢迎读者提供错误反馈和重要文献引用的遗漏。" 《强化学习：入门》是关于强化学习领域的一本经典著作，旨在介绍这一领域的核心概念和方法。强化学习是一种机器学习的方法，通过与环境的交互，智能体学习如何采取行动以最大化长期奖励。该书是第二版，且处于完整草稿状态，截止日期为2018年1月1日。书中涵盖的内容包括： 1. 强化学习的定义和基本原理：在第1章中，作者阐述了强化学习的概念，通过例子和元素解释了强化学习的工作机制。他们讨论了强化学习的局限性和应用范围，并通过一个扩展的例子——井字游戏（Tic-Tac-Toe）来进一步说明强化学习的过程。 2. 表格解决方案方法：强化学习的一个早期和基础方法是表格解决方案，这通常用于环境状态和动作空间相对较小的问题。这种方法涉及到存储所有可能状态-动作对的值，通过不断更新这些值来优化策略。 3. 学习和规划：书中可能涉及了不同的学习算法，如Q学习、SARSA等，以及动态规划等规划技术，这些都是强化学习中用于决策制定和策略改进的重要工具。 4. 函数逼近：对于状态和动作空间庞大的问题，表格解决方案变得不可行，因此函数逼近方法被引入，如神经网络或线性回归，用于近似值函数或策略。 5. 模型学习与预测：强化学习智能体可以学习环境模型，这有助于规划和策略选择。这部分可能会讨论模型学习的算法以及它们如何影响学习效率和性能。 6. 信用分配和延迟学习：强化学习中的一个关键挑战是如何正确地将奖励归因于过去的行为，这部分可能讨论了不同的信用分配策略，如回溯学习和蒙特卡洛方法。 7. 探索与利用：在强化学习中，探索与利用之间的平衡至关重要。书中可能介绍了ε-贪婪策略和其他策略，以在获取新知识和最大化当前奖励之间找到平衡。 8. 马尔科夫决策过程（MDP）和部分可观测马尔科夫决策过程（POMDP）：MDP和POMDP是描述强化学习问题的数学框架，书里会详细解释这些模型及其解决方案。 9. 深度强化学习：随着深度学习的发展，深度强化学习成为了一个热门话题，可能在书中会介绍如何结合深度神经网络进行强化学习，以及相关的突破和挑战。 10. 案例研究：除了理论，书中可能包含多个实际应用的案例，以便读者更好地理解强化学习在现实世界问题中的应用。 11. 历史回顾与文献：每章末尾的“文献与历史评论”部分回顾了相关领域的历史发展，并列出了重要的参考文献，作者希望读者指出可能遗漏的重要贡献。由于本书尚未完成最终版本，作者鼓励读者提供错误报告和文献引用的建议，以便在正式出版前进行修正。如果你发现了错误或者认为有重要文献未被引用，可以通过邮件与作者联系。这本书对于理解强化学习的理论和实践都是宝贵的资源。

资源详情

资源推荐

xvi Summary of Notation

|S| number of elements in set S

t discrete time step

T, T (t) ﬁnal time step of an episode, or of the episode including time step t

action at time t

state at time t, typically due, stochastically, to S

t−1

and A

t−1

reward at time t, typically due, stochastically, to S

t−1

and A

t−1

π policy (decision-making rule)

π(s) action taken in state s under deterministic policy π

π(a|s) probability of taking action a in state s under stochastic policy π

π(a|s, θ) probability of taking action a in state s given parameter vector θ

return (cumulative discounted reward) following time t (Section 3.3)

t:t+n

, G

t:h

n-step return from t + 1 to h(discounted and corrected)

t:h

ﬂat return (undiscounted and uncorrected) from t + 1 to h (Section 5.8)

λ-return (Section 12.1)

t:h

truncated, corrected λ-return (Section 12.3)

λs

λ-return, corrected by estimated state values (Section 12.8)

λa

λ-return, corrected by estimated action values (Section 12.8)

p(s

, r|s, a) probability of transition to state s

with reward r, from state s and action a

p(s

|s, a) probability of transition to state s

, from state s taking action a

r(s, a, s

) expected immediate reward on transition from s to s

under action a

(s) value of state s under policy π (expected return)

∗

(s) value of state s under the optimal policy

(s, a) value of taking action a in state s under policy π

∗

(s, a) value of taking action a in state s under the optimal policy

V, V

array estimates of state-value function v

or v

∗

Q, Q

array estimates of action-value function q

or q

∗

expected approximate action value, e.g.,

π(a|S

t−1

, a)

target for estimate at time t

temporal-diﬀerence error at t (a random variable) (Section 6.1)

w, w

d-vector of weights underlying an approximate value function

, w

t,i

ith component of learnable weight vector

d dimensionality—the number of components of w

alternate dimensionality—the number of components of θ

m number of 1s in a sparse binary feature vector

ˆv(s,w) approximate value of state s given weight vector w

(s) alternate notation for ˆv(s,w)

ˆq(s, a, w) approximate value of state–action pair s, a given weight vector w

x(s) vector of features visible when in state s

x(s, a) vector of features visible when in state s taking action a

(s), x

(s, a) ith component of vector x(s) or x(s, a)

shorthand for x(S

) or x(S

, A

)

x inner product of vectors, w

; e.g., ˆv(s,w)

= w

x(s)

µ(s) on-policy distribution over states (Section 9.2)

µ |S|-vector of the µ(s) for all s ∈ S

kxk

µ-weighted norm of any vector x(s), i.e.,

µ(s)x(s)

(Section 11.4)

Chapter 1

Introduction

The idea that we learn by interacting with our environment is probably the ﬁrst to occur to us when

we think about the nature of learning. When an infant plays, waves its arms, or looks about, it has no

explicit teacher, but it does have a direct sensorimotor connection to its environment. Exercising this

connection produces a wealth of information about cause and eﬀect, about the consequences of actions,

and about what to do in order to achieve goals. Throughout our lives, such interactions are undoubtedly

a major source of knowledge about our environment and ourselves. Whether we are learning to drive a

car or to hold a conversation, we are acutely aware of how our environment responds to what we do, and

we seek to inﬂuence what happens through our behavior. Learning from interaction is a foundational

idea underlying nearly all theories of learning and intelligence.

In this book we explore a computational approach to learning from interaction. Rather than directly

theorizing about how people or animals learn, we primarily explore idealized learning situations and

evaluate the eﬀectiveness of various learning methods.

That is, we adopt the perspective of an artiﬁcial

intelligence researcher or engineer. We explore designs for machines that are eﬀective in solving learning

problems of scientiﬁc or economic interest, evaluating the designs through mathematical analysis or

computational experiments. The approach we explore, called reinforcement learning, is much more

focused on goal-directed learning from interaction than are other approaches to machine learning.

1.1 Reinforcement Learning

Reinforcement learning is learning what to do—how to map situations to actions—so as to maximize

a numerical reward signal. The learner is not told which actions to take, but instead must discover

which actions yield the most reward by trying them. In the most interesting and challenging cases,

actions may aﬀect not only the immediate reward but also the next situation and, through that, all

subsequent rewards. These two characteristics—trial-and-error search and delayed reward—are the two

most important distinguishing features of reinforcement learning.

Reinforcement learning, like many topics whose names end with “ing,” such as machine learning

and mountaineering, is simultaneously a problem, a class of solution methods that work well on the

problem, and the ﬁeld that studies this problems and its solution methods. It is convenient to use a

single name for all three things, but at the same time essential to keep the three conceptually separate.

In particular, the distinction between problems and solution methods is very important in reinforcement

learning; failing to make this distinction is the source of a many confusions.

We formalize the problem of reinforcement learning using ideas from dynamical systems theory,

The relationships to psychology and neuroscience are summarized in Chapters 14 and 15.

因果

通过maximize一个numerical

reward 来决定action

反复试验法

试错法和延迟奖励，两个主要特征

2 CHAPTER 1. INTRODUCTION

speciﬁcally, as the optimal control of incompletely-known Markov decision processes. The details of this

formalization must wait until Chapter 3, but the basic idea is simply to capture the most important

aspects of the real problem facing a learning agent interacting over time with its environment to achieve

a goal. A learning agent must be able to sense the state of its environment to some extent and must be

able to take actions that aﬀect the state. The agent also must have a goal or goals relating to the state of

the environment. Markov decision processes are intended to include just these three aspects—sensation,

action, and goal—in their simplest possible forms without trivializing any of them. Any method that

is well suited to solving such problems we consider to be a reinforcement learning method.

Reinforcement learning is diﬀerent from supervised learning, the kind of learning studied in most

current research in the ﬁeld of machine learning. Supervised learning is learning from a training set

of labeled examples provided by a knowledgable external supervisor. Each example is a description of

a situation together with a speciﬁcation—the label—of the correct action the system should take to

that situation, which is often to identify a category to which the situation belongs. The object of this

kind of learning is for the system to extrapolate, or generalize, its responses so that it acts correctly

in situations not present in the training set. This is an important kind of learning, but alone it is

not adequate for learning from interaction. In interactive problems it is often impractical to obtain

examples of desired behavior that are both correct and representative of all the situations in which the

agent has to act. In uncharted territory—where one would expect learning to be most beneﬁcial—an

agent must be able to learn from its own experience.

Reinforcement learning is also diﬀerent from what machine learning researchers call unsupervised

learning, which is typically about ﬁnding structure hidden in collections of unlabeled data. The terms

supervised learning and unsupervised learning would seem to exhaustively classify machine learning

paradigms, but they do not. Although one might be tempted to think of reinforcement learning as a

kind of unsupervised learning because it does not rely on examples of correct behavior, reinforcement

learning is trying to maximize a reward signal instead of trying to ﬁnd hidden structure. Uncovering

structure in an agent’s experience can certainly be useful in reinforcement learning, but by itself does

not address the reinforcement learning problem of maximizing a reward signal. We therefore consider

reinforcement learning to be a third machine learning paradigm, alongside supervised learning and

unsupervised learning and perhaps other paradigms as well.

One of the challenges that arise in reinforcement learning, and not in other kinds of learning, is the

trade-oﬀ between exploration and exploitation. To obtain a lot of reward, a reinforcement learning

agent must prefer actions that it has tried in the past and found to be eﬀective in producing reward.

But to discover such actions, it has to try actions that it has not selected before. The agent has to

exploit what it has already experienced in order to obtain reward, but it also has to explore in order to

make better action selections in the future. The dilemma is that neither exploration nor exploitation

can be pursued exclusively without failing at the task. The agent must try a variety of actions and

progressively favor those that appear to be best. On a stochastic task, each action must be tried many

times to gain a reliable estimate of its expected reward. The exploration–exploitation dilemma has been

intensively studied by mathematicians for many decades, yet remains unresolved. For now, we simply

note that the entire issue of balancing exploration and exploitation does not even arise in supervised

and unsupervised learning, at least in their purest forms.

Another key feature of reinforcement learning is that it explicitly considers the whole problem of a

goal-directed agent interacting with an uncertain environment. This is in contrast to many approaches

that consider subproblems without addressing how they might ﬁt into a larger picture. For example, we

have mentioned that much of machine learning research is concerned with supervised learning without

explicitly specifying how such an ability would ﬁnally be useful. Other researchers have developed

theories of planning with general goals, but without considering planning’s role in real-time decision

making, or the question of where the predictive models necessary for planning would come from. Al-

though these approaches have yielded many useful results, their focus on isolated subproblems is a

signiﬁcant limitation.

学习智能体

三个方面：

1） sensation

2） action

3） goal

增强学习不同于unsupervised learning的地方

勘探和开发

间的

trad-off

考虑整个problem

剩余443页未读，继续阅读

sukey123

粉丝: 1
资源: 15

强化学习：2018年一月版完整草稿

Reinforcement Learning: An Introduction 2018年 第二版和之前2015中文翻译版

Reinforcement Learning: An Introduction（最新版书籍+代码，February 28, 2018）

reinforcement learning: an introduction.pdf

reinforcement learning : an introduction

reinforcement learning: an introduction

reinforcement learning an introduction 第2版 答案

reforcement learning an introduction电子书

推荐一本逆强化学习的资料

帮助学习强化学习的书籍

tensorflow 强化学习 书

reinforcement learning an introduction 答案

推荐一下学习AI高阶阶段的书籍

强化学习入门资料algorithms for reinforcement learning

多智能体强化学习推荐书籍

python强化学习书籍

推荐10本神经网络的教材

强化学习和知识图谱的学习资料

dqn算法书籍推荐系统

关于惩罚预测误差（punishment prediction error）的相关文献有哪些

适合人工智能深度学习的参考文献

最新资源

Reinforcement Learning: An Introduction 2018年第二版和之前2015中文翻译版

reinforcement learning an introduction 第2版答案

tensorflow 强化学习书