高维连续控制：利用广义优势估计强化学习
需积分: 9 124 浏览量
更新于2024-09-06
收藏 1.71MB PDF 举报
"这篇论文是关于在高维连续控制中使用广义优势估计（Generalized Advantage Estimation，GAE）的，它在强化学习领域具有重要应用。该论文由John Schulman等人在ICLR 2016会议上发表，来自加州大学伯克利分校的电气工程和计算机科学系。"
在强化学习中，策略梯度方法因其能直接优化累积奖励且易于与神经网络等非线性函数近似器结合而受到欢迎。然而，这些方法通常需要大量的样本，并且在面对不断变化的数据时，稳定性和持续改进的难度较大。针对这两个挑战，论文提出了以下解决方案：
首先，为了解决样本需求量大的问题，论文引入了价值函数来显著降低策略梯度估计的方差，尽管这会导致一定的偏差。这里采用了一种指数加权的Advantage函数估计器，类似于Temporal Difference (TD) 学习中的TD(λ)算法。Advantage函数是状态动作值函数Q(s,a)与状态值函数V(s)的差，用于衡量在特定状态下采取某一行动相对于采取最优行动的额外收益。
其次，为了解决数据非平稳导致的优化稳定性问题，论文采用了信任区域优化策略。这种策略同时应用于策略和价值函数的优化，通过限制策略更新的幅度，确保每次更新都在一个确定的信任区域内，从而提高训练的稳定性。信任区域优化能够防止策略过快地变化，避免了因为策略的剧烈波动而导致的学习不稳定。
此外，广义优势估计（GAE）通过平滑优势函数，减少了在连续空间中选择行动的敏感性，从而增强了策略学习的效率。它通过一种线性组合的优势估计，结合了即时和延迟的奖励信息，既降低了方差，又保持了长期奖励的考虑。
这篇论文通过引入GAE和信任区域优化，提供了一种在高维度连续控制任务中更有效、更稳定的强化学习策略。这种方法不仅减少了所需样本数量，还提高了学习过程的稳定性，对于处理复杂环境和高维输入的智能系统设计具有重要的理论和实践意义。
Published as a conference paper at ICLR 2016
Here, the subscript of E enumerates the variables being integrated over, where states and actions are
sampled sequentially from the dynamics model P (s
t+1
| s
t
, a
t
) and policy π(a
t
| s
t
), respectively.
The colon notation a : b refers to the inclusive range (a, a + 1, . . . , b). These formulas are well
known and straightforward to obtain; they follow directly from Proposition 1, which will be stated
shortly.
The choice Ψ
t
= A
π
(s
t
, a
t
) yields almost the lowest possible variance, though in practice, the
advantage function is not known and must be estimated. This statement can be intuitively justified by
the following interpretation of the policy gradient: that a step in the policy gradient direction should
increase the probability of better-than-average actions and decrease the probability of worse-than-
average actions. The advantage function, by it’s definition A
π
(s, a) = Q
π
(s, a) − V
π
(s), measures
whether or not the action is better or worse than the policy’s default behavior. Hence, we should
choose Ψ
t
to be the advantage function A
π
(s
t
, a
t
), so that the gradient term Ψ
t
∇
θ
log π
θ
(a
t
| s
t
)
points in the direction of increased π
θ
(a
t
| s
t
) if and only if A
π
(s
t
, a
t
) > 0. See Greensmith et al.
(2004) for a more rigorous analysis of the variance of policy gradient estimators and the effect of
using a baseline.
We will introduce a parameter γ that allows us to reduce variance by downweighting rewards cor-
responding to delayed effects, at the cost of introducing bias. This parameter corresponds to the
discount factor used in discounted formulations of MDPs, but we treat it as a variance reduction
parameter in an undiscounted problem; this technique was analyzed theoretically by Marbach &
Tsitsiklis (2003); Kakade (2001b); Thomas (2014). The discounted value functions are given by:
V
π,γ
(s
t
)
:
= E
s
t+1:∞
,
a
t:∞
"
∞
X
l=0
γ
l
r
t+l
#
Q
π,γ
(s
t
, a
t
)
:
= E
s
t+1:∞
,
a
t+1:∞
"
∞
X
l=0
γ
l
r
t+l
#
(4)
A
π,γ
(s
t
, a
t
)
:
= Q
π,γ
(s
t
, a
t
) − V
π,γ
(s
t
). (5)
The discounted approximation to the policy gradient is defined as follows:
g
γ
:
= E
s
0:∞
a
0:∞
"
∞
X
t=0
A
π,γ
(s
t
, a
t
)∇
θ
log π
θ
(a
t
| s
t
)
#
. (6)
The following section discusses how to obtain biased (but not too biased) estimators for A
π,γ
, giving
us noisy estimates of the discounted policy gradient in Equation (6).
Before proceeding, we will introduce the notion of a γ-just estimator of the advantage function,
which is an estimator that does not introduce bias when we use it in place of A
π,γ
(which is not
known and must be estimated) in Equation (6) to estimate g
γ
.
1
Consider an advantage estimator
ˆ
A
t
(s
0:∞
, a
0:∞
), which may in general be a function of the entire trajectory.
Definition 1. The estimator
ˆ
A
t
is γ-just if
E
s
0:∞
a
0:∞
h
ˆ
A
t
(s
0:∞
, a
0:∞
)∇
θ
log π
θ
(a
t
| s
t
)
i
= E
s
0:∞
a
0:∞
[A
π,γ
(s
t
, a
t
)∇
θ
log π
θ
(a
t
| s
t
)] . (7)
It follows immediately that if
ˆ
A
t
is γ-just for all t, then
E
s
0:∞
a
0:∞
"
∞
X
t=0
ˆ
A
t
(s
0:∞
, a
0:∞
)∇
θ
log π
θ
(a
t
| s
t
)
#
= g
γ
(8)
One sufficient condition for
ˆ
A
t
to be γ-just is that
ˆ
A
t
decomposes as the difference between two
functions Q
t
and b
t
, where Q
t
can depend on any trajectory variables but gives an unbiased estimator
of the γ-discounted Q-function, and b
t
is an arbitrary function of the states and actions sampled
before a
t
.
Proposition 1. Suppose that
ˆ
A
t
can be written in the form
ˆ
A
t
(s
0:∞
, a
0:∞
) = Q
t
(s
t:∞
, a
t:∞
) −
b
t
(s
0:t
, a
0:t−1
) such that for all (s
t
, a
t
), E
s
t+1:∞
,a
t+1:∞
| s
t
,a
t
[Q
t
(s
t:∞
, a
t:∞
)] = Q
π,γ
(s
t
, a
t
).
Then
ˆ
A is γ-just.
1
Note, that we have already introduced bias by using A
π,γ
in place of A
π
; here we are concerned with
obtaining an unbiased estimate of g
γ
, which is a biased estimate of the policy gradient of the undiscounted
MDP.
3
剩余13页未读，继续阅读
2018-09-13 上传
2019-01-23 上传
2021-03-19 上传
2021-03-19 上传
2021-06-01 上传
2020-06-16 上传
2020-02-20 上传
2019-08-15 上传
2017-05-31 上传
GanD.GanD
- 粉丝: 3
- 资源: 90
我的内容管理
展开
最新资源
- 探索AVL树算法：以Faculdade Senac Porto Alegre实践为例
- 小学语文教学新工具：创新黑板设计解析
- Minecraft服务器管理新插件ServerForms发布
- MATLAB基因网络模型代码实现及开源分享
- 全方位技术项目源码合集：***报名系统
- Phalcon框架实战案例分析
- MATLAB与Python结合实现短期电力负荷预测的DAT300项目解析
- 市场营销教学专用查询装置设计方案
- 随身WiFi高通210 MS8909设备的Root引导文件破解攻略
- 实现服务器端级联：modella与leveldb适配器的应用
- Oracle Linux安装必备依赖包清单与步骤
- Shyer项目：寻找喜欢的聊天伙伴
- MEAN堆栈入门项目： postings-app
- 在线WPS办公功能全接触及应用示例
- 新型带储订盒订书机设计文档
- VB多媒体教学演示系统源代码及技术项目资源大全
