Trust Region Policy Optimization
This problem imposes a constraint that the KL divergence
is bounded at every point in the state space. While it is
motivated by the theory, this problem is impractical to solve
due to the large number of constraints. Instead, we can use
a heuristic approximation which considers the average KL
divergence:
D
ρ
KL
(θ
1
, θ
2
)
:
= E
s∼ρ
[D
KL
(π
θ
1
(·|s) k π
θ
2
(·|s))] .
We therefore propose solving the following optimization
problem to generate a policy update:
maximize
θ
L
θ
old
(θ) (12)
subject to D
ρ
θ
old
KL
(θ
old
, θ) ≤ δ.
Similar policy updates have been proposed in prior work
(Bagnell & Schneider, 2003; Peters & Schaal, 2008b; Pe-
ters et al., 2010), and we compare our approach to prior
methods in Section 7 and in the experiments in Section 8.
Our experiments also show that this type of constrained
update has similar empirical performance to the maximum
KL divergence constraint in Equation (11).
5 Sample-Based Estimation of the Objective
and Constraint
The previous section proposed a constrained optimization
problem on the policy parameters (Equation (12)), which
optimizes an estimate of the expected total reward η sub-
ject to a constraint on the change in the policy at each up-
date. This section describes how the objective and con-
straint functions can be approximated using Monte Carlo
simulation.
We seek to solve the following optimization problem, ob-
tained by expanding L
θ
old
in Equation (12):
maximize
θ
X
s
ρ
θ
old
(s)
X
a
π
θ
(a|s)A
θ
old
(s, a)
subject to D
ρ
θ
old
KL
(θ
old
, θ) ≤ δ. (13)
We first replace
P
s
ρ
θ
old
(s) [. . . ] in the objective by the ex-
pectation
1
1−γ
E
s∼ρ
θ
old
[. . . ]. Next, we replace the advan-
tage values A
θ
old
by the Q-values Q
θ
old
in Equation (13),
which only changes the objective by a constant. Last, we
replace the sum over the actions by an importance sampling
estimator. Using q to denote the sampling distribution, the
contribution of a single s
n
to the loss function is
X
a
π
θ
(a|s
n
)A
θ
old
(s
n
, a) = E
a∼q
π
θ
(a|s
n
)
q(a|s
n
)
A
θ
old
(s
n
, a)
.
Our optimization problem in Equation (13) is exactly
equivalent to the following one, written in terms of expec-
tations:
maximize
θ
E
s∼ρ
θ
old
,a∼q
π
θ
(a|s)
q(a|s)
Q
θ
old
(s, a)
(14)
subject to E
s∼ρ
θ
old
[D
KL
(π
θ
old
(·|s) k π
θ
(·|s))] ≤ δ.
all state-action
pairs used in
objective
trajectories
s
a
nn
ρ
0
1
a
2
s
n
rollout set
two rollouts
using CRN
sampling
trajectories
ρ
0
Figure 1. Left: illustration of single path procedure. Here, we
generate a set of trajectories via simulation of the policy and in-
corporate all state-action pairs (s
n
, a
n
) into the objective. Right:
illustration of vine procedure. We generate a set of “trunk” tra-
jectories, and then generate “branch” rollouts from a subset of the
reached states. For each of these states s
n
, we perform multiple
actions (a
1
and a
2
here) and perform a rollout after each action,
using common random numbers (CRN) to reduce the variance.
All that remains is to replace the expectations by sample
averages and replace the Q value by an empirical estimate.
The following sections describe two different schemes for
performing this estimation.
The first sampling scheme, which we call single path, is
the one that is typically used for policy gradient estima-
tion (Bartlett & Baxter, 2011), and is based on sampling
individual trajectories. The second scheme, which we call
vine, involves constructing a rollout set and then perform-
ing multiple actions from each state in the rollout set. This
method has mostly been explored in the context of policy it-
eration methods (Lagoudakis & Parr, 2003; Gabillon et al.,
2013).
5.1 Single Path
In this estimation procedure, we collect a sequence of
states by sampling s
0
∼ ρ
0
and then simulating the pol-
icy π
θ
old
for some number of timesteps to generate a trajec-
tory s
0
, a
0
, s
1
, a
1
, . . . , s
T −1
, a
T −1
, s
T
. Hence, q(a|s) =
π
θ
old
(a|s). Q
θ
old
(s, a) is computed at each state-action
pair (s
t
, a
t
) by taking the discounted sum of future rewards
along the trajectory.
5.2 Vine
In this estimation procedure, we first sample s
0
∼ ρ
0
and
simulate the policy π
θ
i
to generate a number of trajecto-
ries. We then choose a subset of N states along these tra-
jectories, denoted s
1
, s
2
, . . . , s
N
, which we call the “roll-
out set”. For each state s
n
in the rollout set, we sample
K actions according to a
n,k
∼ q(·|s
n
). Any choice of
q(·|s
n
) with a support that includes the support of π
θ
i
(·|s
n
)
will produce a consistent estimator. In practice, we found
that q(·|s
n
) = π
θ
i
(·|s
n
) works well on continuous prob-
lems, such as robotic locomotion, while the uniform dis-
tribution works well on discrete tasks, such as the Atari
games, where it can sometimes achieve better exploration.
For each action a
n,k
sampled at each state s
n
, we esti-
剩余15页未读，继续阅读
TimothyTan
- 粉丝: 1
- 资源: 4
我的内容管理
展开
最新资源
- 多模态联合稀疏表示在视频目标跟踪中的应用
- Kubernetes资源管控与Gardener开源软件实践解析
- MPI集群监控与负载平衡策略
- 自动化PHP安全漏洞检测：静态代码分析与数据流方法
- 青苔数据CEO程永：技术生态与阿里云开放创新
- 制造业转型： HyperX引领企业上云策略
- 赵维五分享：航空工业电子采购上云实战与运维策略
- 单片机控制的LED点阵显示屏设计及其实现
- 驻云科技李俊涛：AI驱动的云上服务新趋势与挑战
- 6LoWPAN物联网边界路由器：设计与实现
- 猩便利工程师仲小玉：Terraform云资源管理最佳实践与团队协作
- 类差分度改进的互信息特征选择提升文本分类性能
- VERITAS与阿里云合作的混合云转型与数据保护方案
- 云制造中的生产线仿真模型设计与虚拟化研究
- 汪洋在PostgresChina2018分享：高可用 PostgreSQL 工具与架构设计
- 2018 PostgresChina大会：阿里云时空引擎Ganos在PostgreSQL中的创新应用与多模型存储
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
