Algorithm 1 Online SBEED learning with experience replay
1: Initialize w = (w
V
, w
π
, w
ρ
) and π
b
randomly, set .
2: for episode i = 1, . . . , T do
3: for size k = 1, . . . , K do
4: Add new transition (s, a, r, s
0
) into D by executing behavior policy π
b
.
5: end for
6: for iteration j = 1, . . . , N do
7: Update w
j
ρ
by solving
min
w
ρ
b
E
{s,a,s
0
}∼D
h
(δ(s, a, s
0
) − ρ(s, a))
2
i
.
8: Decay the stepsize ζ
j
in rate O(1/j).
9: Compute the stochastic gradients w.r.t. w
V
and w
π
as
b
∇
w
V
¯
`(V, π) and
b
∇
w
π
¯
`(V, π).
10: Update the parameters of primal function by solving the prox-mappings, i.e.,
update V : w
j
V
= P
w
j−1
V
(ζ
j
b
∇
w
V
¯
`(V, π))
update π: w
j
π
= P
w
j−1
π
(ζ
j
b
∇
w
π
¯
`(V, π))
11: end for
12: Update behavior policy π
b
= π
N
.
13: end for
With gradients given above, we may apply stochastic mirror descent to update
w
V
and
w
π
; that is, given a
stochastic gradient direction (for either w
V
or w
π
), we solve the following prox-mapping in each iteration,
P
z
V
(g) = argmin
w
V
hw
V
, gi+ D
V
(w
V
, z
V
),
P
z
π
(g) = argmin
w
π
hw
π
, gi+ D
π
(w
π
, z
π
).
where
z
V
and
z
π
can be viewed the current weight, and
D
V
(
w, z
) and
D
π
(
w, z
) are Bregman divergences.
We can use Euclidean metric for both
w
V
and
w
π
, and possibly KL-divergence for
w
π
. The per-iteration
computation complexity is therefore very low, and the algorithm can be scaled up to complex nonlinear
approximations.
Algorithm 1 instantiates SBEED, combined with experience replay (Lin, 1992) for greater data efficiency,
in an online RL setting. New samples are added to the experience replay buffer
D
at the beginning of each
episode (Lines 3–5) with a behavior policy. Lines 6–11 correspond to the stochastic mirror descent updates
on the primal parameters. Line 12 sets the behavior policy to be the current policy estimate, although other
choices may be used. For example,
π
b
can be a fixed policy (Antos et al., 2008), which is the case we will
analyze in the next section.
Remark (Role of dual variables): The dual variable is obtained by solving
min
ρ
E
s,a,s
0
h
R(s, a) + γV (s
0
) − λ log π(a|s) − ρ(s, a)
2
i
.
The solution to this optimization problem is
ρ
∗
(s, a) = R(s, a) + γE
s
0
|s,a
[V (s
0
)] − λ log π(a|s) .
Therefore, the dual variables try to approximate the one-step smoothed Bellman backup values, given a (
V, π
)
pair. Similarly, in the equivalent (8), the optimal dual variable
ν
(
s, a
) is to fit the one-step smoothed Bellman
error. Therefore, each iteration of SBEED could be understood as first fitting a parametric model to the
one-step Bellman backups (or equivalently, the one-step Bellman error), and then applying stochastic mirror
descent to adjust V and π.
6
剩余27页未读，继续阅读
weixin_40191861_zj
- 粉丝: 77
- 资源: 1万+
我的内容管理
展开
最新资源
- 中国微型数字传声器：技术革新与市场前景
- 智能安防：基于Hi3515的嵌入式云台控制系统设计
- 手机电量低时辐射真增千倍？解析手机使用谣言
- 56F803型DSP驱动的高精度大功率超声波电源控制策略研究
- ARM与GPRS结合的远程监测系统设计
- GPS与RFID技术结合的智能巡检系统设计
- CPLD驱动的低功耗爆炸场温度测试系统设计
- 基于FPGA的智能驱动控制系统：可扩展设计与工业网络协议
- 基于ATmega128和CH374的嵌入式USB接口设计
- 基于AT89C52的温度补偿超声波测距仪：高精度设计与应用
- MSP430F448单片机在交流数字电压表中的应用
- 提升变频器应用效率的12项实用技巧
- STM32F103在数字电镀电源并联均流系统中的应用
- PSpice仿真下的升压开关电源设计：拓扑分析与CCM稳定性提升
- 轻巧高效：MSP430主导的低成本无线传感器网络节点设计
- FPGA在EDA/PLD中实现LVDS接口的应用解析
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
