没有合适的资源?快使用搜索试试~ 我知道了~
首页世界模型:强化学习环境的生成神经网络探索
"世界模型(World Models)是2018年自然语言处理领域的一项重要研究成果,位列年度顶级论文前列,仅次于当时的BERT模型。这篇论文由David Ha和Jürgen Schmidhuber共同发表,深入探讨了如何构建能够生成并理解复杂环境的深度学习神经网络模型。世界模型的核心在于其能够通过无监督学习快速学习到对环境的压缩式空间和时间表示,这对于强化学习中的决策制定至关重要。 该模型的核心概念借鉴了人类基于感官输入构建世界观的原理。我们可以通过模拟人类大脑的方式,训练一个能够捕捉环境特征的世界模型。这个模型能够作为代理(agent)的输入,从而训练出一个小型且高效的策略,使其能够在模拟的环境中解决所需的任务。这种技术的一大创新之处在于,代理可以在其自身的“梦境”——由世界模型生成的虚拟环境中进行完全的训练,然后将学到的策略无缝地转移到实际环境中。 世界模型的训练过程无需大量标注数据,仅依赖于环境互动,这在节省人力和资源的同时,提高了模型的泛化能力。论文的作者还指出,这个模型的内在理念与系统动力学之父Jay Wright Forrester提出的概念相呼应,即我们在脑海中构建的世界模型是一种简化和抽象的版本,只包含关键的概念和关系,而非完整的现实细节。 此外,论文提供了一个交互式的在线资源,链接为<https://worldmodels.github.io>,读者可以通过该平台深入了解世界模型的工作原理和实现方法。世界模型的研究不仅推动了自然语言处理技术的发展,还在人工智能领域的建模和决策制定方面开辟了新的可能性,对于研究者和实践者来说,都是一篇值得收藏的论文。"
资源详情
资源推荐
![](https://csdnimg.cn/release/download_crawler_static/10840177/bg5.jpg)
World Models
In this experiment, the world model (V and M) has no knowl-
edge about the actual reward signals from the environment.
Its task is simply to compress and predict the sequence of
image frames observed. Only the Controller (C) Model
has access to the reward information from the environment.
Since there are a mere 867 parameters inside the linear con-
troller model, evolutionary algorithms such as CMA-ES are
well suited for this optimization task.
We can use the VAE to reconstruct each frame using
z
t
at
each time step to visualize the quality of the information the
agent actually sees during a rollout. The figure below is a
VAE model trained on screenshots from CarRacing-v0.
Figure 10.
Despite losing details during this lossy compression
process, latent vector z captures the essence of each image frame.
In the online version of this article, one can load randomly
chosen screenshots to be encoded into a small latent vector
z
, which is used to reconstruct the original screenshot. One
can also experiment with adjusting the values of the
z
vector
using the slider bars to see how it affects the reconstruction,
or randomize
z
to observe the space of possible screenshots.
3.2. Procedure
To summarize the Car Racing experiment, below are the
steps taken:
1. Collect 10,000 rollouts from a random policy.
2. Train VAE (V) to encode frames into z ∈ R
32
.
3. Train MDN-RNN (M) to model P (z
t+1
| a
t
, z
t
, h
t
).
4. Define Controller (C) as a
t
= W
c
[z
t
h
t
] + b
c
.
5.
Use CMA-ES to solve for a
W
c
and
b
c
that maximizes
the expected cumulative reward.
MODEL PARAMETER COUNT
VAE 4,348,547
MDN-RNN 422,368
CONTROLLER 867
3.3. Experiment Results
V Model Only
Training an agent to drive is not a difficult task if we have a
good representation of the observation. Previous works (Hn-
ermann, 2017; Bling, 2015; Lau, 2016) have shown that
with a good set of hand-engineered information about the
observation, such as LIDAR information, angles, positions
and velocities, one can easily train a small feed-forward
network to take this hand-engineered input and output a
satisfactory navigation policy. For this reason, we first want
to test our agent by handicapping C to only have access to V
but not M, so we define our controller as
a
t
= W
c
z
t
+ b
c
.
Figure 11.
Limiting our controller to see only
z
t
, but not
h
t
results
in wobbly and unstable driving behaviours.
Although the agent is still able to navigate the race track
in this setting, we notice it wobbles around and misses the
tracks on sharper corners. This handicapped agent achieved
an average score of 632
±
251 over 100 random trials,
in line with the performance of other agents on OpenAI
Gym’s leaderboard (Klimov, 2016) and traditional Deep RL
methods such as A3C (Khan & Elibol, 2016; Jang et al.,
2017). Adding a hidden layer to C’s policy network helps
to improve the results to 788
±
141, but not quite enough to
solve this environment.
Full World Model (V and M)
The representation
z
t
provided by our V model only cap-
tures a representation at a moment in time and does not have
much predictive power. In contrast, M is trained to do one
thing, and to do it really well, which is to predict
z
t+1
. Since
M’s prediction of
z
t+1
is produced from the RNN’s hidden
state h
t
at time t, this vector is a good candidate for the set
of learned features we can give to our agent. Combining
z
t
with
h
t
gives our controller C a good representation of both
the current observation, and what to expect in the future.
剩余20页未读,继续阅读
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://profile-avatar.csdnimg.cn/default.jpg!1)
jifeng632
- 粉丝: 0
- 资源: 1
上传资源 快速赚钱
我的内容管理 展开
我的资源 快来上传第一个资源
我的收益
登录查看自己的收益我的积分 登录查看自己的积分
我的C币 登录后查看C币余额
我的收藏
我的下载
下载帮助
![](https://csdnimg.cn/release/wenkucmsfe/public/img/voice.245cc511.png)
最新资源
- Vue实现iOS原生Picker组件:详细解析与实现思路
- Arduino蓝牙小车:参数调试与功能控制
- 百度Java面试精华:200页精选资源涵盖核心知识点
- Swift使用CoreData填坑指南:CoreData在Swift 3.0的变化
- 微距离无线充电器创新设计及其实验探索
- MTK Android平台开发全攻略:44步详解流程
- RecyclerView全面解析:替代ListView的新选择
- Android开发:自动适配中英文键盘解决方案
- Android调用WebService接口教程
- Android开发:BitmapUtil图片处理全解析与实例
- Android多线程断点续传实现详解
- PCA算法在人脸识别会议签到系统中的应用
- EventBus 3.0:Android事件总线详解与实战应用
- Android FileUtil:全面解析文件操作实用技巧与实例
- RecyclerView添加头部和尾部实战教程
- Android实现微博滑动固定顶部栏实战与优化
资源上传下载、课程学习等过程中有任何疑问或建议,欢迎提出宝贵意见哦~我们会及时处理!
点击此处反馈
![](https://img-home.csdnimg.cn/images/20220527035711.png)
![](https://img-home.csdnimg.cn/images/20220527035711.png)
![](https://img-home.csdnimg.cn/images/20220527035111.png)
安全验证
文档复制为VIP权益,开通VIP直接复制
![](https://csdnimg.cn/release/wenkucmsfe/public/img/green-success.6a4acb44.png)