ISSCC 2019 / February 18, 2019 / 8:45 AM
3.2.1 Dynamic Networks, Differentiable Programming: A relatively re
cent
concept in DL is the idea of dynamic networks. Regular DL systems use a
static network of parameterized modules. But, in increasingly many
applications, the network architecture is dynamic and changes for every new
data point. In effect, dynamic DL systems can be seen as the execution trace
of a program, with conditionals and loops that are input-dependent. DL
frameworks such as PyTorch record a “tape” of this execution trace, which
can be played backwards to back-propagate gradients through the program.
This method is known as “autograd”. The phrase “differentiable programming”
designates the process of writing a program with calls to parameterized
functions that automatically compute the gradient of the function’s
output
with respect to the parameters, allowing the function to be finalized through
learning. Dynamic networks are particularly useful in a variety of applications:
for natural-language processing, for data that does not come in the form of a
fixed-sized tensor, for systems that need to activate parts of a large network
on demand in a data-dependent way (such as the Multi-scale DenseNet
architecture [47] shown in Figure 1.1.4) and for “reasoning” networks whose
output is another network specifically designed to answer a particular question
[48, 49] (see Figure 1.1.5).
3.2.2 Neural Networks on Graphs: One of the most exciting recent
developments
in DL is neural networks on graphs [50]. Many problems
are difficult to represent with fixed-size tensors or variable-length
sequences of tensors, but are better represented by graphs whose arcs
and nodes are annotated by tensors. This suggests the use of networks
of differentiable modules whose inputs and outputs are annotated
graphs. The idea goes back to Graph Transformer Networks, built to
recognize character strings [18]. But recent incarnations of graph neural
networks have been applied to 3D meshes, social networks, gene-
regulation networks, and chemical molecules. Convolution operations
can easily be defined on irregular graphs: they are defined as diagonal
operators in the eigenspace of the graph Laplacian, which is a
generalization of the Fourier transform. We foresee an increase in the
usage of such networks for a wide variety of applications, which are
likely to violate the assumptions of current DL hardware.
3.2.3 Graph Embedding Networks: Increasingly, DL is used for large-
scale
embedding of knowledge bases. For example, using a large
knowledge graph composed of triplets (subject, relation, object), such as
(“Barak Obama”,
“was
born in”,
“Hawaii”)
one may train a network to rate
such triplets or to predict one of the elements from the other two. A special
case of this consists in learning a vector for each object and subject, such
as a simple scalar-valued operation between the vectors (distance) will
predict the presence or absence of a particular relation between the object
or subject. These methods, applied on a large scale, are particularly efficient
for recommender systems, and can use hyperbolic metric spaces to
represent hierarchical categories [51].
3.2.4 Memory-Augmented Networks: To endow DL systems with the
abil
ity to reason, they need a short-term memory, to be used as an
episodic memory, or a scratchpad/working memory. For example, if a
system is to answer questions about a series of events (described as a
text), it must be able to store the story in a memory and retrieve the
relevant bits to answer a particular question. This led to the memory-
network architecture [52, 53] in which a recurrent neural nework is
augmented by what amounts to a differentiable associative memory
circuit (see Figure
1.1.5)
. This associative memory can be quite large
and requires finding the nearest neighbors to a key vector very
efficiently. As DL systems are increasingly used for high-level cognitive
tasks, such memory modules will become commonplace and very large,
requiring hardware support.
3.2.5 Complex Inference and Search: Most of today’s DL systems
simply
produce an output given an input. But complex reasoning
requires that the output variable actually be an input to a scoring
network whose scalar output (akin to energy) indicates the
incompatibility between the input and an output proposal. An inference
procedure must search for the output value that minimizes the energy.
This type of model is called an energy-based model [54]. If the energy-
minimizing inference procedure is gradient-based, inference hardware
will need to support back-propagation.
3.2.6 Sparse Activations: As the size of DL systems grows, it is likely that
the modules’ activations will become increasingly sparse, with only a subset
of variables of a subset of modules being activated at any one time. This is
akin to how the brain represents information: on average, neurons
in the brain
are at 2% of their maximum activation, and most neurons are quiet
most of the time, which is good for power dissipation. Examples of
explicitly sparse networks already exist (for processing volumetric
imaging data [56]).
3.2.7 Overall:
New architectural concepts such as dynamic networks, graph data,
associative-memory structures, and inference-through-minimization
procedures are likely to affect the type of hardware architectures that will be
required in the future.
4 The Revolution will not be Supervised
With all the hype around the new AI and DL, the way machines learn today
is vastly less efficient than the way humans and animals learn. Almost all
practical applications of DL use supervised learning (SL), in which the
system is fed the desired output during training, with a tiny minority using
reinforcement learning (RL). Most humans are capable of learning to drive
a car in about 30 hours of training without ever causing accidents. In
contrast, current model-free RL methods would likely require millions of
hours of practice, with numerous accidents, for an autonomous car to learn
to drive. This is not a problem in easy-to-simulate fully-observable
environments with discrete state, such as the game of go or chess. But, it
does not work in the real world! Obviously, our current learning paradigms
are missing a key ingredient.
One hypothesis is that this missing ingredient is self-supervised learning.
The bulk of learning in humans and animals is self-supervised: we learn
enormous amounts of background knowledge about how the world works
by observation in the first days, weeks, and months of life. In particular,
we learn intuitive physics and the properties of the physical world. By the
age of 9 months, babies understand object permanence, stability, animate
vs inanimate objects, stability, gravity, inertia, and so on. The ability to
predict what is going to happen in the world is what allows us to learn to
drive without causing accidents: our world model allows us to anticipate
the consequences of our actions, to maintain the car on the road, and to
avoid disasters.
The idea of self-supervised learning is to train a machine to predict any
subset of its input from other subsets (with a possible overlap between
the subsets). For example, given a 6-frame video clip, one could train a
DL system to predict the last two frames from the first four.
Why should SSL be more efficient than either RL or SL? In RL, the system
produces an output (often an action or sequence of actions) and gets in
return a single scalar value representing the “reward” for this action.
Learning a complex task in this scenario requires a very large number of
trials, and a large number of errors. While the process works fine for fully-
observable games (such as chess and go) where millions of trials can be
generated through self-play, it is largely impractical in the real world. A
model-free RL system would require millions of hours of driving and
numerous crashes to train a car to drive itself. The number of trials required
is large because the feedback from the environment is information-poor.
In SL, the system is given the correct answer, generally in the form of a
target output vector. While this is less information-poor than in RL, it still
requires a lot of training samples to capture the essence of the problem.
On the other hand, SSL asks the machine to predict a large amount of
information in the form of a high-dimensional signal (such as a whole video
frame). More complex models with more parameters can be learned with
a given number of samples or trials. The main difficulty is that predicting
the future of a video is not achievable exactly because the world is not
entirely predictable. If one uses a least-square criterion to train a video
predictor, the resulting predictions are blurry frames: an average of all the
possible futures. To make sharp predictions, one must have a set of latent
variables that, when passed through a predictor, parameterize the set of
plausible predictions. One technique used to train such models is
Generative Adversarial Networks (GAN) [59], which for training uses two
networks simultaneously: a generator that makes predictions using
15DIGEST OF TECHNICAL PAPERS •
1
剩余737页未读，继续阅读
PiersonWong
- 粉丝: 1
- 资源: 3
我的内容管理
展开
最新资源
- 最优条件下三次B样条小波边缘检测算子研究
- 深入解析：wav文件格式结构
- JIRA系统配置指南：代理与SSL设置
- 入门必备：电阻电容识别全解析
- U盘制作启动盘：详细教程解决无光驱装系统难题
- Eclipse快捷键大全：提升开发效率的必备秘籍
- C++ Primer Plus中文版：深入学习C++编程必备
- Eclipse常用快捷键汇总与操作指南
- JavaScript作用域解析与面向对象基础
- 软通动力Java笔试题解析
- 自定义标签配置与使用指南
- Android Intent深度解析：组件通信与广播机制
- 增强MyEclipse代码提示功能设置教程
- x86下VMware环境中Openwrt编译与LuCI集成指南
- S3C2440A嵌入式终端电源管理系统设计探讨
- Intel DTCP-IP技术在数字家庭中的内容保护
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
