The wake-sleep algorithm for unsupervised
neural networks
Geoffrey E Hinton
Peter Dayan Brendan J Frey Radford M Neal
Department of Computer Science
University of Toronto
6 King’s College Road
Toronto M5S 1A4, Canada
3
rd
April 1995
Abstract
An unsupervised learning algorithm for a multilayer network of stochastic
neurons is described. Bottom-up “recognition” connections convert the input
into representations in successive hidden layers and top-down “generative”
connections reconstruct the representation inonelayer fromthe representation
in the layer above. In the “wake” phase, neurons are driven by recognition
connections, andgenerativeconnectionsareadaptedtoincreasetheprobability
that they would reconstruct the correctactivity vector in the layerbelow. In the
“sleep” phase, neurons are driven by generative connections and recognition
connections are adapted to increase the probability that they would produce
the correct activity vector in the layer above.
Supervised learning algorithms for multilayer neural networks face two problems:
Theyrequirea teacher to specify the desired output ofthe networkand theyrequire
some method of communicating error information to all of the connections. The
wake-sleep algorithm avoids both these problems. When there is no external
teaching signal to be matched, some other goal is required to force the hidden
units to extract underlying structure. In the wake-sleep algorithm the goal is to
learn representations that are economical to describe but allow the input to be
reconstructedaccurately. We can quantify this goal by imagining a communication
game in which each vector of raw sensory inputs is communicated to a receiver by
first sending its hidden representation and then sending the difference between the
input vector and its top-down reconstruction from the hidden representation. The
aim of learning is to minimize the “description length” which is the total number
of bits that would be required to communicate the input vectors in this way [1].
No communication actually takes place, but minimizing the description length
that would be required forces the networkto learn economical representations that
capture the underlying regularities in the data [2].
1
剩余11页未读，继续阅读
