![](https://csdnimg.cn/release/download_crawler_static/10269976/bg8.jpg)
At about the same time, in late 2009, deep learning feedforward networks made inroads into
speech recognition, as marked by the NIPS Workshop on Deep Learning for Speech Recognition.
Intensive collaborative work between Microsoft Research and University of Toronto researchers
demonstrated by mid-2010 in Redmond that deep neural networks interfaced with a hidden
Markov model with context-dependent states that define the neural network output layer can
drastically reduce errors in large-vocabulary speech recognition tasks such as voice search. The
same deep neural net model was shown to scale up to Switchboard tasks about one year later at
Microsoft Research Asia. Even earlier, in 2007, LSTM
[51]
trained by CTC
[53]
started to get
excellent results in certain applications.
[54]
This method is now widely used, for example, in
Google's greatly improved speech recognition for all smartphone users.
[55]
As of 2011, the state of the art in deep learning feedforward networks alternates convolutional
layers and max-pooling layers,
[93][94]
topped by several fully connected or sparsely connected
layer followed by a final classification layer. Training is usually done without any unsupervised
pre-training. Since 2011, GPU-based implementations
[93]
of this approach won many pattern
recognition contests, including the IJCNN 2011 Traffic Sign Recognition Competition,
[95]
the ISBI
2012 Segmentation of neuronal structures in EM stacks challenge,
[96]
the ImageNet
Competition,
[97]
and others.
Such supervised deep learning methods also were the first artificial pattern recognizers to achieve
human-competitive performance on certain tasks.
[98]
To overcome the barriers of weak AI represented by deep learning, it is necessary to go beyond
deep learning architectures, because biological brains use both shallow and deep circuits as
reported by brain anatomy
[99]
displaying a wide variety of invariance. Weng
[100]
argued that the
brain self-wires largely according to signal statistics and, therefore, a serial cascade cannot catch
all major statistical dependencies. ANNs were able to guarantee shift invariance to deal with small
and large natural objects in large cluttered scenes, only when invariance extended beyond shift, to
all ANN-learned concepts, such as location, type (object class label), scale, lighting. This was
realized in Developmental Networks (DNs)
[101]
whose embodiments are Where-What Networks,
WWN-1 (2008)
[102]
through WWN-7 (2013).
[103]
Deep neural network architectures
There are huge number of variants of deep architectures. Most of them are branched from some
original parent architectures. It is not always possible to compare the performance of multiple
architectures all together, because they are not all evaluated on the same data sets. Deep
learning is a fast-growing field, and new architectures, variants, or algorithms appear every few
Brief discussion of deep neural networks
A deep neural network (DNN) is an artificial neural network (ANN) with multiple hidden layers of
units between the input and output layers.
[3][5]
Similar to shallow ANNs, DNNs can model complex
non-linear relationships. DNN architectures, e.g., for object detection and parsing, generate
compositional models where the object is expressed as a layered composition of image
primitives.
[104]
The extra layers enable composition of features from lower layers, giving the
potential of modeling complex data with fewer units than a similarly performing shallow network.
[3]
DNNs are typically designed as feedforward networks, but research has very successfully applied
recurrent neural networks, especially LSTM,
[51][105]
for applications such as language modeling.
Deep learning - Wikipedia https://en.wikipedia.org/wiki/Deep_learning
第8页 共38页 2016/11/5 17:59