电池受限设备上的能源高效神经网络设计详解

需积分: 10 133 浏览量更新于2024-07-17 2 收藏 8.32MB PDF 举报

《嵌入式深度学习》是一本专著，由Bert Moons、Daniel Bankman和Marian Verhelst合著，重点关注算法和硬件实现技术，旨在为在嵌入式设备上实现高效的深度学习提供全面的解决方案。本书涵盖了电池限制的可穿戴设备上能源效率高的神经网络设计，从应用层到电路层的全方位优化策略。作者首先讨论了如何在应用层面进行设计，以适应始终在线的神经网络处理需求，确保算法在资源有限的设备上运行得既快速又节能。他们强调了在设计过程中，如何通过协同优化，从算法层面（如卷积神经网络的优化，利用并行处理、数据重用、稀疏操作和低精度计算来提升性能）到计算机架构层面，再到具体的电路实现，以减少计算成本。书中详细阐述了如何设计高效能的CNN处理器，特别是在硬件层面，通过精心设计，能够在处理深度学习任务时充分利用硬件资源，同时兼顾能效。四个实际的硅片原型被用来支持理论和设计概念，它们的物理实现以及所达到的性能指标被深入剖析，以此生动地展示了跨层设计思想如何在实践中发挥作用。该书不仅提供了理论指导，还提供了实践案例，使得读者能够理解和掌握如何在嵌入式系统中成功部署深度学习算法，这对于那些寻求在物联网、自动驾驶汽车、智能家居等领域的设备中实现智能功能的工程师来说，具有很高的实用价值。《嵌入式深度学习：算法、架构与电路的始终在线神经网络处理》是一本结合理论与实战的宝贵参考资料，适合对嵌入式系统和深度学习有深入研究或开发需求的专业人士阅读。版权方面，该书受到Springer Nature Switzerland AG的保护，所有权利保留，包括翻译、再版、再使用、朗诵、广播、微缩胶片复制、任何形式的物理传输或信息存储与检索，以及电子适应和计算机软件使用等。

1.2 Machine Learning 3

performance measure P, if its performance at tasks in T, as measured by P, improves

with experience E.” The following overview is based on Goodfellow et al. (2016),

which is an excellent introduction to the ﬁeld.

1.2.1 Tasks, T

Several tasks T are targeted in machine learning applications. Machine learning

is used for regression analyses (Huang et al. 2012), transcription (Shipp et al.

2002), machine translation (Cho et al. 2014; Bahdanau et al. 2014), anomaly

detection (Chandola et al. 2009;Erfanietal.2016), denoising (Vincent et al. 2010),

and synthesis techniques (Ze et al. 2013; Rokach et al. 2012). Most important in

the context of this dissertation is the classiﬁcation (Glorot et al. 2011; LeCun et al.

2015)task.

In this task, the machine is asked to specify which of k categories some input

belongs to. An example of a classiﬁcation task is object recognition, where the input

is an input image and the output is a numeric code identifying the object in the

image. Another example is speech recognition, where an input audio waveform is

translated into a sequence of classiﬁed phonemes or words. Most modern object

classiﬁcation is best accomplished with deep learning algorithms. Deep learning-

based classiﬁcation applications are hence the main scope of this work.

1.2.2 Performance Measures, P

The performance measure P is a quantitative measure of a machine learning

algorithm’s performance.

P differs depending on the task at hand. For detection, a two-class classiﬁcation

problem, measures for P are precision (true returned positives divided by all

returned positives) and recall (true returned positives divided by all positives). More

generally, in classiﬁcation and transcription tasks, P is the accuracy or error rate.

Here, accuracy is the proportion of correctly classiﬁed examples.

In order to estimate the performance of the machine learning algorithm on data

that it has not seen before, as would happen in the real world, it is crucial to verify

P on a test set of data that is separate from the data used for training the system.

1.2.3 Experience, E

A machine learning algorithm can increase its performance P on a task T by gaining

more experience E. Algorithms can be categorized as supervised and unsupervised

depending on the type of E they have access to during their learning process.

1.3 Deep Learning 5

a pattern recognition or machine learning system required careful engineering and

considerable domain expertise to design a feature extractor that could transform

raw data (such as pixel values of an image) into a suitable internal representation or

feature vector from which a learning subsystem, typically a classiﬁer, could detect

or classify patterns in the input. Examples of such features in the computer vision

context can be edges, gradients, speciﬁc colors, or more automated forms such as

SURF (Bay et al. 2006) or HOG (Dalal and Triggs 2005).

The concept of deep learning breaks with this tradition of using hand-designed

features, as it is a type of representation learning (LeCun et al. 2015). Representation

learning is a set of methods that allow a machine to be fed with raw data and to

automatically discover the representations needed for detection or classiﬁcation.

This is illustrated in Fig. 1.2b, where a raw input image is fed to a deep learning

system. In deep learning algorithms, multiple levels of representation are used. They

are a multi-layered system composed of nonlinear modules each transforming an

input representation (starting with the input images, for example) into an output

representation at a higher, more abstract level. If many such layers, or many such

transformations, are combined, very complex functions can be learned.

The difference between classical approaches to machine learning and represen-

tational deep learning is illustrated in Fig. 1.2. The key aspect of deep learning is

that these layers of features are not designed by hand by human engineers: they

are learned from data using a general purpose learning procedure, typically a form

of stochastic gradient descent (see Sect. 1.3.4). This strategy has been proven to be

very powerful and crucial to many of the recent breakthroughs in Machine Learning

and pattern recognition, as shown in Fig. 1.1.

Instead of hand-designed by a human expert, the intermediate features generated

in representational deep neural network are a mathematical optimum, within the

constraints of the optimization method (see Sect. 1.3.4). These features can be

visualized through the powerful tools from Olah et al. (2017), which were used

to generate Fig. 1.3. Figure 1.3 shows how GoogleNet (Szegedy et al. 2015) builds

up its understanding of images over many layers. The ﬁrst layers extract low-level

Fig. 1.3 Visualizations of features learned by GoogleNet (Szegedy et al. 2015)onIMA-

GENET (Russakovsky et al. 2015), taken from Olah et al. (2017). Learned features vary from

simple (a) in the ﬁrst layers to more complex and abstract (e) in the ﬁnal layers

6 1 Embedded Deep Neural Networks

features such as edges and contrast changes, as shown in Fig. 1.3a. Deeper layers

use that information to extract more complex features, such as textures (Fig. 1.3b),

patterns (Fig. 1.3c), parts (Fig. 1.3d), and ultimately objects (Fig. 1.3e).

The rest of this section introduces and discusses three types of typical neural

network architectures that are used in deep learning to perform this automated

feature extraction. Basic deep feed-forward neural networks are discussed in

Sect. 1.3.1. Convolutional neural networks, their optimized form useful for mod-

eling spatial correlations and leveraging sparse interactions, parameter sharing, and

equivariant representations, are discussed in Sect. 1.3.2. Recurrent neural networks,

used to model sequential correlations, are discussed in Sect. 1.3.3, although they are

somewhat out of scope of this text. Finally, Sect. 1.3.4 dives into some details on

how to efﬁciently train and regularize large-scale neural networks.

1.3.1 Deep Feed-Forward Neural Networks

Deep feed-forward networks are also referred to as multi-layer perceptrons, dense or

fully connected networks (FCN), or artiﬁcial neural networks (ANN). They are the

quintessential deep learning models, which have been around since the 1940s and

1960s. A deep feed-forward neural network has no internal feedback connections,

hence its name. A network with a depth of n layers would compute the following

function: f(x) = f

(n)

(...f

(2)

(1)

(x))),wheref

(1)

is the networks input layer,

2...n−1

are called hidden layers, and f

(n)

is the networks output layer. During

training, f(x)is trained to come as close as possible to f

∗

(x), which is the evaluated

ground truth for a given problem. These networks are called neural,astheyare

loosely inspired by neuroscience.

Figure 1.4 is an illustration of a three layer deep feed-forward neural network. It

has one input and output layer and a single hidden layer. The dimension of each of

Fig. 1.4 Graphical representation of a simple deep feed-forward neural network

1.3 Deep Learning 7

the layers is referred to as the width of the layer. Here each layer has a width of 3.

Each of the units, or neurons, in such a layer performs a dot product or vector-to-

scalar function on an input feature vector, generating a single output feature. A full

layer contains a vector of neurons and hence performs a vector-to-vector function on

an input feature vector x. In order for such networks to model nonlinear functions f ,

a nonlinear activation function has to be added to the neuron functionality. Typically

used options are the sigmoid function a(z) = σ(z), the hyperbolic tangent function

a(z) = tanh(z), or various forms of rectiﬁed linear units (ReLU) a(z) = max(0,z).

The functionality of a single neuron can hence be described as in the following

equation:

O = a





m=0

W [m]×x[m]+B



(1.1)

where a can be any of the activation functions listed above. From an algebraic point

of view, a single fully connected layer (FCL) can hence be seen as a vector–matrix

(input vector x, weight matrix W) product with a per-neuron additive scalar bias B

after which an element-wise activation function a is applied. This means FCNs are

already supported through heavily optimized BLAS and GEMM libraries for x86

and RISC CPUs.

According to the universal approximation theorem (Hornik et al. 1989)the

feedforward network above can be proven to be able to approximate any continuous

function on a closed and bounded subset of R

. This is the case only if the

network contains at least one hidden layer with enough hidden units and if it uses

a “squashing” activation function, such as the σ and ReLU functions. Hence, any

function we are trying to learn can be represented by a deep feed-forward neural

network of sufﬁcient size. However, this does not mean we’ll also be able to train

this network to map this function. The training algorithm might get stuck in a local

minimum or it could converge to the wrong function when it is overﬁtting. On top of

that, deeper networks suffer from vanishing gradients (Hochreiter et al. 2001)when

plain topologies and optimization techniques are used. This means the gradients for

the weights in the ﬁrst layers become too small for them to be signiﬁcantly updated.

In general, the giant networks that might be necessary to learn complex nonlinear

functions are infeasible to train with current technologies. Details on regularization

to prevent overﬁtting and on optimization algorithms are given in Sect. 1.3.4.

1.3.2 Convolutional Neural Networks

Albeit a universal function approximator, deep feed-forward neural networks do

suffer from several ﬂaws, mainly linked to the difﬁculty to train them. Convolutional

neural networks (CNN) are a specialized form of deep feed-forward neural networks

that try solving these ﬂaws. First, on high-dimensional input data such as images,

剩余215页未读，继续阅读

wang1062807258

粉丝: 13
资源: 272

电池受限设备上的能源高效神经网络设计详解

Deep Learning with Keras

Deep Learning（中文版）

embedded deep learning电子书下载

Embedded low-power deep learning with TIDL.pdf-综合文档

Deep Learning with Hadoop.pdf

Deep Learning in Neural Networks: An Overview

Deep-Learning-Control-with-Embedded-Physical-Structure

TensorFlow 1.x Deep Learning Cookbook-PacktPublishing(2017).pdf

TensorFlow+1.x+Deep+Learning+Cookbook-Packt+Publishing(2017).epub

Embedded-ai.bi-weekly：嵌入式AI公众号：NeuralTalk，每周的报告和很棒的Embedded-ai列表

最新资源