深度脉冲神经网络的渐进串联学习方法

需积分: 0 18 浏览量更新于2024-06-26 收藏 1.51MB PDF 举报

"这篇论文探讨了在深度脉冲神经网络（SNNs）中使用进步级联学习方法进行模式识别的策略。作者包括Jibin Wu, Chenglin Xu, Xiao Han, Daquan Zhou, Malu Zhang, Haizhou Li (IEEE Fellow) 和 Kay Chen Tan (IEEE Fellow)。" 深度学习是现代人工智能领域的核心组成部分，它利用多层的神经网络模型来解决复杂问题，如图像识别、自然语言处理等。而脉冲神经网络（SNNs）作为一种生物灵感的计算模型，其工作原理更接近生物大脑，以事件驱动的方式运行，具有低延迟和高计算效率的优势。SNNs通过模拟神经元的脉冲（或称尖峰）通信，实现信息的传递和处理，这使得它们在能源效率和实时性方面优于传统的人工神经网络（ANNs）。然而，训练深度SNNs是一项挑战，因为其过程并不像训练ANNs那样直观。本文提出了一种新颖的ANN到SNN的转换框架，并结合逐层学习方法，称为“进步级联学习”。首先，研究者在离散表示空间中探讨了ANNs与SNNs之间的等价性，设计了一个原始的网络转换方法，该方法利用脉冲计数来近似ANN神经元的激活值。这种方法有助于SNNs保留ANNs的结构和功能，同时利用SNNs的稀疏通信特性。为了弥补这种原始转换可能导致的误差，论文进一步引入了一种逐层学习策略，结合自适应训练调度器对网络权重进行微调。进步级联学习方法通过逐步调整和优化，使SNN能够有效地学习和适应输入数据的模式，从而提高模式识别的准确性和效率。这一创新方法对于深度SNNs的训练提供了新的途径，有望在低功耗设备和实时应用中提升SNNs的表现。通过这种方式，SNNs可以更好地应用于物联网、自动驾驶、边缘计算等场景，这些场景对计算效率和能耗有严格要求。论文的贡献在于，不仅提出了一个有效的SNN训练框架，还促进了深度学习与生物神经科学的交叉融合，为未来的人工智能研究开辟了新的方向。

where r

ðtÞ denotes the ﬁring rate of neuron i at layer l and

max

denotes the maximum ﬁring rate that is determined by

the time step size. a

is the activation value of ANN neuron i

at the ﬁrst layer, V

ðtÞ is the membrane potential of the corre-

sponding spiking neuron, and # is the neuronal ﬁring thresh-

old. M

l1

is the total number of neurons in layer l  1 and b

is the bias term of ANN neuron i at layer l. Ideally, the ﬁring

rate of spiking neurons should be proportional to the activa-

tion value of their ANN counterparts as per the ﬁrst term of

Eq. (1). While the surplus membrane potential that has not

been discharged by the end of simulation will cause an

approximation error as shown by the second term of Eq. (1),

which can be counteracted with a large ﬁring threshold or a

large encoding time window. Since increasing the ﬁring

threshold will inevitably prolong the evidence accumulation

time, a proper ﬁring threshold that can prevent spiking neu-

rons from either under- or over-activating is usually pre-

ferred and the encoding time window is extended to

minimize such a ﬁring rate approximation error.

Besides, this approximation error accumulates gradually

while propagating over layers as shown in Eq. (2), thereby a

further extension of the encoding time window is required

to compensate. As such, a few thousand time steps are typi-

cally required to achieve a competitive accuracy for deep

SNNs with more than 10 layers [28], [29]. From these formu-

lations, it is clear that to approximate the continuous input-

output representation of ANNs with the ﬁring rate of spik-

ing neurons will inevitably lead to the accuracy and latency

trade-off. To overcome this issue, as will be introduced in

the following sections, we propose a novel conversion

method that is grounded on the discrete neural representa-

tion, whereby the spike count, upper bounded by the

encoding time window size, is taken to approximate the dis-

crete input-output representation of ANNs. To make efﬁ-

cient use of the spike count for information representation,

we propose a novel ﬁring threshold determination strategy

such that rapid and efﬁcient pattern recognition can be

achieved with SNNs. To counteract the conversion errors

and hence ensure high accuracies in pattern recognition

tasks, a layer-wise learning method is further proposed to

ﬁne-tune the network.

3RETHINKING ANN-TO-SNN CONVERSION

Over the years, many spiking neuron models are developed

to describe the rich dynamical behavior of biological neurons.

Most of them, however, are too complex for real-world pat-

tern recognition tasks. As discussed in Section 2, for computa-

tional simplicity and ease of conversion, the IF neuron model

is commonly used in ANN-to-SNN conversion works [26],

[27], [28]. Although this simpliﬁed spiking neuron model

does not emulate the rich sub-threshold dynamics of biologi-

cal neurons, it preserves attractive properties of discrete and

sparse communication, therefore, allows for efﬁcient hard-

ware implementation. In this section, we reinvestigate the

approximation of input-output representation between a

ReLU ANN neuron and an integrate-and-ﬁre spiking neuron.

3.1 Spiking Neuron Versus ANN Neuron

Let us consider a discrete-time simulation of spiking neu-

rons with an encoding time window of N

that determines

the inference speed of an SNN. At each time step t, the

incoming spikes to the neuron i at layer l are transduced

into synaptic current z

½t according to

½t¼

l1

½tþb

; (3)

where s

l1

½t indicates the occurrence of an input spike at

time step t, and w

l1

is the synaptic weight between the pre-

synaptic neuron j and the post-synaptic neuron i at layer l.

can be interpreted as a constant injecting current.

The synaptic current z

½t is further integrated into the

membrane potential V

½t as per Eq. (4). Without loss of gen-

erality, a unitary membrane resistance is assumed in this

work. The membrane potential is reset by subtracting the

ﬁring threshold after each ﬁring as described by the last

term of Eq. (4).

½t¼V

½t  1þz

½t#

½t  1: (4)

An output spike is generated whenever the V

½t rises

above the ﬁring threshold #

(determined layer-wise) a s

follows

½t¼QðV

½t#

Þ with QðxÞ¼

1;ifx 0

0; otherwise



(5)

The spike train s

and spike count c

for a time window of

can thus be determined and represented as follows

¼fs

½1; ...;s

½N

g

t¼1

½t:

(6)

For non-spiking ANN neurons, let us describe the neuro-

nal function of neuron i at layer l as

¼ f

l1

þ b

; (7)

which has w

l1

and b

as the weight and bias. x

l1

and a

denote the input and output of the ANN neuron. fðÞ

denotes the activation function, which we use the ReLU in

this work. For ANN-to-SNN conversion, an ANN with the

ReLU neurons is ﬁrst trained, that is called pre-training,

before the conversion.

3.2 Neural Discretization Versus Activation

Quantization

In the conventional ANN-to-SNN conversion studies, the

ﬁring rate of spiking neurons is usually taken to approxi-

mate the continuous input-output representation of the pre-

trained ANN. As discussed in Section 2, a spiking neuron

takes a notoriously long time window to reliably approxi-

mate a continuous value. Recent studies, however, suggest

such a continuous neural representation may not be neces-

sary for ANNs [34]. In fact, there could be little impact on

the network performance when the activation value of

ANN neurons are properly quantized to a low-precision

WU ET AL.: PROGRESSIVE TANDEM LEARNING FOR PATTERN RECOGNITION WITH DEEP SPIKING NEURAL NETWORKS 7827

剩余16页未读，继续阅读

KafCoppelia

粉丝: 196
资源: 24

深度脉冲神经网络的渐进串联学习方法

Progress in Artificial Intelligence and Pattern Recognition

Progressive Mimic Learning：Train Lightweight CNN Models

Progressive visual secret sharing for general access structure with multiple decryptions

PROGRESSIVE_WEB_APPS_WITH_REACT

A modified progressive damage model for simulating low-velocity

Progressive-Web-Apps-with-React:Packt发布的带有React的渐进式Web应用程序

Designing with Progressive Enhancement: Building the Web that Works for Everyone

Progressive refinement for robust image registration

Binned Progressive Quantization for Compressive Sensing

Progressive Web Apps with React

最新资源