深度脉冲神经网络的渐进串联学习方法
需积分: 0 41 浏览量
更新于2024-06-26
收藏 1.51MB PDF 举报
"这篇论文探讨了在深度脉冲神经网络（SNNs）中使用进步级联学习方法进行模式识别的策略。作者包括Jibin Wu, Chenglin Xu, Xiao Han, Daquan Zhou, Malu Zhang, Haizhou Li (IEEE Fellow) 和 Kay Chen Tan (IEEE Fellow)。"
深度学习是现代人工智能领域的核心组成部分，它利用多层的神经网络模型来解决复杂问题，如图像识别、自然语言处理等。而脉冲神经网络（SNNs）作为一种生物灵感的计算模型，其工作原理更接近生物大脑，以事件驱动的方式运行，具有低延迟和高计算效率的优势。SNNs通过模拟神经元的脉冲（或称尖峰）通信，实现信息的传递和处理，这使得它们在能源效率和实时性方面优于传统的人工神经网络（ANNs）。
然而，训练深度SNNs是一项挑战，因为其过程并不像训练ANNs那样直观。本文提出了一种新颖的ANN到SNN的转换框架，并结合逐层学习方法，称为“进步级联学习”。首先，研究者在离散表示空间中探讨了ANNs与SNNs之间的等价性，设计了一个原始的网络转换方法，该方法利用脉冲计数来近似ANN神经元的激活值。这种方法有助于SNNs保留ANNs的结构和功能，同时利用SNNs的稀疏通信特性。
为了弥补这种原始转换可能导致的误差，论文进一步引入了一种逐层学习策略，结合自适应训练调度器对网络权重进行微调。进步级联学习方法通过逐步调整和优化，使SNN能够有效地学习和适应输入数据的模式，从而提高模式识别的准确性和效率。
这一创新方法对于深度SNNs的训练提供了新的途径，有望在低功耗设备和实时应用中提升SNNs的表现。通过这种方式，SNNs可以更好地应用于物联网、自动驾驶、边缘计算等场景，这些场景对计算效率和能耗有严格要求。论文的贡献在于，不仅提出了一个有效的SNN训练框架，还促进了深度学习与生物神经科学的交叉融合，为未来的人工智能研究开辟了新的方向。
where r
l
i
ðtÞ denotes the firing rate of neuron i at layer l and
r
max
denotes the maximum firing rate that is determined by
the time step size. a
1
i
is the activation value of ANN neuron i
at the first layer, V
1
i
ðtÞ is the membrane potential of the corre-
sponding spiking neuron, and # is the neuronal firing thresh-
old. M
l1
is the total number of neurons in layer l 1 and b
l
i
is the bias term of ANN neuron i at layer l. Ideally, the firing
rate of spiking neurons should be proportional to the activa-
tion value of their ANN counterparts as per the first term of
Eq. (1). While the surplus membrane potential that has not
been discharged by the end of simulation will cause an
approximation error as shown by the second term of Eq. (1),
which can be counteracted with a large firing threshold or a
large encoding time window. Since increasing the firing
threshold will inevitably prolong the evidence accumulation
time, a proper firing threshold that can prevent spiking neu-
rons from either under- or over-activating is usually pre-
ferred and the encoding time window is extended to
minimize such a firing rate approximation error.
Besides, this approximation error accumulates gradually
while propagating over layers as shown in Eq. (2), thereby a
further extension of the encoding time window is required
to compensate. As such, a few thousand time steps are typi-
cally required to achieve a competitive accuracy for deep
SNNs with more than 10 layers [28], [29]. From these formu-
lations, it is clear that to approximate the continuous input-
output representation of ANNs with the firing rate of spik-
ing neurons will inevitably lead to the accuracy and latency
trade-off. To overcome this issue, as will be introduced in
the following sections, we propose a novel conversion
method that is grounded on the discrete neural representa-
tion, whereby the spike count, upper bounded by the
encoding time window size, is taken to approximate the dis-
crete input-output representation of ANNs. To make effi-
cient use of the spike count for information representation,
we propose a novel firing threshold determination strategy
such that rapid and efficient pattern recognition can be
achieved with SNNs. To counteract the conversion errors
and hence ensure high accuracies in pattern recognition
tasks, a layer-wise learning method is further proposed to
fine-tune the network.
3RETHINKING ANN-TO-SNN CONVERSION
Over the years, many spiking neuron models are developed
to describe the rich dynamical behavior of biological neurons.
Most of them, however, are too complex for real-world pat-
tern recognition tasks. As discussed in Section 2, for computa-
tional simplicity and ease of conversion, the IF neuron model
is commonly used in ANN-to-SNN conversion works [26],
[27], [28]. Although this simplified spiking neuron model
does not emulate the rich sub-threshold dynamics of biologi-
cal neurons, it preserves attractive properties of discrete and
sparse communication, therefore, allows for efficient hard-
ware implementation. In this section, we reinvestigate the
approximation of input-output representation between a
ReLU ANN neuron and an integrate-and-fire spiking neuron.
3.1 Spiking Neuron Versus ANN Neuron
Let us consider a discrete-time simulation of spiking neu-
rons with an encoding time window of N
s
that determines
the inference speed of an SNN. At each time step t, the
incoming spikes to the neuron i at layer l are transduced
into synaptic current z
l
i
½t according to
z
l
i
½t¼
X
j
w
l1
ij
s
l1
j
½tþb
l
i
; (3)
where s
l1
j
½t indicates the occurrence of an input spike at
time step t, and w
l1
ij
is the synaptic weight between the pre-
synaptic neuron j and the post-synaptic neuron i at layer l.
b
l
i
can be interpreted as a constant injecting current.
The synaptic current z
l
i
½t is further integrated into the
membrane potential V
l
i
½t as per Eq. (4). Without loss of gen-
erality, a unitary membrane resistance is assumed in this
work. The membrane potential is reset by subtracting the
firing threshold after each firing as described by the last
term of Eq. (4).
V
l
i
½t¼V
l
i
½t 1þz
l
i
½t#
l
s
l
i
½t 1: (4)
An output spike is generated whenever the V
l
i
½t rises
above the firing threshold #
l
(determined layer-wise) a s
follows
s
l
i
½t¼QðV
l
i
½t#
l
Þ with QðxÞ¼
1;ifx 0
0; otherwise
:
(5)
The spike train s
l
i
and spike count c
l
i
for a time window of
N
s
can thus be determined and represented as follows
s
l
i
¼fs
l
i
½1; ...;s
l
i
½N
s
g
c
l
i
¼
X
N
s
t¼1
s
l
i
½t:
(6)
For non-spiking ANN neurons, let us describe the neuro-
nal function of neuron i at layer l as
a
l
i
¼ f
X
j
w
l1
ij
x
l1
j
þ b
l
i
!
; (7)
which has w
l1
ij
and b
l
i
as the weight and bias. x
l1
j
and a
l
i
denote the input and output of the ANN neuron. fðÞ
denotes the activation function, which we use the ReLU in
this work. For ANN-to-SNN conversion, an ANN with the
ReLU neurons is first trained, that is called pre-training,
before the conversion.
3.2 Neural Discretization Versus Activation
Quantization
In the conventional ANN-to-SNN conversion studies, the
firing rate of spiking neurons is usually taken to approxi-
mate the continuous input-output representation of the pre-
trained ANN. As discussed in Section 2, a spiking neuron
takes a notoriously long time window to reliably approxi-
mate a continuous value. Recent studies, however, suggest
such a continuous neural representation may not be neces-
sary for ANNs [34]. In fact, there could be little impact on
the network performance when the activation value of
ANN neurons are properly quantized to a low-precision
WU ET AL.: PROGRESSIVE TANDEM LEARNING FOR PATTERN RECOGNITION WITH DEEP SPIKING NEURAL NETWORKS 7827
剩余16页未读，继续阅读
2021-09-28 上传
2018-12-11 上传
2018-01-26 上传
2024-07-30 上传
2024-07-31 上传
2023-03-16 上传
2023-09-19 上传
2024-10-31 上传
2023-04-15 上传
KafCoppelia
- 粉丝: 197
- 资源: 24
我的内容管理
展开
