统计信号处理新进展：从系统到大脑

需积分: 10 30 浏览量更新于2024-08-02 1 收藏 5.86MB PDF 举报

"Haykin S.&Co(eds) New Directions in Statistical Signal Processing- From Systems to Brain" 本书《New Directions in Statistical Signal Processing: From Systems to Brain》是神经信息处理系列的一部分，由Simon Haykin、José C. Príncipe、Terrence J. Sejnowski和John McWhirter共同编辑，旨在推动信号处理与神经计算两个领域的交叉互动。书中通过14个章节深入探讨了这两个领域的新研究，这些章节几乎平分在信号处理和神经计算之间，从大脑出发，涵盖了通信、信号处理和学习系统等多个主题。在信号处理方面，书中的内容包括了对大脑信息处理的计算模型如何帮助我们理解大脑的信息处理方式，以及如何利用“主动听觉”来解决在嘈杂环境中智能机器的“鸡尾酒会问题”。此外，还涉及到了几何方法在盲信号处理中的应用，这是一种新的非视觉信号处理方法，有助于在不完全信息的情况下恢复或识别信号。在神经计算方面，作者们讨论了网络通信中的不确定性，以及图形和网络结构建模的方法。这些方法有助于理解和模拟复杂系统中的相互作用，特别是在生物神经系统中。书中的章节还涵盖了博弈论学习算法，这是一类借鉴了博弈论思想的机器学习算法，可以应用于多代理系统中，使得机器能够自我适应并优化其决策策略。此外，书中还介绍了可观察操作模型（OOMs）作为隐藏马尔科夫模型（HMMs）的替代方案。OOMs提供了一种更直接的方式来描述系统的动态行为，这对于序列数据分析和预测具有重要意义。整体来看，《New Directions in Statistical Signal Processing: From Systems to Brain》是一部集理论与实践于一体的著作，它不仅展示了信号处理技术在神经科学中的应用，也探讨了大脑工作原理如何启发和改进信号处理算法。这本书对于那些在信号处理、神经科学、机器学习和人工智能领域的研究人员和学生来说，是一份极有价值的参考资料，有助于推动这两个领域的交叉研究和发展。

1.2 Sensory Coding 5

showed that when the mapping from inputs to outputs is continuous, nonlinear,

and invertible, maximizing the mutual information between inputs and outputs is

equivalent to simply maximizing the entropy of the output signal. The algorithm

therefore performs a form of ICA.

Infomax-based ICA has also been used to model receptive ﬁelds in visual cortex.

When applied to natural images, in contrast to principal component analysis (PCA),

Infomax-based ICA develops oriented receptive ﬁelds at a variety of spatial scales

that are sparse, spatially localized, and reminiscent of oriented receptive ﬁelds in

primary visual cortex (Bell and Sejnowski, 1997). Another variant of nonlinear

Infomax developed by Okajima and colleagues (Okajima, 2004) has also been

applied to modeling higher levels of visual processing, including combined binocular

disparity and spatial frequency analysis.

1.2.2 Barlow’s Redundancy Reduction Principle

The principle of preserving information may be a good description of the very

earliest stages of sensory coding, but it is unlikely that this one principle will

capture all levels of processing in the brain. Clearly, one can trivially preserve

all the information in the input simply by copying the input to the next level up.

Thus, the idea only makes sense in the context of additional processing constraints.

Implicit in Linsker’s work was the constraint of dimension reduction. However, in

the neocortex, there is no evidence of a progressive reduction in the number of

neurons at successively higher levels of processing.

Barlow proposed a slightly diﬀerent principle of self-organization based on

the idea of producing a minimally redundant code. The information about anredundancy re-

duction principle underlying signal of interest (such as the visual form or the sound of a predator)

may be distributed across many input channels. This makes it diﬃcult to associate

particular stimulus values with distinct responses. Moreover, there is a high degree

of redundancy across diﬀerent channels. Thus, a neural code having minimal

redundancy should make it easier to associate diﬀerent stimulus values with diﬀerent

responses.

The formal, information-theoretic deﬁnition of redundancy is the information

content of the stimulus, less the capacity of the channel used to convey the

information. Unfortunately, quantities dependent upon calculation of entropy are

diﬃcult to compute. Thus, several diﬀerent formulations of Barlow’s principle have

been proposed, under varying assumptions and approximations. One simple way for

a learning algorithm to lower redundancy is reduce correlations among the outputs

(Barlow and F¨oldi´ak, 1989). This can remove second-order but not higher-order

dependencies.

Atick and Redlich proposed minimizing the following measure of redundancy

(Atick and Redlich, 1990):

R =1−

y;s

out

(y)

6 Modeling the Mind: From Circuits to Systems

Input channel

x=s+v1

Recoding

Ax+v2

Figure 1.2 Atick and Redlich’s learning principle was to minimize redundancy in

the output,

y, while preserving information about the input, x.

subject to the constraint of zero information loss (ﬁxed I

y;s

). C

out

(y), the output

channel capacity, is deﬁned to be the maximum of I

y;s

.Thechannelcapacityisatchannel capacity

a maximum when the covariance matrix of the output elements is diagonal, hence

Atick and Redlich used

out

(y)=







Thus, under this formulation, minimizing redundancy amounts to minimizing the

the channel capacity. This model is depicted in ﬁg 1.2. This model was used to

simulate retinal receptive ﬁelds. Under conditions of high noise (low redundancy),

the receptive ﬁelds that emerged were Gaussian-shaped spatial smoothing ﬁlters,

while at low noise levels (high redundancy) on-center oﬀ-surround receptive ﬁelds

resembling second spatial derivative ﬁlters emerged. In fact, cells in the mammalian

retina and lateral geniculate nucleus of the thalamus dynamically adjust their

ﬁltering characteristics as light levels ﬂuctuate between these two extremes under

conditions of low versus high contrast (Shapley and Victor, 1979; Virsu et al., 1977).

Moreover, this strategy of adaptive rescaling of neural responses has been shown to

be optimal with respect to information transmission (Brenner et al., 2000).

Similar learning principles have been applied by Atick and colleagues to model

higher stages of visual processing. Dong and Atick modeled redundancy reduction

across time, in a model of visual neurons in the lateral geniculate nucleus of the

thalamus (Dong and Atick, 1995). In their model, neurons with both lagged and

nonlagged spatiotemporal smoothing ﬁlters emerged. These receptive ﬁelds would

be useful for conveying information about stimulus onsets and oﬀsets. Li and Atick

(1994) modeled redundancy reduction across binocular visual inputs. Their model

generated binocular, oriented receptive ﬁelds at a variety of spatial scales, similar

to those seen in primary visual cortex. Bell and Sejnowski’s Infomax-based ICA

algorithm (Bell and Sejnowski, 1995) is also closely related to Barlow’s minimal

redundancy principal, since the ICA model is restricted to invertible mappings; in

1.2 Sensory Coding 7

Maximize

l(a;b)

(

s + n

)

(

s + n

)

Figure 1.3 Becker and Hinton’s Imax learning principle maximizes the mutual

information between features

a and b extracted from diﬀerent input channels.

this case, the maximization of mutual information amounts to reducing statistical

dependencies among the outputs.

1.2.3 Becker and Hinton’s Imax Principle

The goal of retaining as much information as possible may be a good description

of early sensory coding. However, the brain seems to do much more than simply

preserve information and recode it into a more convenient form. Our perceptual

systems are exquisitely tuned to certain regularities in the world, and consequently

to irregularities which violate our expectations. The things which capture our

attention and thus motivate us to learn and act are those which violate our

expectations about the coherence of the world—the sudden onset of a sound, the

appearance of a looming object or a predator.

In order to be sensitive to changes in our environment, we require internal rep-

resentations which ﬁrst capture the regularities in our environment. Even relatively

low-order regularities, such as the spatial and temporal coherence of sensory sig-

nals, convey important cues for extracting very high level properties about objects.

For example, the coherence of the visual signal across time and space allows us to

segregate the parts of a moving object from its surrounding background, while the

coherenceofauditoryeventsacrossfrequencyandtimepermitsthesegregationof

the auditory input into its multiple distinct sources.

Becker and Hinton (1992) proposed the Imax principle for unsupervised learn-

ing, which dictates that signals of interest should have high mutual information

across diﬀerent sensory channels. In the simplest case, illustrated in ﬁg 1.3, there

are two input sources, x

and x

, conveying information about a common under-

lying Gaussian signal of interest, s, and each channel is corrupted by independent,

additive Gaussian noise:

= s + n

However, the input may be high dimensional and may require a nonlinear

transformation in order to extract the signal. Thus the goal of the learning is to

transform the two input signals into outputs, y

and y

, having maximal mutual

8 Modeling the Mind: From Circuits to Systems

Maximize I(a;b)

Left strip

Right strip

Figure 1.4 Imax architecture used to learn stereo features from binary images.

information. Because the signal is Gaussian and the noise terms are assumed to

be identically distributed, the information in common to the two outputs can be

maximized by maximizing the following log signal-to-noise ratio (SNR):

≈ log



V (y

+ y

)

V (y

− y

)



This could be accomplished by multiple stages of processing in a nonlinear neural

circuit like the one shown in ﬁg 1.4. Becker and Hinton (1992) showed that this

model could extract binocular disparity from random dot stereograms, using the

architecture shown in ﬁg 1.4. Note that this function requires multiple stages

of processing through a network of nonlinear neurons with sigmoidal activation

functions.

The Imax algorithm has been used to learn temporally coherent features

(Becker, 1996; Stone, 1996), and extended to learn multidimensional features (Zemel

and Hinton, 1991). A very similar algorithm for binary units was developed by Kay

and colleagues (Kay, 1992; Phillips et al., 1998). The minimizing disagreement

algorithm (de Sa, 1994) is a probabilistic learning procedure based on principles of

Bayesian classiﬁcation, but is nonetheless very similar to Imax in its objective to

extract classes that are coherent across multiple sensory input channels.

1.2.4 Risannen’s Minimum Description Length Principle

The overall goal of every unsupervised learning algorithm is to discover the im-

portant underlying structure in the data. Learning algorithms based on Shannon

information have the drawback of requiring knowledge of the probability distri-

bution of the data, and/or of the extracted features, and hence tend to be either

very computationally expensive or to make highly simplifying assumptions about

the distributions (e.g. binary or Gaussian variables). An alternative approach is

to develop a model of the data that is somehow optimal with respect to coding

eﬃciency. The minimum description length (MDL) principle, ﬁrst introduced byminimum descrip-

tion length

1.2 Sensory Coding 9

d’

data

model

reconstructed

data

code

-1

Figure 1.5 Minimum description length (MDL) principle.

Rissanen (1978), favors models that provide accurate encoding of the data using as

simple a model as possible. The rationale behind the MDL principle is that the cri-

terion of discovering statistical regularities in data can be quantiﬁed by the length

of the code generated to describe the data.

A large number of learning algorithms have been developed based on the MDL

principle, but only a few of these have attempted to provide plausible accounts of

neural processing. One such example was developed by Zemel and Hinton (1995),

who cast the autoencoder problem within an MDL framework. They proposed

that the goal of learning should be to encode the total cost of communicating

the input data, which depends on three terms, the length of the code, c, the cost of

communicating the model, M (which depends on the coding cost of communicating

how to reconstruct the data), M

−1

, and the reconstruction error:

Cost = Length(c)+Length(M

−1

)+Length(|d − d



as illustrated in ﬁg 1.5. They instantiated these ideas using an autoencoder archi-

tecture, with hidden units whose activations were Gaussian functions of the inputs.

Under a Gaussian model of the input activations, it was assumed that the hidden

unit activations, as a population, encode a point in a lower-dimensional implicit

representational space. For example, a population of place cells in the hippocam-

pus might receive very high dimensional multisensory input, and map this input

onto a population of neural activations which codes implicitly the animal’s spatial

location—a point in a two-dimensional Cartesian space. The population response

could be decoded by averaging together the implicit coordinates of the hidden units,

weighted by their activations. Zemel and Hinton’s cost function incorporated a re-

construction term and a coding cost term that measured the ﬁt of the hidden unit

activations to a Gaussian model of implicit coordinates. The weights of the hidden

units and the coordinates in implicit space were jointly optimized with respect to

this MDL cost.

Algorithms which perform clustering, when cast within a statistical framework,

can also be viewed as a form of MDL learning. Nowlan derived such an algorithm,

called maximum likelihood competitive learning (MLCL), for training neural net-

剩余524页未读，继续阅读

Zylinkultamyyrä

粉丝: 10
资源: 103

统计信号处理新进展：从系统到大脑

rbf神经网络参考文献

CBF常规波束形成算法的参考文献

s.haykin, digital communications

s. haykin, \adaptive filter theory"

s. haykin, neural networks: a comprehensive foundation, prentice hall intern

communication systems 4th edition simon haykin

介绍人工神经网络，并附上参考文献

学习通信原理的参考书籍

傅里叶变换，短时傅里叶变换，小波变换的参考文献

自适应滤波器原理haykin 代码

最新资源