HUANG et al.: GENERALIZED GROWING AND PRUNING RBF (GGAP-RBF) 59
much smaller networks than SMO, and provides a comparable
generalization performance.
The rest of this paper is organized as follows. Section II in-
troduces the definition of significance of neurons and then gives
a simple estimation scheme for the significance. Section III de-
scribes the proposed “significance”-based generalized growing
and pruning RBF (GGAP-RBF) learning algorithm derived for
arbitrary input sampling density functions. Section IV presents
the performance comparison results for GGAP-RBF along with
RAN, RANEKF, and MRAN for two bench-mark problems, i.e.,
California Housing which is a real large-scale complex function
approximation problem and Mackey–Glass chaotic time series
prediction problem. Section V summarizes the conclusions from
this study.
II. D
EFINITION AND
ESTIMATION OF
SIGNIFICANCE
OF NEURONS
This section first introduces the notion of significance for the
hidden neurons based on their statistical average contribution
over all inputs seen so far, although those inputs are discarded
and not stored in the system after being learned.
The output of a RBF network with
neurons for an input
vector
, where is the dimension
of input observation space
and means the transpose
of vectors, is given by
(1)
where
is the weight connecting the th hidden neuron to
the output neuron and
is the response of the th hidden
neuron for an input vector
(2)
where
and are the center and
width of the
th hidden neuron, respectively, .
In sequential learning, a series of training samples are ran-
domly drawn and presented to, and learned by the network one
by one. Let a series of training samples
,be
drawn sequentially and randomly from a range
with a sam-
pling density function of
, where is a subset of an -di-
mensional Euclidian space. The sampling density function
is defined as
(3)
For simplicity, in this paper,
is denoted by . The size
of the range
can be denoted by . After se-
quentially learning
observations, assume that a RBF network
with
neurons has been obtained. The network output for an
input
is given by
(4)
If the neuron
is removed, the output of the RBF network with
the remaining
neurons for the input is
(5)
Thus, for an observation
, the error resulted from removing
neuron
is given by
(6)
where
is the -norm of vectors, indicating the -distance
between two points in Euclidian space.
In theory, the
-norm of the error for all sequentially
learned observations caused by removing the neuron
is
(7)
This can be further written as
(8)
However, the computation complexity of
would be very
high if it were calculated based on all learned observations and
if
is large. On the other hand, in the sequential learning imple-
mentation after learning the training observations
, are no longer stored in the system and the value may
be unknown and not recorded either. In fact, there may pos-
sibly have some more observations to be input further. Thus,
there must be some simpler and better way to calculate
as stated in (8) without the prior knowledge of each specified
observations
.
Suppose that the observations
are
drawn from a sampling range
with a sampling density func-
tion
. Suppose that at an instant of time, observations
have been learned by the sequential learning system.
Let the sampling range
be divided into small spaces
. The size of is represented by .
Since the sampling density function is
there are about
samples in each , where is any point
chosen in
. From (8), we have
(9)