922 IEEE SIGNAL PROCESSING LETTERS, VOL. 19, NO. 12, DECEMBER 2012
Sparse and Redundant Representation
Modeling—What Next?
Michael Elad, Fellow, IEEE
Abstract—Signal processing relies heavily on data models; these
are mathematical constructions imposed on the data source that
force a dimensionality reduction of some sort. The vast activity in
signal processing during the past d ecades is essentially driven by an
evolution of these models and their use in practice. In that respect,
the past decade has been certainly the era of sparse and redundant
representations, a popular and highly effective data model. This
very appealing model led to a long series of intriguing theoretical
and numerical questions, and to many innovative ideas that har-
ness this model to real engineering problems. The new entries re-
cently added to the IEEE-SPL EDICS reflect th e popularity o f this
model and its impact on signal processing research and practice.
Despite the huge succes s of th is model so far, this field i s still at
its infan cy, with many unanswered questions still remaining. This
paper
1
offers a brief presentation of the story of sparse and redun-
dant representation modeling and its impact, and outlines ten key
future research directions in this field.
Index Terms—Data models, dimensionality reduction, sparse
and redundant representations, projection, pursuit.
I. INTRODUCTION—WHO NEEDS MODELS?
O
NE cou ld not imagine the vast progress m ade in sign al
and image processing in the past fifty years without the
central contribution of d ata mo dels. Consider the following ex-
ample as a way of illustrat ing the need f or a model: A signal of
interest
is measur ed in the presence of additive noise,
, pro ducing .Given we would like to
recover
, essentially seeking a decomposit ion of into its two
parts,
and . Despite the fact that we have a full statistical
characterization of the noise, such a separation is impossible,
as the noise model only implies a Gaussian distribution for
,
peaked at none other than
itself. To depar t from this triviality,
we must characterize
as well, so that the two parts can be told
apart.
A model for the signal
is exactly this— a mathematical char-
acterization of the signal. As an example for a possible m odel
and its use, if we kno w that
resides in a subspace of dimension
spanned by the columns of th e matr ix ,this
constitutes a model, and denoising (cleaning the noise) becomes
possible. One could project
onto this subspace, applying the
operation
,inordertofind the closest signal to that
Manuscript received September 23, 2012; accepted October 09, 2012. Date
of publication October 12, 2012; date of current version November 21, 2012.
This work was supported by the ERC Advanced Grant Agreement 320649.
M. Elad is with the Computer Scien ce Department, Technio n—Israel Institute
of Technology, Haifa 32000, Israel (e-mail: elad@cs.technion.ac.il).
Dig
ital Object Identifier 10.1109/LSP.2012.2224655
1
Thi
s is not a regular IEEE-SPL paper, but rather an invited contribution of-
fer
ing a vision for key ad vances in emerging fields.
complies with the model. Put formally, this projection is ob-
tained as the solution of the problem
(1)
where
represents our subspace co nstra
int for the signal.
2
This w ill lead to effective denoising, wi
th noise attenuation by
a factor
on average.
The signal and image processing liter
ature has seen nu-
merous attempts to handle the above-
described denoising
problem. Explicitly or implicitly
, each and every one o f these
many thousands of published metho
ds relies on a specific
model, proposing a way to characte
rize the signal and a method
to exploit this for the recovery o
f
. W hile the above model
example is very simple, it sheds
light on key properties o f
models in general. An effect
ive mo del typically suggests a
dimensionalit y reduction
of some sort; the original
samples
in the signal
are believe
d to be redundant and a much shorter
description (in our examp
le, of length
) can be given, reflecting
thetruedimensionality
of the signal. Another issue is the mi-
gration from the core mo
del formulation to its deployment in
the processi ng task.
In the example we suggested a projection
of
, which is very na
tural. However, when the model becomes
more expressive and
complex, leaving room for more than one
approximation, va
rious possible “estimators” may be proposed;
thus, in general t
here is no one-to-one correspondence between
a model and the way
to practice it, and this leaves much room
for original an
d creative ideas.
While the above
example discussed the denoising problem,
models are nece
ssary f or almost every processing that
may
need to underg
o. Sampling of a sig nal relies on a prior assump-
tion about its
content, so as to guarantee no l oss or limited loss
of informati
on. Compression of a signal is conceptually pos-
sible only b
ecause it has an inner structure that reduces its en-
tropy, whi
ch is captured by a model that describes this signal
with few pa
rameters. Detection of anom a lies in a si gnal or de-
tection o
f target-content can be done only when we rely o n
models f
or th e different contents. Similarly, separation of super-
imposed
signals is done by using models for the distinct parts.
Solvin
g inverse problems such as a tomographic recon stru ction
from p
rojections, super-resolving a signal, inpainting (filling-in
missi
ng values), extrapolating a signal, and deblurring, all rely
on a m
odel for the signals in question in o rder to regularize the
2
If i
s known to be spanned by the columns o f
, it can be written as
for
an arbitrary vector
. Mu ltiplying both sides by we obtain
,since .Substitutingthisrelationbackinto
we get the constraint , as appears in (1).
1070-9908/$31.00 © 2012 IEEE