
generally unknown, we need to estimate the scene mo
-
tion for each frame with reference to one particular frame.
The warping process performed on HR image
x
is actu
-
ally defined in terms of LR pixel spacing when we esti
-
mate it. Thus, this step requires interpolation when the
fractional unit of motion is not equal to the HR sensor
grid. An example for global translation is shown in Figure
4. Here, a circle (䊊) represents the original (reference)
HR image
x
, and a triangle (䉭) and a diamond (䉫)are
globally shifted versions of
x
. If the down-sampling factor
is two, a diamond (䉫) has (0.5, 0.5) subpixel shift for the
horizontal and vertical directions and a triangle (䉭) has a
shift which is less than (0.5,0.5). As shown in Figure 4, a
diamond (䉫) does not need interpolation, but a triangle
(䉭) should be interpolated from
x
since it is not located
on the HR grid. Although one could use ideal interpola-
tion theoretically, in practice, simple methods such as
zero-order hold or bilinear interpolation methods have
been used in many literatures.
Blurring may be caused by an optical system (e.g., out
of focus, diffraction limit, aberration, etc.), relative motion
between the imaging system and the original scene, and
the point spread function (PSF) of the LR sensor. It can be
modeled as linear space invariant (LSI) or linear space vari
-
ant(LSV),anditseffectsonHRimagesarerepresentedby
the matrix
B
k
. In single image restoration applications,
the optical or motion blur is usually considered. In the SR
image reconstruction, however, the finiteness of a physical
dimension in LR sensors is an important factor of blur.
This LR sensor PSF is usually modeled as a spatial averag
-
ing operator as shown in Figure 5. In the use of SR recon
-
struction methods, the characteristics of the blur are
assumed to be known. However, if it is difficult to obtain
this information, blur identification should be incorpo
-
rated into the reconstruction procedure.
The subsampling matrix
D
generates aliased LR im
-
ages from the warped and blurred HR image. Although
the size of LR images is the same here, in more general
cases, we can address the different size of LR images by
using a different subsampling matrix (e.g.,
D
k
). Al
-
though the blurring acts more or less as an anti-aliasing
filter, in SR image reconstruction, it is assumed that
aliasing is always present in LR images.
A slightly different LR image acquisition model can be
derived by discretizing a continuous warped, blurred
scene [24]-[28]. In this case, the observation model must
include the fractional pixels at the border of the blur sup-
port. Although there are some different considerations
between this model and the one in (1), these models can
be unified in a simple matirx-vector form since the LR
pixels are defined as a weighted sum of the related HR
pixels with additive noise [18]. Therefore, we can express
these models without loss of generality as follows:
yWxn
kk k
=+ =, ,...,for kp1
, (2)
where matrix
W
k
of size
()NN LNLN
12
2
112 2
×
repre
-
sents, via blurring, motion, and subsampling, the contri
-
bution of HR pixels in
x
to the LR pixels in
y
k
. Based on
the observation model in (2), the aim of the SR image re
-
construction is to estimate the HR image
x
from the LR
images
y
k
for
kp= 1,...,
.
Most of the SR image reconstruc
-
tion methods proposed in the litera
-
ture consist of the three stages
illustrated in Figure 6: registration,
interpolation, and restoration (i.e.,
inverse procedure). These steps can
be implemented separately or simul
-
taneously according to the recon
-
struction methods adopted. The
estimation of motion information is
referred to as registration, and it is ex
-
tensively studied in various fields of
image processing [67]-[70]. In the
24 IEEE SIGNAL PROCESSING MAGAZINE MAY 2003
: Original HR Grid
: Original HR Pixels
: Shifted HR Pixels
,
s
4. The necessity of interpolation in HR sensor grid.
The SR image reconstruction is
proved to be useful in many
practical cases where multiple
frames of the same scene can
be obtained, including medical
imaging, satellite imaging, and
video applications.
HR Grid
LR Grid
HR Pixel
a
1
a
0
a
2
a
3
HR Image
LR Pixel =
(
)
LR Image
4
Σ
a
i
s
5. Low-resolution sensor PSF.
Authorized licensed use limited to: Peking University. Downloaded on May 18, 2009 at 07:52 from IEEE Xplore. Restrictions apply.