Towards a Practical Face Recognition System:
Robust Registration and Illumination by Sparse Representation
Andrew Wagner, John Wright, Arvind Ganesh, Zihan Zhou, Yi Ma
University of Illinois at Urbana-Champaign, 1308 W. Main st. Urbana, IL 61801
{awagner, jnwright, abalasu2, zzhou7, yima}@illinois.edu
∗
Abstract
Most contemporary face recognition algorithms work
well under laboratory conditions but degrade when tested in
less-controlled environments. This is mostly due to the diffi-
culty of simultaneously handling variations in illumination,
alignment, pose, and occlusion. In this paper, we propose a
simple and practical face recognition system that achieves
a high degree of robustness and stability to all these vari-
ations. We demonstrate how to use tools from sparse rep-
resentation to align a test face image with a set of frontal
training images in the presence of significant registration
error and occlusion. We thoroughly characterize the region
of attraction for our alignment algorithm on public face
datasets such as Multi-PIE. We further study how to obtain
a sufficient set of training illuminations for linearly interpo-
lating practical lighting conditions. We have implemented
a complete face recognition system, including a projector-
based training acquisition system, in order to evaluate how
our algorithms work under practical testing conditions. We
show that our system can efficiently and effectively recog-
nize faces under a variety of realistic conditions, using only
frontal images under the proposed illuminations as training.
1. Introduction
Automatic face recognition remains one of the most ac-
tive areas in computer vision. While classical algorithms
[11, 2] remain popular for their speed and simplicity, they
tend to fail on large-scale, practical tests, falling short of
the ultimate goal of truly automating face recognition for
real-world applications such as access control for facilities,
computer systems and automatic teller machines. These ap-
plications are interesting both for their potential sociolog-
ical impact and also because they allow the possibility of
carefully controlling the acquisition of the training data, al-
lowing more tractable and reliable solutions.
1
In this set-
ting, one promising recent direction, set forth in [14], casts
the recognition problem as one of finding a sparse repre-
sentation of the test image in terms of the training set as a
∗
This work was supported by NSF IIS 08-49292, NSF ECCS 07-01676,
and ONR N00014-09-1-0230 grants. John Wright was partially supported
by a Microsoft Fellowship.
1
Face recognition with less-controlled training samples taken under un-
controlled scenarios remains an active research area as well [7].
Figure 1. Compound effect of registration and illumination.
The task is to identify the girl among 20 subjects, by computing
the sparse representation of her input face with respect to the en-
tire training set. The absolute sum of the coefficients associated
with each subject is plotted on the right. We also show the faces
reconstructed with each subject’s training images weighted by the
associated sparse coefficients. The red line (cross) corresponds to
her true identity, subject 12. Top: The input face is from Viola and
Jones’ face detector (the black box) and all 38 illuminations spec-
ified in Section 3 are used in the training. Middle: The input face
is well-aligned (the white box) with the training by our algorithm
specified in Section 2 but only 24 frontal illuminations are used in
the training for recognition (see Section 3). Bottom: Informative
representation obtained by using both well-aligned input face and
sufficient (all 38) illuminations in the training.
whole, up to some sparse error due to occlusion.
While that work achieves impressive results on public
datasets taken under controlled laboratory conditions such
as Extended Yale B [4], it fails to address two critical as-
pects of real world face recognition: significant variations
in both the image domain and in the image value. We il-
lustrate this with an example in Figure 1. The task is to
identify the girl among 20 subjects. If the test face im-
age, say obtained from an off-the-shelf face detector, has
even a small amount of registration error against the train-
ing images (caused by mild pose, scale, or misalignment),
the representation is no longer informative, even if suffi-
1
597978-1-4244-3991-1/09/$25.00 ©2009 IEEE