Automatic Recognition of Facial Expressions in
Image Sequences: A Review
R A Patil
MNIT Jaipur
rapatil_rtg@yahoo.co.in
Vineet Sahula
MNIT Jaipur
sahula@ieee.org
A. S. Mandal
CEERI Pilani
atanendusekhar.mandal@gmail.com
Abstract— For human beings, facial expression is one of the
most powerful and natural way to communicate their emotions
and intensions. A human being can detect facial expressions
without effort, but for a machine it is very difficult. Automatic
facial expression recognition is an interesting and challenging
problem. Automatic facial expression recognition systems can
be mainly used for human computer interaction and data
driven animation. There are three sub problems while designing
automatic facial expression recognition system, face detection,
extraction of the facial expression information, and classification
of the expression. A system that performs these operations more
accurately and in real time would be crucial to achieve a human-
like interaction between man and machine. This paper reviews
the past work done in solving these problems for image sequences.
I. INTRODUCTION
Machine vision has been defined as the automatic ac-
quisition and analysis of images to obtain desired data for
interpreting a scene or controlling an activity. For a machine
vision system, it is not necessary to copy all the details of the
human visual system, but it is important to understand the true
complexity behind its power and flexibility. Current machine
vision research concerns, not only understanding the process of
vision, but also designing effective vision systems for various
real world applications. Facial expression recognition system
is an example of machine vision system.
For a human being the most powerful, and natural way
of communicating their emotions and intensions is facial
expression. A human being can detect facial expressions in
a scene without effort. But, to develop an automatic system
that performs this task is rather difficult. A demand of au-
tomatically extracting facial expression information has been
continuously increasing. Automatic facial expression analysis
is an interesting and challenging problem, and impacts im-
portant applications in many areas such as human computer
interaction, and data driven animation. Though much progress
has been made, recognizing facial expression with a high
accuracy remains challenge due to subtlety, complexity and
variability of facial expressions. There are six prototypical (or
basic) facial expressions, i.e. surprise, fear, sadness, anger,
disgust and happiness according to the previous researches,
which are known to be universal across human ethnicities and
cultures.
The goal of this paper is to survey the work done in au-
tomating facial expression analysis in facial image sequences.
Section II describes three basic problems related to facial
expression analysis. These problems are face detection in a
facial image sequence, facial expression data extraction and
facial expression classification. Here, we discuss few systems
which deal with each of these problems. We conclude in
section III.
II. FACIAL EXPRESSION ANALYSIS
Facial expression recognition problem can be divided into
three sub problems. (1) face detection (2) feature extraction
(3) classification. To detect the face from the image sequence
is first and the very important step. Next step is to develop
mechanism for extracting the facial features from the observed
facial image sequence. The facial features are the prominent
features of the various parts of the face- eyebrows, eyes, nose,
mouth, and chin. The feature extraction step is often referred to
as tracking the face and its features, in the scene. The final step
is to develop a classifier, which will classify a facial expression
into one of the basic facial expressions [1].
A. Face detection
Most of the methods of facial expression recognition as-
sumes that the conditions under which a facial image sequence
is obtained, are controlled. Usually, the image sequence has the
face in frontal view. Hence, the presence of a face in the scene
is ensured and some global location of the face in the scene
is known a priori. However, determining the exact location
978-1-4244-6653-5/10/$26.00 ©2010 IEEE
2010 5th International Conference on Industrial and Information Systems, ICIIS 2010, Jul 29 - Aug 01, 2010, India