2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII)
Improved Facial Expression Recognition Method Based on ROI Deep Convolutional
Neutral Network
Xiao Sun
School of Computer and Information
Hefei University of Technology
Hefei, Anhui, 230009
Email: sunx@hfut.edu.cn
Man Lv
School of Computer and Information
Hefei University of Technology
Hefei, Anhui, 230009
Email: lvxman@foxmail.com
Changqin Quan
Department of Computational Science
Kobe University
Kobe, Japan, 6578501
Email: quanchqin@gold.kobe-u.ac.jp
Fuji Ren
School of Computer and Information
Hefei University of Technology
Hefei, Anhui, 230009
Email: ren2fuji@gmail.com
Abstract—This paper, we proposed an improved facial expres-
sion recognition (FER) method based on region of interesting
(ROI) to guide the convolutional neutral networks (CNN) focus
on the areas associated with the expression. This method can
not only augment the training data, the relationship between
the different ROI areas is helpful to intensify the reliability of
the predicted targets. In test stage, we investigated two recog-
nition methods: identify the test image directly; implemented
decision fusion strategy on ROI areas. The model we used
is fine-tuned from pre-trained deep CNN instead of training
from scratch. In addition, we presented an innovative region-
based image augmentation method named artificial face to
increase the limited database. This method using expression
retargeting as an expression-preserving data augmentation
which is specific for FER. The performance of the proposed
method has been validated on the public CK+ databases.
1. Introduction
Facial expression recognition (FER) has received a great
deal of attention during the last decade because it is an
important tool when automatic interactions between humans
and machines, such as in developing hospital nurse robot
assistants, automatic animation, and intelligent tutoring sys-
tems. Despite efforts made in developing various methods
for FER [1], existing approaches traditionally lack gen-
eralizability and flexibility when classify images captured
in wild, therefore present a misleading high-accuracy. So,
recognizing facial expression in real time with high accuracy
is still a challenging problem due to image variations caused
by pose, illumination, age and occlusion.
Recently, convolutional neural network (CNN) has been
successfully used in a wide variety of image classification
tasks [2]. However, learning CNNs, amounts to estimating
millions of parameters and requires a very large number of
annotated image samples. This property currently prevents
application of CNNs to FER because the public databases
are limited. So, how to use a small amount of raw data to ef-
fectively expand dataset is a worthy of study. This paper, we
proposed an innovative data augmentation method named
artificial face to increase the limited database. This method
using expression retargeting as an expression-preserving
data augmentation which is specific for FER.
Almost all of the CNNs based method employed whole
face region as input and every part of the face is treated
equally no matter if it is relevant to the facial expression.
Studies in psychological showed that facial features of
expressions are located around the subjects mouth, nose
and eyes. In [3], Paul Ekman proposed the Facial Action
Coding System (FACS) which enumerated these regions and
described how every facial expression can be described as
the combination of multiple action units (AUs). Inspired
by the locations of AUs, we proposed an improved facial
expression recognition method based on region of interesting
(ROI) to guide the CNN focus on the areas associated with
the expression. This method can not only augment the train-
ing data, the relationships between the different ROI areas is
helpful to intensify the reliability of the predicted targets. In
test stage, we investigated two recognition methods: identify
the test image directly; implemented decision fusion strategy
on ROI areas. We employed the pre-trained AlexNet fine-
tuned on FER database to evaluation the performance of our
method.
2. Data Preparation
In order to solve the problem of insufficient data, we
proposed an innovative data augmentation method named
artificial face to increase the limited FER database. Then,
some preprocessing methods were implemented in all im-
ages we generated.
978-1-5386-0563-9/17/$31.00
c
2017 IEEE
256