OCCLUSION HANDLING FRAME RATE UP-CONVERSION
Yong Guo, Zhiyong Gao, Li Chen and Xiaoyun Zhang
Institute of Image Communication and Information Processing
Shanghai Jiao Tong University, Shanghai 200240, China
Shanghai Key Labs of Digital Media Processing and Communication, Shanghai 200240, China
Email: {guoyong1987, zhiyong.gao, hilichen, xiaoyun.zhang}@sjtu.edu.cn
ABSTRACT
Motion-compensated frame interpolation (MCFI) is a tech-
nique used extensively to enhance the temporal resolution of
video sequences. In order to obtain a high quality interpo-
lation, the motion vector field (MVF) between frames must
be well-estimated. However, many current techniques for de-
termining the MVF are prone to errors in occlusion regions.
In this work, we propose an improved algorithm for improv-
ing the quality of MCFI by restoring the unreliable MVF and
pixels in occlusion regions. We first utilize a dual motion
estimation (DME) scheme which performs better in occlu-
sion regions. Occlusion regions are determined by the ratio
of two directional matching errors. Then, MVs in occlusion
regions are refined using an orientation-based refinement (O-
BR) method, which promotes occluded MVs with its orthogo-
nal neighboring MVF. Finally, regional blending (RB) is pro-
posed to restore the unreliable pixels in occlusion regions for
further error concealment. Experimental results demonstrate
that the proposed algorithm provides a better quality than pre-
vious benchmark frame rate up-conversion (FRUC) methods
both objectively and subjectively.
Index Terms— frame rate up-conversion (FRUC), motion-
compensated frame interpolation (MCFI), dual motion esti-
mation (DME), orientation-based refinement (OBR), regional
blending (RB)
1. INTRODUCTION
Frame rate up-conversion (FRUC) is a technique that increas-
es the frame rate of the video by inserting newly generated
frames into the original sequence. FRUC is commonly ap-
plied in format conversion, and also used to reduce motion
artifacts of videos in hold-type displays, such as liquid crys-
tal displays (LCD). Motion-compensated FRUC (MC-FRUC)
is composed of two major steps: motion estimation (ME) and
motion-compensated frame interpolation (MCFI). The perfor-
mance of a FRUC algorithm strongly depends on the accuracy
of the estimated motion information as well as the design of
the interpolation filter.
To improve the accuracy of motion vector (MV) between
successive frames, many pioneering works have been done.
In [1], Haan et al. proposed a 3-D recursive search (3-DRS)
method using spatio-temporal related motion candidates to
get the true motion. Choi et al. [2] proposed a bilateral ME
method to avoid overlapped areas and holes. Huang et al. [3]
employed the reliability information of MV and developed a
hierarchical MV processing method to obtain a more reliable
MVF for FRUC. Although true motion estimation techniques
have developed in literature for FRUC, many current tech-
niques are prone to errors in occlusion regions.
When block-based MCFI is used for FRUC, blocking arti-
facts are usually perceived due to the block-wise motion com-
pensation. To produce better MCFI results, Wang et al. [4] re-
garded artifacts as MCFI noise and proposed a three-iterative
trilateral filter to remove noise and achieved satisfactory visu-
al effects.
Motivated by the above analysis, we propose in this pa-
per an improved FRUC algorithm which makes use of du-
al ME (DME), orientation-based refinement (OBR) and re-
gional blending (RB). The main reason why we use DME
is that it not only offers more accurate motion trajectories,
but also avoids overlapped areas and holes. The DME us-
es the matching ratio of forward and backward prediction-
s in order to perform the validity check of MVF and refine
them with OBR method. We use two MVFs to interpolate t-
wo additional frames, and estimate the pixel reliability from
the difference between these two predictions. Subsequently,
reliability-based regional blending is proposed for further er-
ror concealment.
The rest of this paper is organized as follows. Section II
presents the proposed algorithm. Section III shows the exper-
imental results and evaluates the performance of the proposed
method. Finally, Section IV concludes the paper.
2. PROPOSED ALGORITHM
The proposed FRUC algorithm comprises four components:
dual motion estimation (DME), MVF refinement, adaptive
overlapped block motion compensation (ABOMC) [2] and re-