IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 23, NO. 3, MARCH 2013 445
Iterative True Motion Estimation for
Motion-Compensated Frame Interpolation
DongYoon Kim, Hyungjun Lim, and HyunWook Park, Senior Member, IEEE
Abstract—This paper proposes a new motion-compensated
frame interpolation (MCFI) method. The performance of MCFI
is highly dependent on the accuracy of the estimated motion
vectors (MVs). The proposed iterative motion estimation (ME)
method estimates reliable MVs using linearity checking and
restricted search areas. An adaptive interpolation method is also
proposed to obtain MVs for blocks that do not have reliable
MVs in the iterative ME process. The experimental results show
that the proposed method outperforms the conventional MCFI
methods in terms of the quality of the interpolated frames.
Furthermore, the MVs from the proposed method are more
accurate than those from the conventional methods.
Index Terms—Frame rate upconversion, iterative motion
estimation, motion-compensated frame interpolation, true motion
vector.
I. Introduction
F
RAME RATE upconversion (FRUC) is an upsampling
method for videos in the temporal domain. In order to
compress the video data, temporal downsampling can be used
where FRUC is required to recover the video sequence. For
hold-type displays, such as liquid-crystal displays, FRUC is
very efficient for reducing the motion blur effect [1]–[3].
A simple FRUC method is frame repetition or the averaging
of the previous and next frames. However, these methods result
in blurring and motion jerkiness around the moving objects in
the images. In order to reduce the blurring and motion jerk-
iness, frames can be interpolated using motion-compensated
frame interpolation (MCFI) [1]. MCFI is composed of two
steps: motion estimation (ME) and frame interpolation. The
motion vectors (MVs) are estimated for blocks or pixels of
frames under the assumption that the blocks or pixels move
in the trajectories of their MVs. Frame interpolation is then
performed using the estimated MVs.
During the ME of MCFI, block matching algorithm (BMA)
[1] is typically used because it is simple to implement. Several
ME methods that utilize a BMA [2]–[10] have been proposed
in an effort to increase the accuracy of the estimated MVs.
These methods include bidirectional ME using side match
distortion [2], multiframe-based ME [3], overlapped block ME
Manuscript received November 15, 2011; revised March 8, 2012 and April
27, 2012; accepted June 6, 2012. Date of publication August 2, 2012; date
of current version March 7, 2013. This work was supported in part by the
Samsung Electronics Company in 2012, and the National Research Foundation
of Korea, under Grant 2012-0000125 funded by the Korean Government
(MEST). This paper was recommended by Associate Editor J. F. Arnold.
The authors are with the Department of Electrical Engineering, Korea
Advanced Institute of Science and Technology, Daejeon 305-701, Korea
(e-mail: themoon9@kaist.ac.kr; hyungjun@kaist.ac.kr; hwpark@kaist.edu).
Digital Object Identifier 10.1109/TCSVT.2012.2207271
(OBME) [4], and dual-ME using unidirectional OBME and
bidirectional OBME [5]. In order to improve the accuracy of
the estimated MVs, several postprocessing methods have been
proposed, including vector median filtering to select the me-
dian vector from neighboring MVs [6], adaptive vector median
filtering [7], reliability-based MV processing [8], multistage
MV processing [9], and correlation-based MV processing
[10].
For the frame interpolation, overlapped block motion com-
pensation was used to reduce blocking artifacts [1], [2], [8].
In addition, an autoregressive model was proposed for pixel
interpolation [11], [12]. Trilateral filtering was also proposed
to reduce the interpolation error and to fill holes in the
interpolated frame [13].
In conventional MCFI algorithms, the MV reliability of
a block is defined by computing the cost function of the
block matching. This reliability verification is based on the
assumption that a true MV has a small value for the cost
function because the intensity values of pixels rarely change
between the two frames. However, in videos, there are many
exceptions such as shape variations of objects, noise, illumi-
nation changes, and occlusion. In these situations, some MVs
with small costs may be incorrect MVs. Therefore, merely
considering the difference between blocks in the previous and
next frames is not sufficient when attempting to find the true
motion. The proposed method uses linearity checking between
the MV fields of the unidirectional forward ME (UFME)
and unidirectional backward ME (UBME) in order to define
MV reliability. In a previous study [14], it was demonstrated
that the interpolated block was reliable when the forward
and backward MVs passing through the same block in the
interpolated frame were linear. Therefore, a reliable MV can be
identified by means of the linearity checking of both forward
and backward MVs instead of using the cost function of block
matching.
During the process of the error concealment in video
transmission [15]–[17], the missing blocks are recovered at the
decoder using spatial and temporal information. In earlier work
[15], recursive template matching was proposed to recover
frames with multiple error slices. Similar to this procedure, a
template matching method with restricted search areas (RSAs)
is proposed to interpolate the blocks that fail to obtain reliable
MVs adaptively in the interpolated frame.
This paper is organized as follows. In Section II, the
proposed algorithm is explained. Section III presents the ex-
perimental results. Finally, this paper concludes in Section IV.
1051-8215/$31.00
c
2012 IEEE