may also be well-suited for VLSI implementation using
simplified decision criterion, however they suffer from
obvious performance degradation. Three typical simpli-
fied criterions are sum of absolute difference (SAD), sum
of absolute transformed difference (SATD), and weighted
SAD (WSAD) [3,4]. By employing the Lagrangian optimi-
zation technique, the WSAD criterion achieves superior
performance than SAD or SATD. Nevertheless, its coding
performance degradation compared with genuine RDO
based method is still obvious with unnegligible image
quality degradation.
The performance degradation is mainly derived from
measure simplification of rate and distortion. Suppose S
and S
0
are the original MB and the reconstructed one, and
P is the predicted version of the current MB of a certain
mode. Q
p
and
l
are the MB quantization step and the
Lagrange multiplier for mode decision. Two mode deci-
sion criterions RDcost and WSAD are described in the
following equations:
RDcostðS, Su, mode9Q
p
,
l
Þ
¼ SSDðS, Su, mode, Q
p
Þþ
l
R
MB
ðS, Su, mode, Q
p
Þð1Þ
WSADðS, P, mod9Q
p
,
l
Þ
¼ SADð S, P, mod, Q
p
Þþ
l
u R
MBheader
ðS, P, mod, Q
p
Þð2Þ
Here, SSD (S,S
0
,mode,Q
p
) is the sum of the squared
difference between S and S
0
in the case of Q
p
and
l
, while
SAD ( S, P,mode,Q
p
) is the sum of the absolute difference
between S and P.R
MB
(S,S
0
,mode,Q
p
) is the coding bit of all
syntax elements in the MB in the case of Q
p
and
l
0
.
R
MBheader
(S,P,mode,Q
p
) is the coding bit of the syntax
elements in the MB header.
RDO based mode decision achieves superior coding
performance contributed by Lagrangian optimization.
Genuine distortion is measured with SSD (S,S
0
,mode,Q
p
)
in the case of RDcost criterion, genuine rate is also used in
the case of RDcost measured with R
MB
(S,S
0
,mode,Q
p
) with
all syntax elements considered. Comparatively, only rate
factor is considered for mode decision in the case of
WSAD criterion, in which rate is estimated with SAD
(S,P,mode,Q
p
) and R
MBheader
(S,P,mode,Q
p
). The prediction
residue SAD (S,P,mode,Q
p
) is approximately used as the
rate measure for quantized DCT coefficients.
It is the measure simplifications of rate and distortion in
WSAD that result in the obvious performance degradation
compared with RDcost. In order to sustain the superiority
of AVS, we will focus on RDO based mode decision for
hardware implementation in this work.
It is very computationally intensive due to the abundant
modes adopted in H.264. However, almost all H.264 video
encoder architectures adopt simplified mode decision, and
WSAD, SATD, or SAD criterion was used instead. Relatively,
challenges of RDO based mode decision in AVS video
encoder is relatively lower than H.264. On the one hand,
the numbers of inter and intra modes in AVS are smaller
than that of H.264, and the processing throughput burden
is also lower than that of H.264. On the other hand, the
processing unit granularity in AVS such as DCT, quantiza-
tion, inverse DCT, inverse quantization is 8 8 block in the
rate and distortion calculation loop, while that is 4 4
block in H.264 smaller than AVS. This means that the
circuit consumption for the basic processing unit of AVS is
higher than that of H.264. Thus, it is possible to implement
RDO based mode decision with reasonable mode preselec-
tion to alleviate the throughput burden without too much
additional circuit consumption.
2.3. Computation analysis for RDcost estimation
Fig. 2 shows the framework of RDcost calculation for
one 8 8 block of a certain MB coding mode. First, the
difference between the input block S and its prediction P
is calculated by residue generation. Then, the residue r is
transformed by DCT and followed by quantization (Quant,
Q), and then the coding bit rate R is computed by entropy
coding (EC). The quantized coefficients are also fed into
inverse quantization (Inver Quant, IQ), inverse transform
(IDCT), and compensation to reconstruct the block S
0
. SSD
between the original residue (r) and the reconstructed
residue (r
0
) is computed for distortion measure. In the end,
RDcost is obtained according to R and SSD. RDcost is used
to evaluate the RD performance of all candidate modes,
and the mode with the smallest RDcost is selected for
bitstream generation.
RDO based mode decision for hardware implementa-
tion is challenged by the following two factors. On the one
hand, intrinsic data dependencies exist in video coding
algorithms. At the MB level, integer pixel motion estima-
tion (IME), fractional motion estimation (FME), mode
decision (MD), and intra prediction (IP), deblocking filter
Reference
(reconstructed)
Inter
prediction
Intra
prediction
DCT Q
IDCT IQ
Zigzag
Scan
VLC
coding bits
estimation
Distortion
estimation
Rdcost Comparison
and mode decision
Filter
Inter
Intra
S’
r'
r'
r
P
rS
Fig. 2. Framework of cost function calculation and mode decision.
Hai bing Yin et al. / Signal Processing: Image Communication 25 (2010) 633–647 635