260 IEEE TRANSACTIONS ON BROADCASTING, VOL. 65, NO. 2, JUNE 2019
Prototype Filter Design to Minimize Stopband
Energy With Constraint on Channel Estimation
Performance for OQAM/FBMC Systems
Yuan Tian, Da Chen, Kai Luo , and Tao Jiang , Senior Member, IEEE
Abstract—In this paper, we propose a prototype filter design
method to minimize the stopband energy of the filter with guar-
anteed time-domain channel estimation performance for offset
quadrature amplitude modulation based filter bank multicar-
rier (OQAM/FBMC) systems. First, we analytically derive the
total mean squared error (MSE) of estimated channel impulse
response with two channel estimation schemes, i.e., the linear
minimum mean square error and weighted least square, where
the impact of intrinsic interference from data symbols is taken
into account. Then, we formulate an optimization problem of
filter coefficients to minimize the stopband energy with con-
straint on the total MSE. By exploiting the Heisenberg–Gabor
uncertainty principle, the original optimization problem is trans-
formed into an equality-constrained one. Finally, we employ the
Lagrange multiplier method to solve the transformed problem
and obtain the filter coefficients with the Newton’s method.
Simulation results demonstrate that the proposed filter outper-
forms the isotropic orthogonal transform algorithm and extended
Gaussian function filters considering both the MSE and stopband
energy performances.
Index Terms—OQAM/FBMC, prototype filter, stopband
energy, channel estimation.
I. INTRODUCTION
A
S A PROMISING technique in future wireless com-
munication systems and broadcasting systems [1]–[4],
the offset quadrature amplitude modulation based filter bank
multicarrier (OQAM/FBMC) has been attracting significant
attention in recent years [5]–[8]. In contrast to the orthog-
onal frequency division multiplexing (OFDM) system, the
OQAM/FBMC system allows the usage of flexible pulse
shaping filters rather than the rectangular filter, bringing in
advantages in various aspects. The main advantage is the
Manuscript received January 26, 2018; revised April 19, 2018; accepted
May 8, 2018. Date of publication July 2, 2018; date of current version
June 5, 2019. This work was supported in part by the National Natural Science
Foundation of China under Grant 61601191, in part by the Major Program
of National Natural Science Foundation of Hubei in China under Grant
2016CFA009, the Fundamental Research Funds for the Central Universities
under Grant 2015ZDTD012, the State Key Program of National Natural
Science Foundation of China under Grant 61631015, and the National Natural
Science Foundation of China under Grant 61571200. (Corresponding author:
Tao Jiang.)
The authors are with the School of Electronic Information and
Communications, Huazhong University of Science and Technology,
Wuhan 430074, China (e-mail: yuan_tian@hust.edu.cn; chenda@hust.edu.cn;
kluo@hust.edu.cn; tao.jiang@ieee.org).
Color versions of one or more of the figures in this paper are available
online at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TBC.2018.2847453
significantly reduced out-of-band (OOB) emission, which is
preferred by applications that exploit the opportunistic trans-
mission on the discontinuous spectrum [9]–[11]. Another
major benefit is the increase in transmission rate and spec-
tral efficiency, since no cyclic prefix (CP) is required in the
OQAM/FBMC transmission [12], [13]. As shown in [3], in
the terrestrial digital video broadcasting (DVB-T) standard, the
OQAM/FBMC technique can improve the data rate by 13% in
comparison with OFDM. Moreover, the OQAM/FBMC system
performs well in the asynchronous context as it does not
require strict synchronization [14], [15].
As the key component in the OQAM/FBMC system, the
prototype filter significantly affects two aspects of system
performance [16]. On the one hand, the power spectral density
(PSD) of the prototype filter determines the PSD of the trans-
mitted OQAM/FBMC signal, thus the prototype filter affects
the OOB emission performance of the system [17], [18]. On
the other hand, the prototype filter determines the intrinsic
interference in the OQAM/FBMC system. Since the intrin-
sic interference damages the pilot symbols in the channel
estimation, the prototype filter affects the channel estimation
performance of the system [19]–[21]. Since low OOB emission
is beneficial for achieving high spectral efficiency and accurate
channel estimation is necessary for reliable signal detection at
the receiver [22], the design of prototype filters is crucial for
the OQAM/FBMC system to achieve desired performances.
To achieve good OOB performance of the OQAM/FBMC
system, the general approach is to optimize the stopband
performance of the prototype filter. In the literature, differ-
ent optimization criteria have been proposed, including the
least-squares (LS) criterion to minimize the stopband energy
of the prototype filter [23]–[25], the minimax criterion to
minimize the maximum stopband ripple [25], [26], and the
peak-constrained least-squares (PCLS) criterion to minimize
the stopband energy with constraint on the maximum stopband
ripple of the filter [27], [28]. Moreover, the prototype filter
can be expressed by various methods, such as the frequency
sampling technique [29], [30], the windowing based tech-
nique [31], and the direct design method [32]. Among these
methods, the direct design method aims at optimizing all the
filter coefficients that affect the performance of the filter, thus
it has the potential to achieve the best performance. By utiliz-
ing the direct design method, the minimum stopband energy
of the prototype filter was obtained under the nearly perfect
reconstruction (NPR) constraint in [32]. Obviously, the design
0018-9316
c
2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.