A NEW PANSHARPENING METHOD USING OBJECTNESS BASED SALIENCY ANALYSIS
AND SALIENCY GUIDED DEEP RESIDUAL NETWORK
Libao Zhang*, Jue Zhang, Xinran Lyu, and Jie Ma
College of Information Science and Technology, Beijing Normal University, Beijing 100875, China
ABSTRACT
Pansharpening is a fundamental and crucial task in the
remote sensing community. For remote sensing images,
there is a significant difference in demands for spatial and
spectral resolution in different regions. From this
perspective, we propose a new pansharpening method using
objectness based saliency analysis and saliency guided deep
residual network to boost the fusion accuracy. We first
develop an objectness based saliency analysis by
incorporating texture feature and objectness measurements
to estimate saliency values in images and thereby help
discriminate different demands for spatial improvement and
spectral preservation. Inspired by the impressive
performance of deep learning, we subsequently construct a
saliency guided deep residual network to implement
pansharpening. In addition, in order to produce images with
subtler details, we design a new loss function, the
normalized mean square error, particularly for the
pansharpening task. Experiments support the superiority of
our proposal over six competing methods.
Index Terms—Image fusion, pansharpening, deep
residual network, saliency, normalized mean square error
1. INTRODUCTION
Due to physical constraints, satellite sensors today cannot
obtain remote sensing images with fine spatial and spectral
resolutions simultaneously. Pansharpening refers to methods
that generate high-resolution multi-spectral (HRMS) images
by combining high spatial resolution panchromatic (Pan)
images and low-resolution multi-spectral (MS) images [1].
According to strategies employed in injection models,
most pansharpening methods can be divided into two
families: multiresolution analysis (MRA) and component
substitution (CS) based methods. The two major types of
strategies provide excellent fusion performance, but
disadvantages of these methods are hard to ignore [2].
MRA-based pansharpening [3-5] has high spectral fidelity
*Corresponding author: libaozhang@163.com. This work was supported in
part by the Beijing Natural Science Foundation under Grant L182029, in
part by the National Natural Science Foundation of China under Grant
61571050, in part by BNU Interdisciplinary Research Foundation for the
First-Year Doctoral Candidates under Grant No. BNUXKJC1801.
but usually fails to provide sufficient spatial increment. CS
based methods such as Gram-Schmidt adaptive (GSA)
method [6], and band-dependent spatial-detail (BDSD)
model [7] have varying degrees of spectral distortion.
In recent years, as deep neural networks have achieved
great success in the field of computer vision, researchers in
the remote sensing community applied deep learning
techniques to pansharpening [8-10]. For example, Masi et al.
borrowed a simple three-layer pansharpening neural network
similar to SRCNN, a network recently proposed for super-
resolution [9]. Wei et al. took advantages of residual
learning and then developed an extremely deep
convolutional filtering framework to improve the results [10].
After deeply investigating the literature, we found
marked differences in requirements for spatial and spectral
resolutions of different regions in remote sensing images
[11]. Residential areas expect to have rich spatial details, in
order to yield better results for object detection whereas
forests and grassland are usually leveraged in vegetation
classification, in which spectral preservation is particularly
crucial. Obviously, the primary task to fulfill the
requirements is to discriminate those regions.
Saliency analysis originates from research on human
attention mechanism and aims to localize distinct or unique
objects that can attract people’s interest immediately in
scenes [12]. The yielded saliency maps have been
introduced to various applications in the computer vision
community such as object detection, image retrieval and
image compression. It is feasible to leverage saliency
analysis to partition regions with different demands of
spectral and spatial quality.
Motivated by these aspects, we design a new
pansharpening method using objectness based saliency
analysis and saliency guided deep neural network (as shown
in Fig. 1), which has the following contributions:
1) We develop a saliency guided deep residual network
by incorporating objectness based saliency analysis with a
deep residual network, which aims at satisfying different
needs of spatial and spectral resolution in different regions.
2) Instead of using l
2
loss, we train the network with a
newly proposed normalized mean square error, which helps
prevent blurring results and provide sharpened images with
subtler details.