PAN-SHARPENING BASED ON MULTILEVEL COUPLED DEEP NETWORK
Wanting Cai
1
, Yang Xu
1
, Zebin Wu
1,2,3*
, Hongyi Liu
4
, Ling Qian
5
, Zhihui Wei
1
1
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China
2
Nanjing Robot Research Institute Co. Ltd, Nanjing, 211135, China
3
Lianyungang E-Port Information Development Co. Ltd, Lianyungang, 222042, China
4
School of Science, Nanjing University of Science and Technology, Nanjing 210094, China
5
School of Electronic and Optical Engineering, Nanjing University of Science and Technology, Nanjing 210094, China
Pan-sharpening is a common image-fusion method. To
improve the quality of fused images, a multilevel deep
learning Pan-sharpening method is proposed in this paper.
In the training phase, we introduce Coupled Sparse
Denoising Autoencorder (CSDA) to reconstruct
high-Resolution (HR) multispectral (MS) image from
low-Resolution (LR) MS image and HR Panchromatic
(Pan) image. CSDA has four networks including LM-HP
network, HR-MS network, feature mapping network and
fine-tuning network. The hidden features in LM-HP
network and HR-MS network as well as the mapping
function between the two features are learned through joint
optimization. In LM-HP and HR-MS networks, the hidden
features of image patch pairs are extracted by the sparse
autoencoder. A sparse denoising autoencoder is used to
build the nonlinear mapping between the extracted features.
In the testing phase, the LR-MS and HR-Pan images
patches are fed to the CSDA network to reconstruct the
fused HR-MS image. The experimental results show that
the proposed method is better than the traditional
pans-sharpening methods.
Index Terms—Pan-sharpening, deep learning, sparse
autoencoder, multilevel coupled networks
1. INTRODUCTION
Pan-sharpening is an important method to realize remote
sensing image fusion. It is a method of generating
high-resolution (HR) multi-spectral (MS) image using
low-resolution (LR) MS image and single-band HR
panchromatic (Pan) image. Among them, HR-Pan image
provides higher spatial information and precise geometric
analysis, and LR-MS image provides the spectral
information. The spatial and spectral information of
HR-MS image are both high.
1
This work was supported in part by the National Natural Science
Foundation of China under Grant No. 61772274, 61471199, 61701238,
91538108, 61671243, 11431015, the Fundamental Research Funds for the
Central Universities under Grant No.30917015104, the Jiangsu Provincial
Natural Science Foundation of China under Grant BK20170858, the
Jiangsu Province Six Top Talents Project of China WLW-011.
*Corresponding author. Email: zebin.wu@gmail.com
Compared with the classical methods, deep learning
has the ability to represent more general inputs and extract
higher level representation. In recent years, many
researchers have applied deep learning to the
Pan-sharpening. Huang et al.[1] proposed a deep neural
network (DNN) algorithm with a method of stacking sparse
denoising autoencoder. However, this method cannot
extract deep features of two images well. In addition, Wei
et al.[2] proposed the deep residual pan-sharpening neural
network (DRPNN).
In recent year, the idea of coupling has also been
gradually applied to neural networks. Guo et al.[3]
proposed Super-resolution (SR) with Coupled
Backpropagation (SR-CBP). Zeng et al. [4] proposed a
Coupled Deep Autoencorder (CDA) based on a couple of
autoencoders for image SR. An effective feature
representation is obtained by jointly learning low resolution
image patches and high resolution image patches. However,
the non-linear mapping is a single-layer network, which has
the limited effect.
In this paper, a Pan-sharpening method named
coupled sparse denoising autoencoder (CSDA) is proposed.
The proposed method can jointly learn multilevel neural
networks. Based on Pan-sharpening needs to extract the
spatial features of the HR-Pan and the spectral features of
the LR-MS. By training other databases, we can get
end-to-end contact directly during testing Then, the HR MS
image is obtained as the output of the network. Xu [5]
shows that when the number of hidden neurons is less than
the number of inputs, the autoencoder can achieve the
effect of data compression. When the number of hidden
neurons is more, the sparsity constraint is imposed on the
hidden units, then the sparse autoencoder will still discover
interesting structure in the data. The role of sparsity allows
sparse automatic encoders to extract useful features with as
few neurons as possible in a large number of dimensions.
Therefore, Sparse AutoEncoder (SAE) is adopted in the
proposed method to extract the features of the data due to
the number of hidden neurons is more .
2. PAN-SHARPENING METHOD WITH CSDA
2.1 Overview of the proposed method
7046978-1-5386-7150-4/18/$31.00 ©2018 IEEE IGARSS 2018