Spectral Super-resolution for RGB Images
Using Class-based BP Neural Networks
Xiaolin Han
1
, Jing Yu
2
, Jing-Hao Xue
3
and Weidong Sun
1
1. State Key Laboratory of Intelligent Technology and Systems,
Beijing National Research Center for Information Science and Technology;
Department of Electronic Engineering, Tsinghua Univ., Beijing 100084, China
2. College of Computer Science and Technology, Beijing Univ. of Technology, Beijing100124, China
3. Department of Statistical Science, University College London, London, WC1E 6BT, U.K.
Abstract—Hyperspectral images are of high spectral
resolution and have been widely used in many applications, but
the imaging process to achieve high spectral resolution is at the
expense of spatial resolution. This paper aims to construct a
high-spatial-resolution hyperspectral (HHS) image from a high-
spatial-resolution RGB image, by proposing a novel class-based
spectral super-resolution method. With the help of a set of RGB
and HHS image-pairs, our proposed method learns nonlinear
spectral mappings between RGB and HHS image-pairs using
class-based back propagation neural networks (BPNNs). In the
training stage, unsupervised clustering is used to divide an RGB
image into several classes according to spectral correlation, and
the spectrum-pairs from the classified RGB images and the
corresponding HHS images are used to train the BPNNs, to
establish the nonlinear spectral mapping for each class. In the
spectral super-resolution stage, a supervised classification is used
to classify the given RGB image into the classes determined
during the training stage, and the final HHS image is
reconstructed from the classified given RGB image using the
trained BPNNs. Comparisons on three standard datasets, ICVL,
CAVE and NUS, demonstrate that, our proposed method
achieves a better spectral super-resolution quality than related
state-of-the-art methods.
Keywords—spectral super-resolution, BP neural network,
spectral classification, spectral mapping
I. I
NTRODUCTION
Hyperspectral (HS) images with tens or hundreds of
spectral bands can provide abundant spectral information, and
have been widely used in environment monitoring [1][2],
image classification [3][4], target detection [5][6] and so on.
However, the imaging process to achieve high spectral
resolution is at the expense of spatial resolution [7]. Compared
with HS images, RGB images usually have much higher
spatial resolution, but only have three spectral bands, and this
greatly limits its effectiveness in the above-mentioned
applications. Fortunately, the spectral information lost in RGB
image may be recovered using the relationship between RGB
and high-spatial-resolution hyperspectral (HHS) image-pairs
provided by some generalized image databases. In other
words, spectral super-resolution of RGB image is an
alternative way to obtain the HHS image, if we can establish
the spectral mapping from RGB to the hyperspectral spectral
bands, using a large number of RGB and HHS image-pair
samples.
Spectral super-resolution methods can be mainly divided
into two groups: dictionary learning based and neural network
based methods. Among the dictionary learning based methods,
Arad et al. [8] have proposed a sparse representation method
to obtain hyperspectral images from RGB images. Specifically,
a spectral dictionary for hyperspectral and the corresponding
RGB image-pairs is learned, using the hyperspectral image
priors provided by the K-singular value decomposition (K-
SVD) algorithm [9]. The sparse coefficients are estimated by
the greedy orthogonal matching pursuit (OMP) algorithm [10]
for the spectral dictionary learned above. To further improve
the quality of reconstructed HHS image, Aeschbacher et al.
[11] have re-implemented the above method [8] for better
accuracy and runtime, and also proposed a shallow learned
spectral reconstruction method based on the A+ method
proposed for fast spatial super-resolution [12]. The
comparable performance of [11] indicates its feasibility in the
spectral super-resolution.
In order to further accurately establish the spectral
mapping under a large number of RGB and HHS image-pair
samples, neural network based methods have been developed
very recently. Nguyen et al. [13] have proposed a radial basis
function network based method to reconstruct a hyperspectral
response from a single RGB image, with known spectral
response function. It is a nonlinear mapping with a white-
balancing process to reduce the effect of different illumination
conditions. Galliani et al. [14] have proposed a deep
convolutional neural network (CNN) method to learn an end-
to-end mapping from RGB images to hyperspectral images. It
has 56 layers, and can bring a better performance than that of
the dictionary learning based methods. Inspired by the above
methods, Can et al. [15] have proposed a rather shallow CNN
method with residual blocks to learn the spectral mapping
from RGB to HHS images. In addition, in order to increasing
the number of image-pair samples for a better learning result,
data augmentation [16] is also utilized, such as image rotating,
flipping and downscaling. The common thread of the above
CNN based methods is to establish the spectral mapping using
image patches, such as a patch size of 36×36 used in [15] and
a patch size of 64×64 used in [14], which means that the index
This work was supported in part by the National Natural Science
Foundation of China (61501008), and Beijing Natural Science Foundatio
(4172002).
978-1-5386-6602-9/18/$31.00 ©2018 IEEE
721