![](https://csdnimg.cn/release/download_crawler_static/12016486/bg3.jpg)
general discussion regarding our approach with pointers to
future work in Section 5. We conclude in Section 6.
2LITERATURE REVIEW
Semantic pixel-wise segmentation is an active topic of
research, fuelled by challenging datasets [20], [21], [22], [24],
[25]. Before the arrival of deep networks, the best performing
methods mostly relied on hand engineered features classify-
ing pixels independently. Typically, a patch is fed into a clas-
sifier, e.g., Random Forest [26], [27] or Boosting [28], [29] to
predict the class probabilities of the center pixel. Features
based on appearance [26] or SfM and appearance [27], [28],
[29] have been explored for the CamVid road scene under-
standing test [21]. These per-pixel noisy predictions (often
called unary terms) from the classifiers are then smoothed by
using a pair-wise or higher order CRF [28], [29] to improve
the accuracy. More recent approaches have aimed to pro-
duce high quality unaries by trying to predict the labels for
all the pixels in a patch as opposed to only the center pixel.
This improves the results of Random Forest based unaries
[30] but thin structured classes are classified poorly. Dense
depth maps computed from the CamVid video have also
been used as input for classification using Random Forests
[31]. Another approach argues for the use of a combination
of popular hand designed features and spatio-temporal
super-pixelization to obtain higher accuracy [32]. The best
performing technique on the CamVid test [29] addresses the
imbalance among label frequencies by combining object
detection outputs with classifier predictions in a CRF frame-
work. The result of all these techniques indicate the need for
improved features for classification.
Indoor RGBD pixel-wise semantic segmentation has also
gained popularity since the release of the NYU dataset [24].
This dataset showed the usefulness of the depth channel to
improve segmentation. Their approach used features such
as RGB-SIFT, depth-SIFT and pixel location as input to a
neural network classifier to predict pixel unaries. The noisy
unaries are then smoothed using a CRF. Improvements
were made using a richer feature set including LBP and
region segmentation to obtain higher accuracy [33] followed
by a CRF. In more recent work [24], both class segmentation
and support relationships are inferred together using a com-
bination of RGB and depth based cues. Another approach
focuses on real-time joint reconstruction and semantic seg-
mentation, where Random Forests are used as the classifier
[34]. Gupta et al. [35] use boundary detection and hierarchi-
cal grouping before performing category segmentation.
The common attribute in all these approaches is the use of
hand engineered features for classification of either RGB or
RGBD images.
The success of deep convolutional neural networks for
object classification has more recently led researchers to
exploit their feature learning capabilities for structured pre-
diction problems such as segmentation. There have also
been attempts to apply networks designed for object catego-
rization to segmentation, particularly by replicating the
deepest layer features in blocks to match image dimensions
[6], [36], [37], [38]. However, the resulting classification is
blocky [37]. Another approach using recurrent neural net-
works [39] merges several low resolution predictions to cre-
ate input image resolution predictions. These techniques
are already an improvement over hand engineered features
[6] but their ability to delineate boundaries is poor.
Newer deep architectures [2], [4], [9], [12], [17] particu-
larly designed for segmentation have advanced the state-of-
the-art by learning to decode or map low resolution image
representations to pixel-wise predictions. The encoder net-
work which produces these low resolution representations
in all of these architectures is the VGG16 classification net-
work [1] which has 13 convolutional layers and three fully
connected layers. This encoder network weights are typi-
cally pre-trained on the large ImageNet object classification
dataset [40]. The decoder network varies between these
architectures and is the part which is responsible for pro-
ducing multi-dimensional features for each pixel for
classification.
Each decoder in the Fully Convolutional Network archi-
tecture [2] learns to upsample its input feature map(s) and
combines them with the corresponding encoder feature
map to produce the input to the next decoder. It is an archi-
tecture which has a large number of trainable parameters in
the encoder network (134 M) but a very small decoder net-
work (0.5 M). The overall large size of this network makes it
hard to train end-to-end on a relevant task. Therefore, the
authors use a stage-wise training process. Here each
decoder in the decoder network is progressively added to
an existing trained network. The network is grown until no
further increase in performance is observed. This growth is
stopped after three decoders thus ignoring high resolution
feature maps can certainly lead to loss of edge information
[4]. Apart from training related issues, the need to reuse the
encoder feature maps in the decoder makes it memory
intensive in test time. We study this network in more detail
as it the core of other recent architectures [9], [10].
The predictive performance of FCN has been improved
further by appending the FCN with a recurrent neural net-
work (RNN) [9] and fine-tuning them on large datasets [20],
[41]. The RNN layers mimic the sharp boundary delineation
capabilities of CRFs while exploiting the feature representa-
tion power of FCN’s. They show a significant improvement
over FCN-8 but also show that this difference is reduced
when more training data is used to train FCN-8. The main
advantage of the CRF-RNN is revealed when it is jointly
trained with an architecture such as the FCN-8. The fact that
joint training helps is also shown in other recent results [42],
[43]. Interestingly, the deconvolutional network [4] per-
forms significantly better than FCN although at the cost of a
more complex training and inference. This however raises
the question as to whether the perceived advantage of the
CRF-RNN would be reduced as the core feed-forward seg-
mentation engine is made better. In any case, the CRF-RNN
network can be appended to any deep segmentation archi-
tecture including SegNet.
Multi-scale deep architectures are also being pursued
[12], [43]. They come in two flavours, (i) those which use
input images at a few scales and corresponding deep fea-
ture extraction networks, and (ii) those which combine fea-
ture maps from different layers of a single deep architecture
[10], [44]. The common idea is to use features extracted at
multiple scales to provide both local and global context [45]
and the using feature maps of the early encoding layers
retain more high frequency detail leading to sharper class
BADRINARAYANAN ET AL.: SEGNET: A DEEP CONVOLUTIONAL ENCODER-DECODER ARCHITECTURE FOR IMAGE SEGMENT ATION 2483