Video Rain Streak Removal By Multiscale Convolutional Sparse Coding
Minghan Li
1
, Qi Xie
1
, Qian Zhao
1
, Wei Wei
1
, Shuhang Gu
2
, Jing Tao
1
, Deyu Meng
1 ∗
1
National Engineering Laboratory for Algorithm and Analysis Technologiy on Big Data and Ministry
of Education Key Lab of Intelligent Networks and Network Security, Xian Jiaotong University
2
computer vision lab, eth, zurich
{liminghan, xq.liwu}@stu.xjtu.edu.cn, timmy.zhaoqian@gmail.com,
weiweiwe@stu.xjtu.edu.cn, shuhanggu@gmail.com,{jtao, dymeng}@mail.xjtu.edu.cn
Abstract
Videos captured by outdoor surveillance equipments
sometimes contain unexpected rain streaks, which brings d-
ifficulty in subsequent video processing tasks. Rain streak
removal from a video is thus an important topic in recent
computer vision research. In this paper, we raise two in-
trinsic characteristics specifically possessed by rain streak-
s. Firstly, the rain streaks in a video contain repetitive local
patterns sparsely scattered over different positions of the
video. Secondly, the rain streaks are with multiscale config-
urations due to their occurrence on positions with different
distances to the cameras. Based on such understanding, we
specifically formulate both characteristics into a multiscale
convolutional sparse coding (MS-CSC) model for the video
rain streak removal task. Specifically, we use multiple con-
volutional filters convolved on the sparse feature maps to
deliver the former characteristic, and further use multiscale
filters to represent different scales of rain streaks. Such a
new encoding manner makes the proposed method capable
of properly extracting rain streaks from videos, thus getting
fine video deraining effects. Experiments implemented on
synthetic and real videos verify the superiority of the pro-
posed method, as compared with the state-of-the-art ones
along this research line, both visually and quantitatively.
1. Introduction
Rainy videos captured by outdoor surveillance equip-
ments may degenerate the performance of subsequent video
processing tasks, like human detection [8], person re-
identification [10], stereo correspondence [14], object track-
ing and recognition [29], and scene analysis [19]. Thus, re-
moving rain streaks from a video is an important issue and
has attracted much attention in computer vision.
Since first raised by Garg and Nayar [12] in 2004, many
∗
Deyu Meng is the corresponding author.
Background
Rain streaks
Moving objects
Rain layer 3
Map 3
5 5
Filter3
Input video
(a)
(b)
⊗
=
Rain layer 2
Map 2
9 9
Filter2
Map 1
Rain layer 1
13 13
Filter1
=
⊗
×
=
⊗
×
×
Figure 1. An natural rainy video (upper) is separated into three
layers (middle) of background scene, rain streaks and moving ob-
jects by the proposed multiscale convolutional sparse coding (MS-
CSC) model. The rain streaks can be decomposed into diverse rain
structures (lower row (a)), corresponding to different scales of rain
appearance. All those decompositions are attained through three
scale filters convolved on sparse feature maps (lower row (b)).
methods have been proposed for this task and attained good
performance under different rain circumstances. Many of
these methods implement the task by carefully formulating
certain physical characteristics of rain steaks, e.g. photo-
metric appearance [13], geometrical features [30], chromat-
ic consistency [25], spatio-temporal configurations [34], lo-
cal structure correlations [7], and design certain techniques
for quantitatively formulating these prior rain knowledge
to facilitate a proper separation of rain streaks from the
video background [20]. Some recent methods along this
line achieve decent performance, by pre-training a discrimi-
nator with some pre-annotated sample pairs, with or without
1
6644