窗内视雨滴：卷积神经网络去除雨滴影响的方法

systems, particularly the ones used in safety driving, out-door surveillance, intelligent vehicle systems and cameradrones; see e.g. [12,23]. There is certainly a need for devel-oping computational methods that can effectively removeraindrops from an image, i.e. restore the visual distortionscaused by the raindrops on images.The optical model of an image taken through windowwith raindrops is quite complex. For our purpose, we con-sider a simple linear model [23]:24630通过窗户看透雨滴的深度学习0全宇晖 1 ，邓世杰 1 ，陈溢新 1 ，季辉 201 华南理工大学计算机科学与工程学院，中国广州 510006 2 新加坡国立大学数学系，新加坡 1190760{ csyhquan@scut.edu.cn; shijie.deng.cs@foxmail.com; yx.chen.cs@foxmail.com; matjh@nus.edu.sg } �0摘要0在雨天通过玻璃窗拍照时，图像会受到附着在玻璃表面的雨滴的影响而变得模糊和损坏。从图像中去除雨滴的影响是一个具有挑战性的问题。关键任务是如何准确和稳健地识别图像中的雨滴区域。本文提出了一种用于从图像中去除雨滴影响的卷积神经网络（CNN）。在所提出的CNN中，我们引入了一个双重注意机制，同时使用基于形状的注意力和通道重新校准来指导CNN。基于形状的注意力利用雨滴的物理形状先验，即凸性和轮廓闭合性，来准确定位雨滴，而通道重新校准则提高了处理外观变化的雨滴时的鲁棒性。实验结果表明，所提出的CNN在定量指标和视觉质量方面优于现有方法。01. 引言0在雨天，有许多情况下需要通过玻璃窗或车窗拍摄室外场景的照片。由于这些场景中的玻璃窗被雨滴覆盖，所拍摄的图像受到这些雨滴造成的影响。在雨天拍摄室外时，相机镜头被雨滴覆盖也会出现同样的现象。雨滴对图像的影响会显著降低其可见性。请参见图1，以了解通过带有雨滴的窗户拍摄的真实图像的示例。除了视觉质量下降外，雨滴还对室外计算机视觉系统的性能产生负面影响，特别是在安全驾驶、室外监控、智能车辆系统和摄像机无人机等领域中使用的系统；参见[12,23]。因此，需要开发能够有效去除图像中雨滴的计算方法，即恢复雨滴对图像的视觉失真。通过窗户拍摄带有雨滴的图像的光学模型非常复杂。为了我们的目的，我们考虑一个简单的线性模型[23]：0*本研究得到了中国国家自然科学基金（61602184、61872151、U1611461、61672241、61702112）、广东省自然科学基金（2017A030313376）、中国中央高校基本科研业务费（x2js-D2181690）和新加坡教育部学术研究基金（R146000229114、MOE2017-T2-2-156）的支持。0I = (1 - A) ⊙ L + A ⊙ R，(1)0其中 ⊙ 表示逐元素相乘，I、L、R ∈ R C × M × N分别表示带有雨滴的图像、潜在无雨滴层和雨滴层。矩阵 A∈ [0, 1] C × M × N 表示透明度矩阵。A的每个元素表示相应像素的光路径被雨滴覆盖的百分比。由于潜在图像层 L 指的是玻璃后面的场景，我们也将 L称为背景图像/层。0(a) (b)0(c) (d)0图1：通过带有雨滴的窗户拍摄的图像示例。0不同的 A 值将导致不同类型的视觉失真。(c) Qian et al.’s [12](d) OursSingle-image methods.Instead of using multiple images,single-image methods only take one image as input. Theproblem of removing raindrops from a single image is muchmore challenging. The progress along this line has beenstagnated until the rapid progress of deep learning in recentyears. Eigen et al. [1] used a shallow convolutional neuralnetwork (CNN) with only three layers for the task. Thismethod does not work well on the images with large anddense raindrops, as shown in [12].Recently, Qian et al. [12] proposed a generative adver-sarial network (GAN) for raindrop removal. They traineda recurrent network with spatial attention mechanism forgenerating attentive raindrop masks from the input image.The generated raindrop mask is then used to help restoringthe latent image using a CNN with a discrimination loss.The performance of Qian et al.’s method highly dependson the availability of the high-quality raindrop masks thatare used as the ground-truths for training the neural net-work (NN). Nevertheless, it is very difficult to prepare suchtraining data. A dataset for training NN-based raindrop re-movers is also presented by Qian et al. [12], whose ground-truth raindrop masks are not very accurate owing to severalfactors, e.g. the binarization of the mask which leads to ac-curacy loss, and the imperfect alignment of rainy image andraindrop-free image. See Fig. 2 for an illustration of theresults from existing methods. There is certainly a lot of24640空间失真。考虑一个由 r 索引的像素。当 A(r) = 0时，该像素属于背景图像，没有雨滴的视觉效果；当 0