深度神经网络的噪声鲁棒性分析：视觉解释与防御策略

需积分: 0 172 浏览量更新于2024-08-05 收藏 3.73MB PDF 举报

本研究论文标题为《清华大学刘世霞团队于2018年在arXiv上发表的"可视化：深入分析深度神经网络的噪声鲁棒性"》。该文章关注的是深度神经网络（DNNs）在面对恶意生成的对抗性示例时的脆弱性。对抗性示例是通过微小但故意设计的扰动，使得原本应被正确分类的样本被误导，从而对深度学习模型的性能构成威胁。这种现象尤其在安全关键应用领域如自动驾驶中显得尤为重要，因为错误预测可能导致严重后果。研究的核心内容集中在分析深度神经网络如何处理这些对抗性示例，特别是在图像识别方面。作者发现，例如在熊猫图像的案例中，对抗性例子通过隐藏关键特征（如熊猫的眼睛）来欺骗DNN，导致模型无法正确检测到熊猫的脸（见图1）。图1(a)展示了原始输入图像，图(b)显示了在层级和特征图级别的数据路径可视化，揭示了模型在处理过程中的路径选择；而图(c)则是神经元可视化，直观地显示出模型对不同特征的响应。论文的主要贡献在于提出了一种方法来深入理解DNN在噪声环境下的行为，并探讨了其在误分类中的根源。通过对对抗性示例的详细剖析，作者希望能够揭示模型的弱点，从而为提高深度学习系统的鲁棒性和安全性提供理论依据。此外，论文可能还讨论了如何通过改进模型结构、优化训练策略或者开发防御机制来提升模型对对抗性噪声的抵抗能力。这篇论文提供了对深度神经网络在对抗性攻击下工作原理的洞察，对于提升人工智能系统的稳健性和防范潜在风险具有重要意义。同时，它也激发了后续的研究兴趣，尤其是在开发更强大的防御技术以及评估模型鲁棒性的方法上。

conv5_3..

conv5_3:

-0.0 199.9

Inputs

Datapath Extraction

Datapath Visualization

Noise

Normal image

Adversarial example

DNN

Panda

Monkey

Normal image

Adversarial example

Feature map levelLayer level Neuron level

Class 1

Class n

...

Panda

Monkey

Class 1

Class n

...

DNN

(a)

(b)

Pad

conv1

pool1

block1

block2

unit_1

unit_2

unit_3

unit_4

unit_5

unit_6

unit_7

unit_8

unit_9

unit_10..

unit_11..

unit_12..

unit_13..

unit_14..

unit_15..

unit_16..

unit_17..

unit_18..

unit_19..

unit_20..

unit_21..

unit_22..

unit_23..

preact

conv1

conv2

shortcu..

conv3

add

preact

conv1

conv2

conv3

add

preact

conv1

conv2

conv3

add

postnor..

pool5

logits

Spatial..

predict..

11/11 : giant pa..

11/11 : guenon, ..

resnet_v2_101/block4/unit_3/bo

ttleneck_v2/conv1:441

Activations

Learned feature

0.0 2.1

Figure 2: AEVis contains two modules: (a) a datapath extraction module and (b) a datapath visualization module that illustrates datapaths in

multiple levels: layer level, feature map level, and neuron level.

3 THE DESIGN OF AEVIS

3.1 Motivation

The development of AEVis is collaborated with the machine learning

team that won the ﬁrst place in the NIPS 2017 non-targeted adver-

sarial attack and targeted adversarial attack competitions, which aim

at attacking CNNs [

]. Despite their promising results, the

experts found that the research process was inefﬁcient and incon-

venient, especially the explanation of the model outputs. In their

research process, a central step is explaining misclassiﬁcation in-

troduced by adversarial examples. Understanding why an error has

been made helps the experts detect the weakness of the model and

further design a more effective attacking/defending approach. To

this end, they desire to understand the roles of the neurons and their

connections for prediction. Because there are millions of neurons in

a CNN, examining all neurons and their connections is prohibitive.

In the prediction of a set of examples, the experts usually

extract

and

examine

the critical neurons and their connections, which are

referred to as datapaths in their ﬁeld.

To extract datapaths, the experts often treat the most activated

neurons as the critical neurons [

]. However, they are not satisﬁed

with the current activation-based approach because it may result in

misleading results. For instance, considering an image with highly

recognizable secondary objects, which are mixed with the main

object in the image. The activations of the neurons that detect the

secondary objects are also large, however, the experts are not inter-

ested in them because these neurons are often irrelevant to the predic-

tion of the main object. Currently, the experts have to rely on their

knowledge to manually ignore these neurons in the analysis process.

After extracting datapaths, the experts examine them to under-

stand their roles for prediction. Currently, they utilize discrepancy

maps [

], heat maps [

], and weight visualization [

] to under-

stand the role of the datapaths. Although these methods can help

the experts at the neuron level, they commented that there lacked

an effective exploration mechanism enabling them to investigate the

extracted datapaths from high-level layers to individual neurons.

3.2 Requirement Analysis

To collect the requirements of our tool, we follow the human-

centered design process [

], which involves two experts (E

and E

) from the winning team of the NIPS 2017 competition. The

design process consists of several iterations. In each iteration, we

present the developed prototype to the experts, probe further re-

quirements, and modify our tool accordingly. We have identiﬁed

the following high-level requirements in this process. Among these

requirements,

and

are two initial requirements, while

and

R4 are gradually identiﬁed in the development.

R1 - Extracting the datapath for a set of examples of interest.

Both experts expressed the need for extracting the datapath of an

example, which serves as the basis for analyzing why an adversarial

example is misclassiﬁed. In a CNN, different neurons learn to

detect different features [

]. Thus, the roles of the neurons are

different for the prediction of an example. E

said that analyzing

the datapath can greatly save experts’ effort because they are able

to focus on the critical neurons instead of examining all neurons.

Besides the datapath for individual examples, E

emphasized the

need for extracting the common datapath for a set of examples of

the same class. He commented that the datapath of one example

sometimes is not representative for the image class. For example,

given an image of a panda’s face, the extracted datapath will probably

not include the neuron detecting the body of a panda, which is also

a very important feature to classify a panda.

R2 - Providing an overview of the datapath.

In a large CNN, a

datapath often contains millions of neurons and connections. Di-

rectly presenting all neurons in a datapath will induce severe visual

clutter. Thus, it is necessary to provide experts an overview of a

datapath. E

commented, “I cannot examine all the neurons in a dat-

apath because there are too many of them. In the examining process,

I often start by selecting an important layer based on my knowledge,

and examine the neurons in that layer to analyze the learned features

and the activations of these neurons. The problem of this method is

when dealing with a new architecture, I may not know which layer to

剩余11页未读，继续阅读

我就是月下

粉丝: 30
资源: 336

深度神经网络的噪声鲁棒性分析：视觉解释与防御策略

2018-arXiv-Graph Neural Networks A Review of Methods and Applications.pdf

[2018-arXiv].（多任务学习，谷歌）.Universal Sentence Encoder.v21

matlab求解二元一次方程组代码-symplectic-arxiv18a:用于2018arXiv论文的MATLAB代码讨论了用于稳定器代码的

nof1_arXiv:arXiv 的 N-of-1 审核

Markov-Lipschitz-Deep-Learning

python-deep-speech:Deep Speech 论文的 Python 实现

matlabfig生成代码-ScaDec-deep-learning-diffractive-tomography:深度学习有效且准确地反转多

matlab无线通信的代码-Reproducible-Deep-Learning-in-Communication:可复制的深度学习交流

数据融合matlab代码-deep-text-recog-list-repl:深层文字识别列表替换

arxiv2018-贝叶斯合奏

最新资源