提升图卷积网络性能：批量虚拟对抗训练方法

需积分: 0 2 浏览量更新于2024-08-05 收藏 363KB PDF 举报

"2019年清华大学的研究论文《Batch Virtual Adversarial Training for Graph Convolutional Networks》(BatchVirtualAdversarialTrainingforGraphConvolutionalNetworks)由Zhijie Deng, Yinpeng Dong, 和 Jun Zhu三位作者共同完成，他们隶属于清华大学计算机科学与技术系、智能科技与系统国家重点实验室以及生物启发计算研究中心。这篇论文针对图卷积网络(GCNs)提出了一种创新的正则化方法，旨在解决GCNs在处理图结构数据时忽视模型输出分布平滑性的局限性。论文的核心是批量虚拟对抗训练(Batch Virtual Adversarial Training, BVAT)，这是一种针对GCNs设计的新颖策略。它认识到，对于图数据，模型的输出应当对输入附近的局部扰动具有一定的稳健性。为此，研究者提出了两种算法：基于采样的BVAT和优化驱动的BVAT。前者通过寻找远离彼此的节点的虚拟对抗扰动来增强模型的稳定性；后者则通过优化过程为所有节点生成虚拟对抗扰动，从而提升模型的全局性能。实验部分在三个著名的文献引用网络数据集（Cora, Citeseer, 和 Pubmed）以及一个知识图谱数据集Nell上进行了广泛的验证。结果表明，这种方法有效提高了模型的鲁棒性和性能，尤其是在半监督学习任务中，它显著超越了当时的最先进的方法，确立了在图数据处理领域的前沿地位。因此，Batch Virtual Adversarial Training为GCNs的稳定性和泛化能力提供了重要的改进途径，对于理解复杂网络结构的学习机制以及实际应用中的安全性和可靠性具有重要意义。"

Batch Virtual Adversarial Training for

Graph Convolutional Networks

Zhijie Deng, Yinpeng Dong and Jun Zhu

Dept. of Comp. Sci. & Tech., State Key Lab of Intell. Tech. & Sys., TNList Lab,

Center for Bio-Inspired Computing Research, Tsinghua University, Beijing, 100084, China

Abstract

We present batch virtual adversarial training (BVAT), a

novel regularization method for graph convolutional networks

(GCNs). BVAT addresses the shortcoming of GCNs that do

not consider the smoothness of the model’s output distribu-

tion against local perturbations around the input. We propose

two algorithms, sample-based BVAT and optimization-based

BVAT, which are suitable to promote the smoothness of the

model for graph-structured data by either ﬁnding virtual adver-

sarial perturbations for a subset of nodes far from each other

or generating virtual adversarial perturbations for all nodes

with an optimization process. Extensive experiments on three

citation network datasets Cora, Citeseer and Pubmed and a

knowledge graph dataset Nell validate the effectiveness of the

proposed method, which establishes state-of-the-art results in

the semi-supervised node classiﬁcation tasks.

1 Introduction

Recent models for graph-structured data (Kipf and Welling

2017; Hamilton, Ying, and Leskovec 2017; Veli

ckovi

c et

2018) demonstrate remarkable performance in semi-

supervised node classiﬁcation tasks. These methods essen-

tially adopt different aggregators to aggregate feature infor-

mation from the neighborhood of a node. The aggregators

take the connectivity patterns and node features into con-

sideration and enable the information propagation through

edges (e.g., the gradient information can be distributed from

nodes with labels to nodes without labels). By aggregating

the features of a node with its nearby neighbors, the mod-

els can promote the smoothness between nodes in a neigh-

borhood, which is helpful for semi-supervised node classi-

ﬁcation based on the assumption that connected nodes in

the graph are likely to have similar representations (Kipf

and Welling 2017). However, these methods only consider

the smoothness between nodes in a neighborhood without

considering the smoothness of the output distribution. Sev-

eral studies have conﬁrmed that smoothing the output dis-

tribution of neural networks (i.e., encourage the neural net-

works to produce similar outputs) against local perturba-

tions around the input can improve the generalization perfor-

mance in supervised and especially semi-supervised learn-

ing (Wager, Wang, and Liang 2013; Sajjadi, Javanmardi, and

Tasdizen 2016; Laine and Aila 2017; Miyato et al

2017;

Luo et al

2018). Moreover, it’s crucial to encourage the

smoothness of the output distribution for aggregator-based

graph models since that the receptive ﬁeld (e.g., Fig. 1a)

of a single node grows exponentially with respect to the

number of aggregators in a model (Chen and Zhu 2017), and

neural networks tend to be non-smooth with such high dimen-

sional input space (Goodfellow, Shlens, and Szegedy 2015;

Peck et al

2017). Therefore, it is necessary for aggregator-

based graph models to encourage the

smoothness of their

output distribution

in addition to the smoothness in the

neighborhood.

In this work, we focus on graph convolutional networks

(GCNs) (Kipf and Welling 2017), a typical and effective

instance of aggregator-based graph learning models, and

the proposed algorithms are generally applicable for other

models such as graph attention networks (Veli

ckovi

c et al

2018), GraphSAGE (Hamilton, Ying, and Leskovec 2017),

etc. GCN model has ﬁxed aggregators based on the adjacency

matrix of input graph and is a special form of Laplacian

smoothing (Li, Han, and Wu 2018). Considering an undi-

rected graph

G = (V, E)

with

N = |V|

nodes and

|E|

edges,

GCNs are built upon the symmetric sparse adjacency matrix

A ∈ R

N×N

. The input feature matrix of all nodes in

denoted as

X ∈ R

N×D

where

is the feature dimension

of each node. GCNs use a normalized version of

as the

propagation matrix P

A = A + I,

and P =

−

In a

-layer GCN, each layer aggregates and processes infor-

mation from itself and its nearby neighbors with the following

propagation rule

(0)

= X, H

(l+1)

= σ(P H

(l)

), (1)

where

(l)

is the activation matrix of the

-th layer,

(l)

the trainable transformation matrix and

denotes an activa-

tion function. Particularly, the prediction of a node

depends

on the input features of nodes in its receptive ﬁeld (as shown

in Fig. 1a), whose dimension grows exponentially with re-

spect to the number of layers. The GCN model tends to be

non-smooth with high dimensional input space, so it’s neces-

sary to add regularization terms to promote the smoothness

GCNs represent a directed graph as an undirected bipartite

graph with additional nodes that represent edges in the original

graph.

arXiv:1902.09192v1 [cs.LG] 25 Feb 2019

下载后可阅读完整内容，剩余8页未读，立即下载

一曲歌长安

粉丝: 221
资源: 302

提升图卷积网络性能：批量虚拟对抗训练方法

The code for Distortion-Convolution-Module-for-Semantic-Segmenta

Once-for-All-Adversarial-Training:[NeurIPS 2020]“一次全面的对抗训练

load data inpath'/logs_batch/logs/topic_start/2019-06-09'into table hive_batch.ods_start_log partition(dt='2019-06-09'); -bash: syntax error near unexpected token `('

load data inpath '/logs_batch/logs/topic_start/2019-06-09' into table hive_batch.ods_start_log partition(dt='2019-06-09');

为什么对ResNet34模型单独使用large-batch training, learning rate warmup, no bias decay, low-precision training和共同使用large-batch training与learning rate warmup和large-batch training与low-precision training时模型性能会得到提升，

resnet恒等模块和卷积残差块的原理

image-to-Image Translation with Conditional Adversarial Networks

Error: Error while compiling statement: FAILED: SemanticException Line 1:17 Invalid path ''/logs_batch/logs/topic_start/2019-06-09'': No files matching path hdfs://hacluster/logs_batch/logs/topic_start/2019-06-09 (state=42000,code=40000)

最新资源