![](https://csdnimg.cn/release/download_crawler_static/86324375/bg1.jpg)
Batch Virtual Adversarial Training for
Graph Convolutional Networks
Zhijie Deng, Yinpeng Dong and Jun Zhu
Dept. of Comp. Sci. & Tech., State Key Lab of Intell. Tech. & Sys., TNList Lab,
Center for Bio-Inspired Computing Research, Tsinghua University, Beijing, 100084, China
Abstract
We present batch virtual adversarial training (BVAT), a
novel regularization method for graph convolutional networks
(GCNs). BVAT addresses the shortcoming of GCNs that do
not consider the smoothness of the model’s output distribu-
tion against local perturbations around the input. We propose
two algorithms, sample-based BVAT and optimization-based
BVAT, which are suitable to promote the smoothness of the
model for graph-structured data by either finding virtual adver-
sarial perturbations for a subset of nodes far from each other
or generating virtual adversarial perturbations for all nodes
with an optimization process. Extensive experiments on three
citation network datasets Cora, Citeseer and Pubmed and a
knowledge graph dataset Nell validate the effectiveness of the
proposed method, which establishes state-of-the-art results in
the semi-supervised node classification tasks.
1 Introduction
Recent models for graph-structured data (Kipf and Welling
2017; Hamilton, Ying, and Leskovec 2017; Veli
ˇ
ckovi
´
c et
al
.
2018) demonstrate remarkable performance in semi-
supervised node classification tasks. These methods essen-
tially adopt different aggregators to aggregate feature infor-
mation from the neighborhood of a node. The aggregators
take the connectivity patterns and node features into con-
sideration and enable the information propagation through
edges (e.g., the gradient information can be distributed from
nodes with labels to nodes without labels). By aggregating
the features of a node with its nearby neighbors, the mod-
els can promote the smoothness between nodes in a neigh-
borhood, which is helpful for semi-supervised node classi-
fication based on the assumption that connected nodes in
the graph are likely to have similar representations (Kipf
and Welling 2017). However, these methods only consider
the smoothness between nodes in a neighborhood without
considering the smoothness of the output distribution. Sev-
eral studies have confirmed that smoothing the output dis-
tribution of neural networks (i.e., encourage the neural net-
works to produce similar outputs) against local perturba-
tions around the input can improve the generalization perfor-
mance in supervised and especially semi-supervised learn-
ing (Wager, Wang, and Liang 2013; Sajjadi, Javanmardi, and
Tasdizen 2016; Laine and Aila 2017; Miyato et al
.
2017;
Luo et al
.
2018). Moreover, it’s crucial to encourage the
smoothness of the output distribution for aggregator-based
graph models since that the receptive field (e.g., Fig. 1a)
of a single node grows exponentially with respect to the
number of aggregators in a model (Chen and Zhu 2017), and
neural networks tend to be non-smooth with such high dimen-
sional input space (Goodfellow, Shlens, and Szegedy 2015;
Peck et al
.
2017). Therefore, it is necessary for aggregator-
based graph models to encourage the
smoothness of their
output distribution
in addition to the smoothness in the
neighborhood.
In this work, we focus on graph convolutional networks
(GCNs) (Kipf and Welling 2017), a typical and effective
instance of aggregator-based graph learning models, and
the proposed algorithms are generally applicable for other
models such as graph attention networks (Veli
ˇ
ckovi
´
c et al
.
2018), GraphSAGE (Hamilton, Ying, and Leskovec 2017),
etc. GCN model has fixed aggregators based on the adjacency
matrix of input graph and is a special form of Laplacian
smoothing (Li, Han, and Wu 2018). Considering an undi-
rected graph
1
G = (V, E)
with
N = |V|
nodes and
|E|
edges,
GCNs are built upon the symmetric sparse adjacency matrix
A ∈ R
N×N
. The input feature matrix of all nodes in
V
is
denoted as
X ∈ R
N×D
where
D
is the feature dimension
of each node. GCNs use a normalized version of
A
as the
propagation matrix P
˜
A = A + I,
˜
D
ii
=
X
j
˜
A
ij
and P =
˜
D
−
1
2
˜
A
˜
D
−
1
2
.
In a
K
-layer GCN, each layer aggregates and processes infor-
mation from itself and its nearby neighbors with the following
propagation rule
H
(0)
= X, H
(l+1)
= σ(P H
(l)
W
(l)
), (1)
where
H
(l)
is the activation matrix of the
l
-th layer,
W
(l)
is
the trainable transformation matrix and
σ
denotes an activa-
tion function. Particularly, the prediction of a node
u
depends
on the input features of nodes in its receptive field (as shown
in Fig. 1a), whose dimension grows exponentially with re-
spect to the number of layers. The GCN model tends to be
non-smooth with high dimensional input space, so it’s neces-
sary to add regularization terms to promote the smoothness
1
GCNs represent a directed graph as an undirected bipartite
graph with additional nodes that represent edges in the original
graph.
arXiv:1902.09192v1 [cs.LG] 25 Feb 2019