XGNN: Towards Model-Level Explanations of Graph
Neural Networks
Hao Yuan
hao.yuan@tamu.edu
Texas A&M University
College Station, Texas, United States
Jiliang Tang
tangjili@msu.edu
Michigan State University
East Lansing, Michigan, United States
Xia Hu
hu@cse.tamu.edu
Texas A&M University
College Station, Texas, United States
Shuiwang ji
sji@tamu.edu
Texas A&M University
College Station, Texas, United States
ABSTRACT
Graphs neural networks (GNNs) learn node features by aggregating
and combining neighbor information, which have achieved promis-
ing performance on many graph tasks. However, GNNs are mostly
treated as black-boxes and lack human intelligible explanations.
Thus, they cannot be fully trusted and used in certain application
domains if GNN models cannot be explained. In this work, we
propose a novel approach, known as XGNN, to interpret GNNs
at the model-level. Our approach can provide high-level insights
and generic understanding of how GNNs work. In particular, we
propose to explain GNNs by training a graph generator so that
the generated graph patterns maximize a certain prediction of the
model. We formulate the graph generation as a reinforcement learn-
ing task, where for each step, the graph generator predicts how to
add an edge into the current graph. The graph generator is trained
via a policy gradient method based on information from the trained
GNNs. In addition, we incorporate several graph rules to encour-
age the generated graphs to be valid. Experimental results on both
synthetic and real-world datasets show that our proposed methods
help understand and verify the trained GNNs. Furthermore, our
experimental results indicate that the generated graphs can provide
guidance on how to improve the trained GNNs.
CCS CONCEPTS
• Computing methodologies → Neural networks
; Articial in-
telligence; • Mathematics of computing → Graph algorithms.
KEYWORDS
Deep learning, Interpretability, Graph Neural Networks
ACM Reference Format:
Hao Yuan, Jiliang Tang, Xia Hu, and Shuiwang ji. 2020. XGNN: Towards
Model-Level Explanations of Graph Neural Networks. In Proceedings of the
26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for prot or commercial advantage and that copies bear this notice and the full citation
on the rst page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specic permission and/or a
fee. Request permissions from permissions@acm.org.
KDD ’20, August 23–27, 2020, Virtual Event, CA, USA
© 2020 Association for Computing Machinery.
ACM ISBN 978-1-4503-7998-4/20/08.. . $15.00
https://doi.org/10.1145/3394486.3403085
(KDD ’20), August 23–27, 2020, Virtual Event, CA, USA. ACM, New York, NY,
USA, 9 pages. https://doi.org/10.1145/3394486.3403085
1 INTRODUCTION
Graph Neural Networks (GNNs) have shown their eectiveness and
obtained the state-of-the-art performance on dierent graph tasks,
such as node classication [
11
,
37
], graph classication [
39
,
47
],
and link prediction [
46
]. In addition, extensive eorts have been
made towards dierent graph operations, such as graph convolu-
tion [
13
,
16
,
19
], graph pooling [
20
,
44
], and graph attention [
10
,
36
,
37
]. Since graph data widely exist in dierent real-world appli-
cations, such as social networks, chemistry, and biology, GNNs are
becoming increasingly important and useful. Despite their great
performance, GNNs share the same drawback as other deep learn-
ing models; that is, they are usually treated as black-boxes and
lack human-intelligible explanations. Without understanding and
verifying the inner working mechanisms, GNNs cannot be fully
trusted, which prevents their use in critical applications pertaining
to fairness, privacy, and safety [
7
,
40
]. For example, we can train a
GNN model to predict the eects of drugs where we treat each drug
as a molecular graph. Without exploring the working mechanisms,
we do not know what chemical groups in a molecular graph lead
to the predictions. Then we cannot verify whether the rules of
the GNN model are consistent with real-world chemical rules, and
hence we cannot fully trust the GNN model. This raises the need
of developing interpretation techniques for GNNs.
Recently, several interpretations techniques have been proposed
to explain deep learning models on image and text data. Depending
on what kind of interpretations are provided, existing techniques
can be categorized into example-level [
5
,
9
,
29
,
31
,
32
,
43
,
45
,
48
]
or model-level [
8
,
24
,
25
] methods. Example-level interpretations
explain the prediction for a given input example, by determin-
ing important features in the input or the decision procedure for
this input through the model. Common techniques in this cate-
gory include gradient-based methods [
31
,
32
,
43
], visualizations
of intermediate feature maps [
29
,
48
], and occlusion-based meth-
ods [
5
,
9
,
45
]. Instead of providing input-dependent explanations,
model-level interpretations aim to explain the general behavior of
the model by investigating what input patterns can lead to a certain
prediction, without respect to any specic input example. Input
optimization [
8
,
24
–
26
] is the most popular model-level interpreta-
tion method. These two categories of interpretation methods aim at
arXiv:2006.02587v1 [cs.LG] 3 Jun 2020