图神经网络与图核函数融合：图神经切线核
需积分: 0 151 浏览量
更新于2024-06-30
收藏 1.02MB PDF 举报
"本文介绍了Graph Neural Tangent Kernel (GNTK)，一种将图神经网络（GNN）与图核（GK）相结合的新方法，旨在融合两者的优点，提高图数据处理的性能。"
在计算机科学领域，特别是机器学习和数据挖掘中，图数据的处理是一个重要的研究方向。传统的图核方法提供了理论上的保证，并且易于训练，但它们的表现往往受限于手工设计的组合特征，这可能限制了它们对复杂图结构的理解。相反，图神经网络通过多层架构和非线性激活函数能够提取图的高阶信息，从而在实践中通常表现出更好的性能。然而，GNNs的训练过程具有大量的超参数和非凸优化问题，这使得训练变得困难，且其理论保证相对不足。
为了解决这些问题，该论文提出了Graph Neural Tangent Kernel。GNTK是基于无限宽的图神经网络，它利用神经网络的泰勒展开，将网络的权重视为无穷小，这样可以得到一个固定的、可解析的核函数。这种新的图核能够保留GNN的表达能力，同时避免了训练过程中非凸优化的问题，因为它们在无穷宽极限下可以进行封闭形式的分析。这使得GNTK在理论上更容易理解，并且可能更容易进行训练。
GNTK的提出为图数据的表示学习提供了一个新视角，它融合了图神经网络的高效特征学习和图核的理论稳定性。通过这种方式，GNTK能够在有限的计算资源下，更有效地利用GNN的参数，提高模型的表达能力和泛化能力。此外，由于GNTK的解析形式，它们为理解和分析GNN的行为提供了一种新的工具，这对于进一步研究GNN的理论性质和优化策略至关重要。
"2019-Graph Neural Tangent Kernel, Fusing Graph Neural Networks w"这篇论文引入了一种创新的图学习框架，结合了图神经网络和图核的优点，为图数据的深度学习带来了更强大且理论上可解析的方法。这种方法有望推动图数据处理领域的进步，尤其是在图分类、图聚类和图预测任务中。
READOUT Operation.
To get the representation of an entire graph
h
G
after
L
steps of
aggregation, we take the summation over all node features, i.e.,
h
G
= READOUT
n
h
(L)
u
, u ∈ V
o
=
X
u∈V
h
(L)
u
.
There are more sophisticated READOUT operations than a simple summation Xu et al. [2018],
Zhang et al. [2018a], Ying et al. [2018]. Jumping Knowledge Network (JK-Net) Xu et al. [2018]
considers graph structures of different granularity, and aggregates graph features across all layers as
h
G
= READOUT
JK
n
h
(`)
u
, u ∈ V, ` ∈ [L]
o
=
X
u∈V
h
h
(0)
u
; . . . ; h
(L)
u
i
. (3)
Building GNNs using BLOCK and READOUT.
Most modern GNNs are constructed using
the BLOCK operation and the READOUT operation Xu et al. [2019]. We denote the number
of
BLOCK
operations (aggregation steps) in a GNN by
L
. For each
` ∈
[
L
] and
u ∈ V
, we
define
h
(`)
u
=
BLOCK
(`)
(
u
). The graph-level feature is then
h
G
=
READOUT
n
h
(L)
u
, u ∈ V
o
or
h
G
=
READOUT
JK
n
h
(`)
u
, u ∈ V, ` ∈ [L]
o
, depending on whether jumping knowledge is applied
or not.
3 GNTK Formulas
In this section we present our general recipe which translates a GNN architecture to its corresponding
GNTK. We first provide some intuitions on neural tangent kernels (NTKs). We refer readers to
Jacot et al. [2018], Arora et al. [2019a] for more comprehensive descriptions.
3.1 Intuition of the Formulas
Consider a general neural network
f
(
θ, x
)
∈ R
where
θ ∈ R
m
is all the parameters in the network
and
x
is the input. Given a training dataset
{
(
x
i
, y
i
)
n
i=1
}
, consider training the neural network by
minimizing the squared loss over training data
`(θ) =
1
2
n
X
i=1
(f(θ, x
i
) −y
i
)
2
.
Suppose we minimize the squared loss
`
(
θ
) by gradient descent with infinitesimally small learning
rate, i.e.,
dθ(t)
dt
=
−∇`
(
θ
(
t
)). Let
u
(
t
) = (
f
(
θ, x
i
))
n
i=1
be the network outputs.
u
(
t
) follows the
evolution
du
dt
= −H(t)(u(t) − y),
where
H(t)
ij
=
∂f(θ(t), x
i
)
∂θ
,
∂f(θ(t), x
j
)
∂θ
for (i, j) ∈ [n] ×[n].
Recent advances in optimization of neural networks have shown, for sufficiently over-parameterized
neural networks, the matrix
H
(
t
) keeps almost unchanged during the training process Arora et al.
[2019b,a], Du et al. [2019, 2018], Jacot et al. [2018], in which case the training dynamics is identical
4
剩余16页未读，继续阅读
点击了解资源详情
点击了解资源详情
点击了解资源详情
2022-08-04 上传
2022-08-08 上传
2022-08-04 上传
2022-08-08 上传
2022-08-04 上传
2019-09-21 上传
天眼妹
- 粉丝: 29
- 资源: 332
我的内容管理
展开
