图神经网络在NLP中的应用：一份全面的127页综述

需积分: 49 107 浏览量更新于2024-07-09 收藏 2.86MB PDF 举报

"这篇综述详细探讨了图神经网络（GNN）在自然语言处理（NLP）中的应用，提供了一本127页的详细文档，旨在全面介绍这一领域的最新进展和技术。近年来，深度学习已成为解决NLP任务的关键手段，而GNN则通过将文本数据转换为图形结构，有效地捕捉文本的结构信息，从而在处理各种NLP问题上展现出潜力。文章由来自不同研究机构的专家撰写，涵盖了GNN在NLP任务中的发展、技术和挑战，以及未来的研究方向。" 本文主要关注的是如何利用图神经网络在自然语言处理中的优势。传统的NLP方法通常将文本视为一串顺序的单词或标记，但这种方式往往无法充分捕获文本内在的复杂关系。图神经网络的出现，为解决这个问题提供了新的思路。GNN能够将文本中的实体、概念和其他元素表示为节点，它们之间的关系作为边，构建出一个能够反映语义和结构信息的图模型。 GNN的工作原理是通过在图中传播和聚合信息，每个节点的特征会与其邻居节点的特征进行交互，从而更新自身的表示。经过多轮迭代，每个节点可以获得全局上下文的信息，使得整个图的表示更具有表达力。这种特性使得GNN在诸如情感分析、句法分析、问答系统、知识图谱建模、机器翻译等NLP任务中表现优越。在NLP中，GNN的应用可以分为几个主要类别：文本分类与情感分析，其中GNN能够捕捉句子的依赖关系；句法分析，通过构建依存树形图来理解句子结构；实体识别与关系抽取，构建实体网络以识别实体及其相互关系；问答系统，构建图以捕获问题和文档之间的关联；以及知识图谱的构建与推理，GNN可以帮助发现和推理实体之间的复杂关系。尽管GNN在NLP领域展现出了强大的能力，但也面临着一些挑战，如模型的解释性、计算效率、图的构建和预处理、以及泛化能力等。此外，如何有效地结合传统NLP方法与深度学习，以及如何利用预训练模型提升GNN的性能，也是当前研究的重点。随着技术的发展，GNN在NLP中的应用将继续深入，未来可能会有更多创新性的架构和算法出现，以应对更加复杂的语言理解和生成任务。这篇综述为读者提供了一个全面的视角，以理解GNN在NLP中的核心思想、最新成果，并为相关领域的研究者提供了宝贵的参考。

LINGFEI WU AND YU CHEN, ET AL.

sentences). The nodes set can be denoted as: 1) non-terminal nodes set V

(e.g. S and NP) and 2)

terminal nodes set V

words

. The constituency relation set are associated with the tree’s edges, which

can be denoted as R

cons

⊆ V

× (V

+ V

words

Step 2: Constituency Graph. A constituency graph G(V, E) consists of both the non-terminal nodes

and the terminal nodes V

words

, and the constituency edges as well as the sequential edges.

Similar to the dependency graph, given a paragraph para and the constituency relation set R

cons

for each constituency relation (w

, rel

i,j

, w

) ∈ R

cons

, one adds the nodes v

(for the word w

) and

(for the word w

) and a directed edge from node v

to node v

. And then for each word nodes pair

, v

) for the words which are adjacent in the original text, one adds an undirected edge between

them with the speciﬁc sequential type. These sequential edges are used to reserve the sequential

information (Li et al., 2020a; Xu et al., 2018c).

name

person

describe-01

“Paul”

fighter

:ARG2

:ARG1

:ARG0 :name :op1

Figure 3: An example of AMR graph, the original sentence is ”Pual’s description of himself: a

ﬁghter”.

AMR Graph Construction The AMR graphs are rooted, labeled, directed, acyclic graphs, which

are widely used to represent the high-level semantic relations between abstract concepts of the

unstructured and concrete natural text. Different from the syntactic idiosyncrasies, the AMR is

the high-level semantic abstraction. More concretely, the different sentences that are semantically

similar may share the same AMR parsing results, e.g., ”Paul described himself as a ﬁghter” and

”Paul’s description of himself: a ﬁghter”, as shown in Fig. 3. Despise the fact that the AMR is biased

toward English, it is a powerful auxiliary representation for linguistic analysis (Song et al., 2018e;

Damonte and Cohen, 2019; Wang et al., 2020f). Similar to the previously introduced dependency

and constituency trees, an AMR graph is derived from an AMR parsing tree. Next, we focus on

introducing the general procedure of constructing the AMR graph based on the AMR parsing tree.

We will discuss the basic concept of AMR relation and then show how to convert the relations into

an AMR graph.

Step 1: AMR Relations. Conceptually, there are two types of nodes in the AMR parsing tree: 1)

the name (e.g. ”Paul”) is the speciﬁc value of the node instance and 2) the concepts are either

English words (e.g. ”boy”), PropBank framesets (Kingsbury and Palmer, 2002) (e.g. ”want-01”), or

special keywords. The name nodes are the unique identities, while the concept nodes are shared by

different instances. The edges that connect nodes are called relations (e.g. :ARG0 and :name). One

may extract these AMR relations from the node pairs with edges, which is denoted as (n

, r

i,j

, n

) ∈

amr

Step 2: AMR Graph. The AMR graph G(V, E), which is rooted, labeled, directed, acyclic graph

(DAG), consists of the AMR nodes and AMR relations discussed above. Similar to the dependency

and constituency graphs, given the sentence sent and the AMR relation set R

amr

, for each relation

GRAPH NEURAL NETWORKS FOR NLP: A SURVEY

, r

i,j

, n

) ∈ R

amr

, one adds the nodes v

(for the AMR node n

) and v

(for the AMR node n

)

and add a directed edge from node v

to node v

with edge type r

i,j

Text input: Paul, a renowned

computer scientist, grew up in

Seattle. He attended Lakeside

School.

grew up in

attended

Paul

a renowned …

Seattle

Lakeside

School

Coreference

OpenIE

Figure 4: An example for IE graph construction which contains both the Co-reference process and

the Open Information Extraction process.

Information Extraction Graph Construction The information extraction graph (IE Graph) aims

to extract the structural information to represent the high-level information among natural sentences,

e.g., text-based documents. These extracted relations that capture relations across distant sentences

have been demonstrated helpful in many NLP tasks (Wu et al., 2020c; Vashishth et al., 2018; Gupta

et al., 2019). In what follows, we will discuss the technical details on how to construct an IE graph

for a given paragraph para (Huang et al., 2020b; Vashishth et al., 2018). We divide this process

into two three basic steps: 1) coreference resolution, 2) constructing IE relations, and 3) graph

construction.

Step 1: Coreference Resolution. Coreference resolution is the basic procedure for informa-

tion extraction task which aims to ﬁnd expressions that refer to the same entities in the text se-

quence (Huang et al., 2020b). As shown in Figure 4, the name ”Pual”, the noun-term ”He” and ”a

renowned computer scientist” may refer to the same object (person). Many NLP tools such as Ope-

nIE (Angeli et al., 2015) provide coreference resolution function to achieve this goal. We denotes

the coreference cluster C as a set of phrases referring to the same object. Given a paragraph, one

can obtain the coreference sets C = {C

, C

, ..., C

} extracting from unstructured data.

Step 2: IE Relations. To construct an IE graph, the ﬁrst step is to extract the triples from the para-

graphs, which could be completed by leveraging some well-known information extraction systems

(i.e. OpenIE (Angeli et al., 2015)). We call each triple (subject, predicate, object) as a relation,

which is denoted (n

, r

i,j

, n

) ∈ R

. It is worth noting if two triples differ only by one argument,

and the other arguments overlap, one only keep the longer triple.

Step 3: IE Graph Construction. The IE graph G(V, E) consists of the IE nodes and IE rela-

tions discussed above. Given the paragraph para and the IE relation set R

, for each relation

, r

i,j

, n

) ∈ R

, one adds the nodes v

(for the subject n

) and v

(for the object n

) and add a

directed edge from node v

to node v

with the corresponding predicate types (Huang et al., 2020b).

And then, for each coreference cluster C

∈ C, one may collapse all coreferential phrases in C

into one node. This could help greatly reduce the number of nodes and eliminate the ambiguity by

keeping only one node.

Discourse Graph Construction Many NLP tasks suffer from long dependency challenge when

the candidate document is too long. The discourse graph, which describes how two sentences are

logically connected to one another, are proved effective to tackle such challenge (Christensen et al.,

2013). In the following subsection, we will brieﬂy discuss the discourse relations between given

LINGFEI WU AND YU CHEN, ET AL.

sentences and then introduce the general procedure to construct the discourse graphs (Song et al.,

2018d; Li et al., 2020b; Yasunaga et al., 2017; Xu et al., 2020a).

Step 1: Discourse Relation. The discourse relations derive from the discourse analysis, which aims

to identify sentence-wise ordering constraints over a set of sentences. Given two sentences sent

and sent

, one can deﬁne the discourse relation as (sent

, sent

), which represents the discourse

relation ”sentence sent

can be placed after sentence sent

.” The discourse analysis has been ex-

plored for years, and many theories have been developed for modeling discourse relations such

as the Rhetorical Structure Theory (RST) (Mann and Thompson, 1987) and G-Flow (Christensen

et al., 2013). In many NLP tasks, given a document doc, one ﬁrstly segments doc into sentences set

V = {sent

, sent

, ..., sent

}. Then one applies discourse analysis to get the pairwise discourse

relation set denoted as R

sep

⊆ V × V.

Step 2: Discourse Graph. The discourse graph G(V, E) consists of the sentences nodes and dis-

course relations discussed above. Given the document doc and the discourse relation set R

dis

, for

each relation (sent

, sent

) ∈ R

dis

, one adds the nodes v

(for the sentence sent

) and v

(for the

sentence sent

) and add a directed edge from node v

to node v

Question: who acted in the movies directed by the director of [Some Mother's Son]

Answer: Don Cheadle, Joaquin Phoenix

[Some Mother's Son]

Hotel Rwanda

Ter ry G eorge

Reservation Road

directed_by

starred_actors

Don Cheadle

starred_actors

Joaquin Phoenix

Get the concept sub-

graph from KB

directed_by

Figure 5: An example for knowledge graph construction, where the knowledge base (KB) used and

the generated concept graph are both from the dataset MetaQA (Zhang et al., 2018b).

Knowledge Graph Construction Knowledge Graph (KG) that captures entities and relations

can greatly facilitate learning and reasoning in many NLP applications. In general, the KGs can be

divided into two main categories depending on their graph construction approaches. Many applica-

tions treat the KG as the compact and interpretable intermediate representation of the unstructured

data (e.g., the document) (Wu et al., 2020c; Koncel-Kedziorski et al., 2019; Huang et al., 2020b).

Conceptually, it is almost similar to the IE graph, which we have discussed previously. On the

other hand, many other works (Wu et al., 2020b; Ye et al., 2019; Bansal et al., 2019; Yang et al.,

2019) incorporate the existing knowledge bases such as YAGO (Suchanek et al., 2008)) and Con-

ceptNet (Speer et al., 2017) to further enhance the performance of downstream tasks (Zhao et al.,

2020c). In what follows, we will brieﬂy discuss the second category of KG from the view of the

graph construction.

The KG can be denoted as G(V, E), which is usually constructed by elements in knowledge

base. Formally, one deﬁnes the triple (e

, rel, e

) as the basic elements in the knowledge base, in

GRAPH NEURAL NETWORKS FOR NLP: A SURVEY

which e

is the source entity, e

is the target entity, and rel is the relation type. Then one adds two

nodes v

(for the source element e

) and v

(for the target element e

) in the KG and add a directed

edge from node v

to node v

with edge type rel. An example of such KG is shown in Fig. 5.

It is worth noting that the KG plays different roles in various applications. In some applications

(e.g. knowledge graph completion and knowledge base question answering), KG is always treated

as part of the inputs. In this scenario (Ye et al., 2019; Zhang et al., 2020g; Li et al., 2019; Wu

et al., 2020a), researchers typically use the whole KG G as the learning object. But for some other

applications (e.g. natural language translation), the KG can be treated as the data augmentation

method. In this case, the whole KG such as ConceptNet (Speer et al., 2017) is usually too large and

noisy for some domain-speciﬁc applications (Kapanipathi et al., 2020; Lin et al., 2019a), and thus

it is not suitable to use the whole graph as inputs. In contrast, as shown in Figure 5, one instead

usually constructs subgraphs from the given query (it is often the text-based inputs like the queries

in the reading comprehension task) (Xu et al., 2019b; Teru et al., 2020; Kapanipathi et al., 2020).

The construction methods could may vary dramatically in the literature. Here, we only present

one representative method for illustration purpose (Teru et al., 2020). The ﬁrst thing for constructing

KG is to fetch the term instances in the given query. Then, they could link the term instances to the

concepts in the KG by some matching algorithms such as max-substring matching. The concepts

are regarded as the initial nodes in the extracted subgraph. Next step is to fetch the 1-hop neighbors

of the initial nodes in the KG. Additionally, one may calculate the relevance of the neighbors with

the initial nodes by applying some graph node relevance model such as the Personalized PageRank

(PPR) algorithm (Page et al., 1999). Then based on the results, one may further prune out the edges

with relevance score that is below the conﬁdence threshold and remove the isolated neighbors. The

remaining ﬁnal subgraph is then used to feed any graph representation learning module later.

Coreference Graph Construction In linguistics, coreference (or co-reference) occurs when two

or more terms in a given paragraph refer to the same object. Many works have demonstrated that

such phenomenon is helpful for better understanding the complex structure and logic of the corpus

and resolve the ambiguities (Xu et al., 2020c; De Cao et al., 2018; Sahu et al., 2019). To effec-

tively leverage the coreference information, the coreference graph is constructed to explicitly model

the implicit coreference relations. Given a set of phrases, a coreference graph can link the nodes

(phrases) which refer to the same entity in the text corpus. In the following subsection, we focus on

the coreference graph construction for a paragraph para consisting of m sentences. We will brieﬂy

discuss the coreference relation and then discuss the approaches for building the coreference graph

in various NLP tasks (De Cao et al., 2018; Sahu et al., 2019; Qian et al., 2019; Xu et al., 2020c,a;

Luan et al., 2019). It is worth noting that although it is similar to the IE graph’s ﬁrst step, the coref-

erence graph will explicitly model the coreference relationship by graph instead of collapse into one

node.

Step 1: Coreference Relation. The coreference relations can be obtained easily by the coreference

resolution system, as discussed in IE graph construction. Similarly, we can obtain the coreference

clusters C given a speciﬁc paragraph. All phrases in a cluster C

∈ C refer to the same object.

Step 2: Coreference Graph. The coreference graph are built on the coreference relation set R

coref

It can be generally divided into two main category depending on the node type: 1) phrases (or

mentions) (Koncel-Kedziorski et al., 2019; Luan et al., 2019; De Cao et al., 2018), 2) words (Sahu

et al., 2019). For the ﬁrst class, the coreference graph G consists of all mentions in relation set

coref

. For each phrase pair p

, p

in cluster C

∈ C, one may add an undirected edge between

LINGFEI WU AND YU CHEN, ET AL.

node v

(for the phrase p

) and node v

(for the phrase p

). For the second case, the coreference

graph G consists of words. One minor difference is that one only links the ﬁrst word of each phrase

for each associated phrases.

Sentence

TF-IDF vector

Sentence

souness backs smith

for scotland

uk will stand firm on

eu rebate

qpr keeper day heads

for preston

former ni minister

scott dies

Cranes: Flying giant

returning to Ireland

after 300 years

Figure 6: An example for similarity graph construction. We use sentences as nodes and initialize

their features with TF-IDF vectors. Larger similarity values correspond to thicker dashed lines.

Similarity Graph Construction The similarity graphs aim to quantify the similarity between

nodes, which are widely used in many NLP tasks (Liu et al., 2019a; Linmei et al., 2019; Yasunaga

et al., 2017). Since the similarity graph is typically application-oriented, we focus on the basic pro-

cedure of constructing the similarity graph for various types of elements such as entities, sentences

and documents, and neglect the application speciﬁc details. It is worth noting that the similarity

graph construction is conducted during preprocessing and is not jointly trained with the remaining

learning system in an end-to-end manner. One example of similarity graph is shown in Fig. 6.

Step 1: Similarity Graph. Given a corpus C, in a similarity graph G(V, E), the graph nodes can

be deﬁned in different granularity levels such as entities, sentences and documents. We denote the

basic node set as V regardless of speciﬁc node types. One can calculate the node features by various

mechanisms such as TF-IDF for sentences (or documents) (Liu et al., 2019a; Yasunaga et al., 2017)

and embeddings for entities (Linmei et al., 2019). Then, similarity scores between node pairs can

be computed by various metrics such as cosine similarity (Liu et al., 2019a; Linmei et al., 2019;

Yasunaga et al., 2017), and used to indicate edge weights of the node pairs.

Step 2: Sparse mechanism. The initial similarity graph is typically dense even some edge weights

are very small or even negative. These values can be treated as noise, which plays little roles in

the similarity graph. Thus various sparse techniques are proposed to further improve the quality

of graph by sparsifying a graph. One widely used sparse method is k-NN (Liu et al., 2019a).

Speciﬁcally, for node v

and its’ neighbor set N(v

), one only reserves edges by keeping k largest

edge weights and dropping the remaining edges. The other widely used method is −sparse (Linmei

et al., 2019; Yasunaga et al., 2017). In particular, one will remove the edges whose weights are

smaller than the certain threshold .

剩余126页未读，继续阅读

syp_net

粉丝: 158
资源: 1187

图神经网络在NLP中的应用：一份全面的127页综述

图神经网络在自然语言处理中的应用

GNN4NLP-Papers:近期关于图神经网络方法应用于 NLP 领域的论文列表

自然语言生成技术综述：从规则到神经网络

"基于神经网络的文档特征提取方法在系统评价中的应用

基于Keras的图神经网络处理库

Spark NLP：分布式自然语言处理库

文本摘要领域研究综述及趋势

深入浅出图神经网络:gnn原理解析 pdf

图神经网络未来的发展趋势

图神经网络的未来发展方向

最新资源