图LSTM结合上下文门控机制提升语言理解

6 浏览量更新于2024-08-26 收藏 1.27MB PDF 举报

"本文研究了一种新的深度学习模型——具有上下文门控机制的图LSTM，用于提升语音语言理解（Spoken Language Understanding, SLU）的性能。该模型克服了传统循环神经网络（RNN）在处理SLU任务时的局限性，并引入了语义关联和上下文信息的有效利用策略。" 近年来，语音语言理解作为自然语言处理的一个重要分支，受到了广泛的关注。这项任务通常包括意图检测和槽填充两个部分。自Yao等人在2013年的研究以来，大多数SLU系统都基于循环神经网络（RNN），然而RNN由于其序列性质，存在一些固有的限制，例如难以并行计算和处理长距离依赖。为了解决这些问题，本论文提出了一种名为Graph LSTM的新方法。Graph LSTM首先将文本数据转化为图结构，然后利用消息传递机制学习节点表示。这种转换使得模型能够捕捉到文本中的非线性和结构信息，特别是槽与意图之间的语义关联，这在传统的序列模型中可能被忽视。此外，论文进一步提出了一个上下文门控机制，以更有效地利用上下文信息进行槽填充。这一机制能够动态地调整和融合来自不同上下文的信号，从而提高对槽值预测的准确性。在广泛的评估中，这个增强的Graph LSTM模型展示了其在SLU任务上的优越性能，证明了其在理解和解析复杂语言结构方面的潜力。通过引入图结构和上下文门控，该研究为SLU提供了一个新的视角，不仅提高了模型的表达能力，还提升了处理语言理解任务的效率。这一创新可能对未来的语音识别、对话系统和自然语言处理应用产生深远影响。

Graph LSTM with Context-Gated Mechanism for Spoken Language

Understanding

Linhao Zhang

, Dehong Ma

, Xiaodong Zhang

, Xiaohui Yan

, Houfeng Wang

MOE Key Lab of Computational Linguistics, Peking University, Beijing, 100871, China

CBG Intelligence Engineering Dept, Huawei Technologies, China

{zhanglinhao, madehong, zxdcs, wanghf}@pku.edu.cn

yanxiaohui2@huawei.com

Abstract

Much research in recent years has focused on spoken lan-

guage understanding (SLU), which usually involves two

tasks: intent detection and slot ﬁlling. Since Yao et al.(2013),

almost all SLU systems are RNN-based, which have been

shown to suffer various limitations due to their sequential na-

ture. In this paper, we propose to tackle this task with Graph

LSTM, which ﬁrst converts text into a graph and then utilizes

the message passing mechanism to learn the node representa-

tion. Not only the Graph LSTM addresses the limitations of

sequential models, but it can also help to utilize the seman-

tic correlation between slot and intent. We further propose a

context-gated mechanism to make better use of context infor-

mation for slot ﬁlling. Our extensive evaluation shows that the

proposed model outperforms the state-of-the-art results by a

large margin.

Introduction

Spoken language understanding (SLU) is an essential part

of dialog system. It usually involves two tasks: intent de-

tection (ID) and slot ﬁlling (SF). Typically, ID is regarded

as a semantic utterance classiﬁcation problem, and differ-

ent classiﬁcation methods can be applied (Haffner, Tur, and

Wright 2003; T

ur et al. 2011; Deng et al. 2012). Meanwhile,

SF is usually treated as a sequence labeling problem. Pop-

ular approaches to perform SF include support vector ma-

chines (SVMs) and conditional random ﬁelds (CRFs) (Laf-

ferty, McCallum, and Pereira 2001).

Yao et al.(2013) adapted RNN language models to per-

form SLU, outperforming previous CRF-based models by

a large margin. RNN-based methods (including LSTM and

GRU) have since deﬁned the state-of-the-art in SLU research

(Mesnil et al. 2015; Liu and Lane 2016; Zhang and Wang

2016; Goo et al. 2018; Niu et al. 2019).

Despite their success, these RNN-based models have been

shown to suffer various limitations. Firstly, their inherently

sequential nature precludes parallelization within training

examples (Vaswani et al. 2017). Secondly, local n-grams are

not fully exploited in their models. In SLU, slots are not only

determined by the associated items, but also local context.

 2020, Association for the Advancement of Artiﬁcial

Figure 1: An example of SLU utterance with intent and an-

notated slots using the IOB scheme. The B- preﬁx before a

tag indicates that the tag is the beginning of a slot, and an I-

preﬁx before a tag indicates that the tag is inside a slot. An

O tag indicates that a token belongs to no slot.

As shown in Figure 1, the corresponding slot label for Seat-

tle is B-fromloc, but it could also be B-toloc, if the utterance

is transformed into show ﬂights from San Diego to Seat-

tle. Thirdly, the sequential nature of RNN-based methods

leads to weaker power in capturing long-range dependen-

cies, which accounts for a large portion of SF errors (T

ur,

Hakkani-T

ur, and Heck 2010).

In this paper, we propose to use Graph LSTM to tackle

these problems. There are many variants of Graph LSTM

(Liang et al. 2016; Peng et al. 2017; Zayats and Ostendorf

2018; Song et al. 2018; Zhang, Liu, and Song 2018). In this

paper, we choose the S-LSTM (Zhang, Liu, and Song 2018)

because it is ideally suited for this task.

The main idea of S-LSTM is to model the hidden states

of all words simultaneously rather than sequentially, hence

can solve the non-parallelization problem. Speciﬁcally, the

S-LSTM views the whole sentence as a single graph, which

consists of word-level nodes and a sentence-level node.

These nodes are updated simultaneously through message

passing mechanism. Since message passing is conducted be-

tween consecutive word-level nodes, and between sentence-

level node and each word-level node, both local n-grams and

long-range dependencies are better captured.

Compared to other variants of Graph LSTM, the S-LSTM

has a special sentence-level node, making it ideally suited to

utilize the semantic correlation between slot and intent. We

note that intent and slot are not independent but intrinsically

correlated. As the example shown in Figure 1, an utterance

is more likely to contain departure and arrival cities if its

intent is to ﬁnd a ﬂight, and vice versa. For joint ID and

SF, we use the ﬁnal word-level nodes of S-LSTM for slots

下载后可阅读完整内容，剩余7页未读，立即下载

weixin_38502183

粉丝: 11
资源: 972

图LSTM结合上下文门控机制提升语言理解

添加attention机制的LSTM时间序列预测（matlab）

BiLSTM_RNN-LSTM_RNN_short_lstm神经网络_LSTM_源码.zip

解析LSTM中的门控机制

如何理解LSTM及其在自然语言处理中的应用

trajectories_lstm:LSTM神经网络用于从连续测量数据重建轨迹

使用RNN-LSTM语言建模

lstm_LSTM_

LSTM 的例子 单向LSTM 双向LSTM 多层LSTM.zip

LSTM.rar_LSTM_LSTM MATLAB_lstm matlab_lstm matlab程序_lstm的matlab

最新资源

LSTM 的例子单向LSTM 双向LSTM 多层LSTM.zip