深度学习在自然语言处理中的应用：Yoav Goldberg的神经网络方法

需积分: 9 16 浏览量更新于2024-07-19 收藏 1.9MB PDF 举报

"Neural_Network_Methods_in_Natural_Language_Processing - Yoav Goldberg, 高清完整版本，2017年出版" 《自然语言处理中的神经网络方法》是Yoav Goldberg在2017年出版的一本关于自然语言处理（NLP）的专著。这本书深入探讨了神经网络在处理自然语言数据时的应用。全书分为两大部分，第一部分（Part I 和 Part II）主要介绍了监督机器学习的基础，以及前馈神经网络的基本概念。作者还讨论了如何在语言数据上应用机器学习，以及如何使用基于向量而非符号的表示方法来处理单词。书中首先讲解了监督机器学习的核心原理，这是一种通过训练数据来预测未知数据的方法。在NLP中，这种技术被广泛应用于诸如文本分类、情感分析、机器翻译等任务。监督学习的关键在于构建一个模型，该模型可以从输入（例如，一段文本）中学习，并生成相应的输出（如类别标签或翻译结果）。接着，书中介绍了前馈神经网络（Feed-Forward Neural Networks），这是一种基本的神经网络架构，其中信息仅沿着一个方向传递，即从输入层到隐藏层，再到输出层。这些网络通过多层非线性转换处理输入，从而能够捕获复杂的数据模式。在NLP中，神经网络常用于构建词嵌入（Word Embeddings），将单词转换成连续的向量空间，使得语义相近的词在空间中的距离较近。此外，书中的一个关键概念是计算图（Computation Graph）。这是一个抽象概念，用于描述神经网络计算过程中的操作流程，它简化了反向传播算法的实现，使优化过程更为高效。计算图在训练过程中至关重要，因为它允许自动微分，这是神经网络训练中更新权重所必需的。第二部分可能涉及更高级的主题，如循环神经网络（Recurrent Neural Networks, RNNs）和长短期记忆网络（Long Short-Term Memory, LSTM），它们能够处理序列数据，如句子和段落，考虑上下文信息。此外，可能还会讨论卷积神经网络（Convolutional Neural Networks, CNNs）在文本分类和信息抽取中的应用，以及更现代的Transformer模型和自注意力机制（Self-Attention）在生成模型（如BERT和GPT系列）中的作用。《自然语言处理中的神经网络方法》是理解和应用神经网络解决NLP问题的宝贵资源，涵盖了从基础理论到先进模型的全面知识。对于想深入学习这一领域的读者来说，这本书提供了丰富的实践指导和理论背景。

C H A P T E R 1

Introduction

1.1 THE CHALLENGES OF NATURAL LANGUAGE

PROCESSING

Natural language processing (NLP) is the ﬁeld of designing methods and algorithms that take

as input or produce as output unstructured, natural language data. Human language is highly

ambiguous (consider the sentence I ate pizza with friends, and compare it to I ate pizza with

olives), and also highly variable (the core message of I ate pizza with friends can also be expressed as

friends and I shared some pizza). It is also ever changing and evolving. People are great at producing

language and understanding language, and are capable of expressing, perceiving, and interpreting

very elaborate and nuanced meanings. At the same time, while we humans are great users of

language, we are also very poor at formally understanding and describing the rules that govern

language.

Understanding and producing language using computers is thus highly challenging. Indeed,

the best known set of methods for dealing with language data are using supervised machine learn-

ing algorithms, that attempt to infer usage patterns and regularities from a set of pre-annotated

input and output pairs. Consider, for example, the task of classifying a document into one of four

categories: S, P, G, and E. Obviously, the words in the documents

provide very strong hints, but which words provide what hints? Writing up rules for this task is

rather challenging. However, readers can easily categorize a document into its topic, and then,

based on a few hundred human-categorized examples in each category, let a supervised machine

learning algorithm come up with the patterns of word usage that help categorize the documents.

Machine learning methods excel at problem domains where a good set of rules is very hard to

deﬁne but annotating the expected output for a given input is relatively simple.

Besides the challenges of dealing with ambiguous and variable inputs in a system with ill-

deﬁned and unspeciﬁed set of rules, natural language exhibits an additional set of properties that

make it even more challenging for computational approaches, including machine learning: it is

discrete, compositional, and sparse.

Language is symbolic and discrete. e basic elements of written language are characters.

Characters form words that in turn denote objects, concepts, events, actions, and ideas. Both

characters and words are discrete symbols: words such as “hamburger” or “pizza” each evoke in us

a certain mental representations, but they are also distinct symbols, whose meaning is external to

them and left to be interpreted in our heads. ere is no inherent relation between “hamburger”

and “pizza” that can be inferred from the symbols themselves, or from the individual letters they

2 1. INTRODUCTION

are made of. Compare that to concepts such as color, prevalent in machine vision, or acoustic

signals: these concepts are continuous, allowing, for example, to move from a colorful image to a

gray-scale one using a simple mathematical operation, or to compare two diﬀerent colors based

on inherent properties such as hue and intensity. is cannot be easily done with words—there

is no simple operation that will allow us to move from the word “red” to the word “pink” without

using a large lookup table or a dictionary.

Language is also compositional: letters form words, and words form phrases and sentences.

e meaning of a phrase can be larger than the meaning of the individual words that comprise it,

and follows a set of intricate rules. In order to interpret a text, we thus need to work beyond the

level of letters and words, and look at long sequences of words such as sentences, or even complete

documents.

e combination of the above properties leads to data sparseness. e way in which words

(discrete symbols) can be combined to form meanings is practically inﬁnite. e number of possi-

ble valid sentences is tremendous: we could never hope to enumerate all of them. Open a random

book, and the vast majority of sentences within it you have not seen or heard before. Moreover,

it is likely that many sequences of four-words that appear in the book are also novel to you. If you

were to look at a newspaper from just 10 years ago, or imagine one 10 years in the future, many

of the words, in particular names of persons, brands, and corporations, but also slang words and

technical terms, will be novel as well. ere is no clear way of generalizing from one sentence to

another, or deﬁning the similarity between sentences, that does not depend on their meaning—

which is unobserved to us. is is very challenging when we come to learn from examples: even

with a huge example set we are very likely to observe events that never occurred in the example

set, and that are very diﬀerent than all the examples that did occur in it.

1.2 NEURAL NETWORKS AND DEEP LEARNING

Deep learning is a branch of machine learning. It is a re-branded name for neural networks—a

family of learning techniques that was historically inspired by the way computation works in the

brain, and which can be characterized as learning of parameterized diﬀerentiable mathematical

functions.¹ e name deep-learning stems from the fact that many layers of these diﬀerentiable

function are often chained together.

While all of machine learning can be characterized as learning to make predictions based

on past observations, deep learning approaches work by learning to not only predict but also to

correctly represent the data, such that it is suitable for prediction. Given a large set of desired input-

output mapping, deep learning approaches work by feeding the data into a network that produces

successive transformations of the input data until a ﬁnal transformation predicts the output. e

transformations produced by the network are learned from the given input-output mappings,

such that each transformation makes it easier to relate the data to the desired label.

¹In this book we take the mathematical view rather than the brain-inspired view.

1.3. DEEP LEARNING IN NLP 3

While the human designer is in charge of designing the network architecture and training

regime, providing the network with a proper set of input-output examples, and encoding the input

data in a suitable way, a lot of the heavy-lifting of learning the correct representation is performed

automatically by the network, supported by the network’s architecture.

1.3 DEEP LEARNING IN NLP

Neural networks provide a powerful learning machinery that is very appealing for use in natural

language problems. A major component in neural networks for language is the use of an embed-

ding layer, a mapping of discrete symbols to continuous vectors in a relatively low dimensional

space. When embedding words, they transform from being isolated distinct symbols into math-

ematical objects that can be operated on. In particular, distance between vectors can be equated

to distance between words, making it easier to generalize the behavior from one word to another.

is representation of words as vectors is learned by the network as part of the training process.

Going up the hierarchy, the network also learns to combine word vectors in a way that is useful for

prediction. is capability alleviates to some extent the discreteness and data-sparsity problems.

ere are two major kinds of neural network architectures, that can be combined in various

ways: feed-forward networks and recurrent/recursive networks.

Feed-forward networks, in particular multi-layer perceptrons (MLPs), allow to work with

ﬁxed sized inputs, or with variable length inputs in which we can disregard the order of the ele-

ments. When feeding the network with a set of input components, it learns to combine them in

a meaningful way. MLPs can be used whenever a linear model was previously used. e nonlin-

earity of the network, as well as the ability to easily integrate pre-trained word embeddings, often

lead to superior classiﬁcation accuracy.

Convolutional feed-forward networks are specialized architectures that excel at extracting

local patterns in the data: they are fed arbitrarily sized inputs, and are capable of extracting mean-

ingful local patterns that are sensitive to word order, regardless of where they appear in the input.

ese work very well for identifying indicative phrases or idioms of up to a ﬁxed length in long

sentences or documents.

Recurrent neural networks (RNNs) are specialized models for sequential data. ese are

network components that take as input a sequence of items, and produce a ﬁxed size vector that

summarizes that sequence. As “summarizing a sequence” means diﬀerent things for diﬀerent tasks

(i.e., the information needed to answer a question about the sentiment of a sentence is diﬀerent

from the information needed to answer a question about its grammaticality), recurrent networks

are rarely used as standalone component, and their power is in being trainable components that

can be fed into other network components, and trained to work in tandem with them. For ex-

ample, the output of a recurrent network can be fed into a feed-forward network that will try

to predict some value. e recurrent network is used as an input-transformer that is trained to

produce informative representations for the feed-forward network that will operate on top of it.

Recurrent networks are very impressive models for sequences, and are arguably the most exciting

4 1. INTRODUCTION

oﬀer of neural networks for language processing. ey allow abandoning the markov assumption

that was prevalent in NLP for decades, and designing models that can condition on entire sen-

tences, while taking word order into account when it is needed, and not suﬀering much from

statistical estimation problems stemming from data sparsity. is capability leads to impressive

gains in language-modeling, the task of predicting the probability of the next word in a sequence

(or, equivalently, the probability of a sequence), which is a cornerstone of many NLP applications.

Recursive networks extend recurrent networks from sequences to trees.

Many of the problems in natural language are structured, requiring the production of com-

plex output structures such as sequences or trees, and neural network models can accommodate

that need as well, either by adapting known structured-prediction algorithms for linear models,

or by using novel architectures such as sequence-to-sequence (encoder-decoder) models, which

we refer to in this book as conditioned-generation models. Such models are at the heart of state-

of-the-art machine translation.

Finally, many language prediction tasks are related to each other, in the sense that knowing

to perform one of them will help in learning to perform the others. In addition, while we may

have a shortage of supervised (labeled) training data, we have ample supply of raw text (unlabeled

data). Can we learn from related tasks and un-annotated data? Neural network approaches pro-

vide exciting opportunities for both MTL (learning from related problems) and semi-supervised

learning (learning from external, unannotated data).

1.3.1 SUCCESS STORIES

Fully connected feed-forward neural networks (MLPs) can, for the most part, be used as a drop-in

replacement wherever a linear learner is used. is includes binary and multi-class classiﬁcation

problems, as well as more complex structured prediction problems. e nonlinearity of the net-

work, as well as the ability to easily integrate pre-trained word embeddings, often lead to superior

classiﬁcation accuracy. A series of works² managed to obtain improved syntactic parsing results

by simply replacing the linear model of a parser with a fully connected feed-forward network.

Straightforward applications of a feed-forward network as a classiﬁer replacement (usually cou-

pled with the use of pre-trained word vectors) provide beneﬁts for many language tasks, including

the very well basic task of language modeling³ CCG supertagging,⁴ dialog state tracking,⁵ and

pre-ordering for statistical machine translation.⁶ Iyyer et al. [2015] demonstrate that multi-layer

feed-forward networks can provide competitive results on sentiment classiﬁcation and factoid

question answering. Zhou et al. [2015] and Andor et al. [2016] integrate them in a beam-search

structured-prediction system, achieving stellar accuracies on syntactic parsing, sequence tagging

and other tasks.

²[Chen and Manning, 2014, Durrett and Klein, 2015, Pei et al., 2015, Weiss et al., 2015]

³See Chapter 9, as well as Bengio et al. [2003], Vaswani et al. [2013].

⁴[Lewis and Steedman, 2014]

⁵[Henderson et al., 2013]

⁶[de Gispert et al., 2015]

1.3. DEEP LEARNING IN NLP 5

Networks with convolutional and pooling layers are useful for classiﬁcation tasks in which

we expect to ﬁnd strong local clues regarding class membership, but these clues can appear in

diﬀerent places in the input. For example, in a document classiﬁcation task, a single key phrase

(or an ngram) can help in determining the topic of the document [Johnson and Zhang, 2015].

We would like to learn that certain sequences of words are good indicators of the topic, and do not

necessarily care where they appear in the document. Convolutional and pooling layers allow the

model to learn to ﬁnd such local indicators, regardless of their position. Convolutional and pool-

ing architecture show promising results on many tasks, including document classiﬁcation,⁷ short-

text categorization,⁸ sentiment classiﬁcation,⁹ relation-type classiﬁcation between entities,¹⁰ event

detection,¹¹ paraphrase identiﬁcation,¹² semantic role labeling,¹³ question answering,¹⁴ predict-

ing box-oﬃce revenues of movies based on critic reviews,¹⁵ modeling text interestingness,¹⁶ and

modeling the relation between character-sequences and part-of-speech tags.¹⁷

In natural language we often work with structured data of arbitrary sizes, such as sequences

and trees. We would like to be able to capture regularities in such structures, or to model similari-

ties between such structures. Recurrent and recursive architectures allow working with sequences

and trees while preserving a lot of the structural information. Recurrent networks [Elman, 1990]

are designed to model sequences, while recursive networks [Goller and Küchler, 1996] are gen-

eralizations of recurrent networks that can handle trees. Recurrent models have been shown to

produce very strong results for language modeling,¹⁸ as well as for sequence tagging,¹⁹ machine

translation,²⁰ parsing,²¹ and many other tasks including noisy text normalization,²² dialog state

tracking,²³ response generation,²⁴ and modeling the relation between character sequences and

part-of-speech tags.²⁵

⁷[Johnson and Zhang, 2015]

⁸[Wang et al., 2015a]

⁹[Kalchbrenner et al., 2014, Kim, 2014]

¹⁰[dos Santos et al., 2015, Zeng et al., 2014]

¹¹[Chen et al., 2015, Nguyen and Grishman, 2015]

¹²[Yin and Schütze, 2015]

¹³[Collobert et al., 2011]

¹⁴[Dong et al., 2015]

¹⁵[Bitvai and Cohn, 2015]

¹⁶[Gao et al., 2014]

¹⁷[dos Santos and Zadrozny, 2014]

¹⁸Some notable works are Adel et al. [2013], Auli and Gao [2014], Auli et al. [2013], Duh et al. [2013], Jozefowicz et al. [2016],

Mikolov [2012], Mikolov et al. [2010, 2011].

¹⁹[Irsoy and Cardie, 2014, Ling et al., 2015b, Xu et al., 2015]

²⁰[Cho et al., 2014b, Sundermeyer et al., 2014, Sutskever et al., 2014, Tamura et al., 2014]

²¹[Dyer et al., 2015, Kiperwasser and Goldberg, 2016b, Watanabe and Sumita, 2015]

²²[Chrupala, 2014]

²³[Mrkšić et al., 2015]

²⁴[Kannan et al., 2016, Sordoni et al., 2015]

²⁵[Ling et al., 2015b]

剩余282页未读，继续阅读

张博208

粉丝: 1078
资源: 46

深度学习在自然语言处理中的应用：Yoav Goldberg的神经网络方法

Neural Network Methods for Natural Language Processing

Neural network methods for natural language processing

高清版Neural Network Methods in Natural Language Processing April 17, 2017

Neural Network Methods in Natural Language Processing

Neural Network Methods in Natural Language Processing epub

neural network methods for natural language processing

NN methods for NLP Neural Network Methods for Natural Language Processing

(Yoav Goldberg)Neural Network Methods for Natural Language Processing适合nlp进阶

Neural Network Methods for Natural Language Process

Neural Representations of Natural Language.pdf

最新资源