Jacob Eisenstein的自然语言处理教程

需积分: 16 67 浏览量更新于2024-07-18 收藏 5.68MB PDF 举报

"《Natural Language Processing》是Jacob Eisenstein撰写的一本关于自然语言处理的教材，共计583页，包含目录。这本书涵盖了自然语言处理的基础知识，与计算语言学和人工智能领域紧密相关，适合对NLP感兴趣的读者学习。" 本书深入浅出地介绍了自然语言处理（NLP）的基本概念和方法，旨在帮助读者理解这一复杂领域的核心主题。在开篇的"Introduction"部分，作者首先讨论了自然语言处理的范畴及其与邻近学科的关系，如信息检索、机器学习和认知科学等。他强调了三个贯穿NLP研究的主题： 1. 学习与知识：NLP涉及到如何从大量文本数据中学习规律，并利用这些规律来理解和生成人类语言。 2. 搜索与学习：在处理自然语言时，如何有效地搜索信息以及应用学习算法来提升性能。 3. 关系、组合和分布视角：探讨语言的结构特性，包括词语之间的关系、句子的组合规则以及词汇分布假设。接着，书中详细讲解了不同类型的机器学习方法在NLP中的应用，如： - 线性文本分类：从Naive Bayes模型开始，介绍如何处理文本分类问题。Naive Bayes模型基于概率理论，通过计算特征条件概率来进行预测。书中还讨论了特征选择、参数估计、平滑技术、超参数设置等关键步骤。 - 判别式学习：介绍了感知机模型，包括基本感知机和平均感知机。感知机是一种简单但有效的在线学习算法，用于分类任务。此外，书中也提到了损失函数和大 margin 分类的概念，这为支持向量机（SVM）等算法奠定了基础。SVM通过最大化决策边界来提高分类的准确性，而松弛变量则允许SVM在训练数据不完全线性可分的情况下工作。 - 逻辑回归：作为一种广泛使用的分类方法，逻辑回归能处理多分类问题，并且具有正则化能力，有助于防止过拟合。书中详细阐述了逻辑回归的梯度计算和正则化策略。此外，书中还可能涵盖了其他重要主题，如语言建模、句法分析、语义解析、情感分析、机器翻译等，这些都是自然语言处理的核心组成部分。通过这本书，读者将能够建立起坚实的NLP理论基础，并了解如何在实际问题中应用这些理论。对于计算机科学、人工智能或相关专业的学生以及对此领域感兴趣的从业者来说，这是一本极有价值的参考资料。

14 CHAPTER 1. INTRODUCTION

natural language processing is focused on the design and analysis of computational al-352

gorithms and representations for processing natural human language. The goal of natu-353

ral language processing is to provide new computational capabilities around human lan-354

guage: for example, extracting information from texts, translating between languages, an-355

swering questions, holding a conversation, taking instructions, and so on. Fundamental356

linguistic insights may be crucial for accomplishing these tasks, but success is ultimately357

measured by whether and how well the job gets done.358

Machine Learning Contemporary approaches to natural language processing rely heav-359

ily on machine learning, which makes it possible to build complex computer programs360

from examples. Machine learning provides an array of general techniques for tasks like361

converting a sequence of discrete tokens in one vocabulary to a sequence of discrete to-362

kens in another vocabulary — a generalization of what normal people might call “transla-363

tion.” Much of today’s natural language processing research can be thought of as applied364

machine learning. However, natural language processing has characteristics that distin-365

guish it from many of machine learning’s other application domains.366

• Unlike images or audio, text data is fundamentally discrete, with meaning created367

by combinatorial arrangements of symbolic units. This is particularly consequential368

for applications in which text is the output, such as translation and summarization,369

because it is not possible to gradually approach an optimal solution.370

• Although the set of words is discrete, new words are always being created. Further-371

more, the distribution over words (and other linguistic elements) resembles that of a372

power law (Zipf, 1949): there will be a few words that are very frequent, and a long373

tail of words that are rare. A consequence is that natural language processing algo-374

rithms must be especially robust to observations that do not occur in the training375

data.376

• Language is recursive: units such as words can combine to create phrases, which377

can combine by the very same principles to create larger phrases. For example, a378

noun phrase can be created by combining a smaller noun phrase with a preposi-379

tional phrase, as in the whiteness of the whale. The prepositional phrase is created by380

combining a preposition (in this case, of ) with another noun phrase (the whale). In381

this way, it is possible to create arbitrarily long phrases, such as,382

(1.1) . . . huge globular pieces of the whale of the bigness of a human head.

383

The meaning of such a phrase must be analyzed in accord with the underlying hier-384

archical structure. In this case, huge globular pieces of the whale acts as a single noun385

phrase, which is conjoined with the prepositional phrase of the bigness of a human386

Throughout the text, this notation will be used to introduce linguistic examples.

Jacob Eisenstein. Draft of August 16, 2018.

1.1. NATURAL LANGUAGE PROCESSING AND ITS NEIGHBORS 15

head. The interpretation would be different if instead, huge globular pieces were con-387

joined with the prepositional phrase of the whale of the bigness of a human head —388

implying a disappointingly small whale. Even though text appears as a sequence,389

machine learning methods must account for its implicit recursive structure.390

Artiﬁcial Intelligence The goal of artiﬁcial intelligence is to build software and robots391

with the same range of abilities as humans (Russell and Norvig, 2009). Natural language392

processing is relevant to this goal in several ways. The capacity for language is one of the393

central features of human intelligence, and no artiﬁcial intelligence program could be said394

to be complete without the ability to communicate in words.

395

Much of artiﬁcial intelligence research is dedicated to the development of systems396

that can reason from premises to a conclusion, but such algorithms are only as good as397

what they know (Dreyfus, 1992). Natural language processing is a potential solution to398

the “knowledge bottleneck”, by acquiring knowledge from natural language texts, and399

perhaps also from conversations; This idea goes all the way back to Turing’s 1949 pa-400

per Computing Machinery and Intelligence, which proposed the Turing test and helped to401

launch the ﬁeld of artiﬁcial intelligence (Turing, 2009).402

Conversely, reasoning is sometimes essential for basic tasks of language processing,403

such as determining who a pronoun refers to. Winograd schemas are examples in which404

a single word changes the likely referent of a pronoun, in a way that seems to require405

knowledge and reasoning to decode (Levesque et al., 2011). For example,406

(1.2) The trophy doesn’t ﬁt into the brown suitcase because it is too [small/large].407

When the ﬁnal word is small, then the pronoun it refers to the suitcase; when the ﬁnal408

word is large, then it refers to the trophy. Solving this example requires spatial reasoning;409

other schemas require reasoning about actions and their effects, emotions and intentions,410

and social conventions.411

The Winograd schemas demonstrate that natural language understanding cannot be412

achieved in isolation from knowledge and reasoning. Yet the history of artiﬁcial intelli-413

gence has been one of increasing specialization: with the growing volume of research in414

subdisciplines such as natural language processing, machine learning, and computer vi-415

This view is shared by some, but not all, prominent researchers in artiﬁcial intelligence. Michael

Jordan, a specialist in machine learning, has said that if he had a billion dollars to spend on any large

research project, he would spend it on natural language processing (https://www.reddit.com/r/

MachineLearning/comments/2fxi6v/ama_michael_i_jordan/). On the other hand, in a public dis-

cussion about the future of artiﬁcial intelligence in February 2018, computer vision researcher Yann Lecun

argued that despite its many practical applications, language is perhaps “number 300” in the priority list

for artiﬁcial intelligence research, and that it would be a great achievement if AI could attain the capa-

bilities of an orangutan, which do not include language (http://www.abigailsee.com/2018/02/21/

deep-learning-structure-and-innate-priors.html).

Under contract with MIT Press, shared under CC-BY-NC-ND license.

16 CHAPTER 1. INTRODUCTION

sion, it is difﬁcult for anyone to maintain expertise across the entire ﬁeld. Still, recent work416

has demonstrated interesting connections between natural language processing and other417

areas of AI, including computer vision (e.g., Antol et al., 2015) and game playing (e.g.,418

Branavan et al., 2009). The dominance of machine learning throughout artiﬁcial intel-419

ligence has led to a broad consensus on representations such as graphical models and420

knowledge graphs, and on algorithms such as backpropagation and combinatorial opti-421

mization. Many of the algorithms and representations covered in this text are part of this422

consensus.423

Computer Science The discrete and recursive nature of natural language invites the ap-424

plication of theoretical ideas from computer science. Linguists such as Chomsky and425

Montague have shown how formal language theory can help to explain the syntax and426

semantics of natural language. Theoretical models such as ﬁnite-state and pushdown au-427

tomata are the basis for many practical natural language processing systems. Algorithms428

for searching the combinatorial space of analyses of natural language utterances can be429

analyzed in terms of their computational complexity, and theoretically motivated approx-430

imations can sometimes be applied.431

The study of computer systems is also relevant to natural language processing. Large432

datasets of unlabeled text are a natural application for parallelization techniques like433

MapReduce (Dean and Ghemawat, 2008; Lin and Dyer, 2010); high-volume data sources434

such as social media are a natural application for approximate streaming and sketching435

techniques (Goyal et al., 2009). When deep neural networks are implemented in pro-436

duction systems, it is possible to eke out speed gains using techniques such as reduced-437

precision arithmetic (Wu et al., 2016). Many classical natural language processing algo-438

rithms are not naturally suited to graphics processing unit (GPU) parallelization, suggest-439

ing directions for further research at the intersection of natural language processing and440

computing hardware (Yi et al., 2011).441

Speech Processing Natural language is often communicated in spoken form, and speech442

recognition is the task of converting an audio signal to text. From one perspective, this is443

a signal processing problem, which might be viewed as a preprocessing step before nat-444

ural language processing can be applied. However, context plays a critical role in speech445

recognition by human listeners: knowledge of the surrounding words inﬂuences percep-446

tion and helps to correct for noise (Miller et al., 1951). For this reason, speech recognition447

is often integrated with text analysis, particularly with statistical language model, which448

quantify the probability of a sequence of text (see chapter 6). Beyond speech recognition,449

the broader ﬁeld of speech processing includes the study of speech-based dialogue sys-450

tems, which are brieﬂy discussed in chapter 19. Historically, speech processing has often451

been pursued in electrical engineering departments, while natural language processing452

Jacob Eisenstein. Draft of August 16, 2018.

1.2. THREE THEMES IN NATURAL LANGUAGE PROCESSING 17

has been the purview of computer scientists. For this reason, the extent of interaction453

between these two disciplines is less than it might otherwise be.454

Others Natural language processing plays a signiﬁcant role in emerging interdisciplinary455

ﬁelds like computational social science and the digital humanities. Text classiﬁcation456

(chapter 4), clustering (chapter 5), and information extraction (chapter 17) are particularly457

useful tools; another is probabilistic topic models (Blei, 2012), which are not covered in458

this text. Information retrieval (Manning et al., 2008) makes use of similar tools, and459

conversely, techniques such as latent semantic analysis (§ 14.3) have roots in information460

retrieval. Text mining is sometimes used to refer to the application of data mining tech-461

niques, especially classiﬁcation and clustering, to text. While there is no clear distinction462

between text mining and natural language processing (nor between data mining and ma-463

chine learning), text mining is typically less concerned with linguistic structure, and more464

interested in fast, scalable algorithms.465

1.2 Three themes in natural language processing466

Natural language processing covers a diverse range of tasks, methods, and linguistic phe-467

nomena. But despite the apparent incommensurability between, say, the summarization468

of scientiﬁc articles (§ 16.3.4.1) and the identiﬁcation of sufﬁx patterns in Spanish verbs469

(§ 9.1.4.3), some general themes emerge. Each of these themes can be expressed as an470

opposition between two extreme viewpoints on how to process natural language, and in471

each case, existing approaches can be placed on a continuum between these two extremes.472

1.2.1 Learning and knowledge473

A recurring topic of debate is the relative importance of machine learning and linguistic474

knowledge. On one extreme, advocates of “natural language processing from scratch” (Col-475

lobert et al., 2011) propose to use machine learning to train end-to-end systems that trans-476

mute raw text into any desired output structure: e.g., a summary, database, or transla-477

tion. On the other extreme, the core work of natural language processing is sometimes478

taken to be transforming text into a stack of general-purpose linguistic structures: from479

subword units called morphemes, to word-level parts-of-speech, to tree-structured repre-480

sentations of grammar, and beyond, to logic-based representations of meaning. In theory,481

these general-purpose structures should then be able to support any desired application.482

The end-to-end learning approach has been buoyed by recent results in computer vi-483

sion and speech recognition, in which advances in machine learning have swept away484

expert-engineered representations based on the fundamentals of optics and phonology (Krizhevsky485

et al., 2012; Graves and Jaitly, 2014). But while some amount of machine learning is an el-486

ement of nearly every contemporary approach to natural language processing, linguistic487

Under contract with MIT Press, shared under CC-BY-NC-ND license.

18 CHAPTER 1. INTRODUCTION

representations such as syntax trees have not yet gone the way of the visual edge detector488

or the auditory triphone. Linguists have argued for the existence of a “language faculty”489

in all human beings, which encodes a set of abstractions specially designed to facilitate490

the understanding and production of language. The argument for the existence of such491

a language faculty is based on the observation that children learn language faster and492

from fewer examples than would be reasonably possible, if language was learned from493

experience alone.

Regardless of the cognitive validity of these arguments, it seems that494

linguistic structures are particularly important in scenarios where training data is limited.495

Moving away from the extreme ends of the continuum, there are a number of ways in496

which knowledge and learning can be combined in natural language processing. Many497

supervised learning systems make use of carefully engineered features, which transform498

the data into a representation that can facilitate learning. For example, in a task like doc-499

ument classiﬁcation, it may be useful to identify each word’s stem, so that a learning500

system can more easily generalize across related terms such as whale, whales, whalers, and501

whaling. This is particularly important in the many languages that exceed English in the502

complexity of the system of afﬁxes that can attach to words. Such features could be ob-503

tained from a hand-crafted resource, like a dictionary that maps each word to a single504

root form. Alternatively, features can be obtained from the output of a general-purpose505

language processing system, such as a parser or part-of-speech tagger, which may itself506

be built on supervised machine learning.507

Another synthesis of learning and knowledge is in model structure: building machine508

learning models whose architectures are inspired by linguistic theories. For example, the509

organization of sentences is often described as compositional, with meaning of larger510

units gradually constructed from the meaning of their smaller constituents. This idea511

can be built into the architecture of a deep neural network, which is then trained using512

contemporary deep learning techniques (Dyer et al., 2016).513

The debate about the relative importance of machine learning and linguistic knowl-514

edge sometimes becomes heated. No machine learning specialist likes to be told that their515

engineering methodology is unscientiﬁc alchemy;

nor does a linguist want to hear that516

the search for general linguistic principles and structures has been made irrelevant by big517

data. Yet there is clearly room for both types of research: we need to know how far we518

can go with end-to-end learning alone, while at the same time, we continue the search for519

linguistic representations that generalize across applications, scenarios, and languages.520

For more on the history of this debate, see Church (2011); for an optimistic view of the521

potential symbiosis between computational linguistics and deep learning, see Manning522

The Language Instinct (Pinker, 2003) articulates these arguments in an engaging and popular style. For

arguments against the innateness of language, see Elman et al. (1998).

Ali Rahimi argued that much of deep learning research was similar to “alchemy” in a presentation at

the 2017 conference on Neural Information Processing Systems. He was advocating for more learning theory,

not more linguistics.

Jacob Eisenstein. Draft of August 16, 2018.

剩余582页未读，继续阅读

diudiuiper

粉丝: 0
资源: 1

Jacob Eisenstein的自然语言处理教程

Python Natural Language Processing

eisenstein-nlp-notes2018年6月份最新版

【Manning经典教材】Natural Language Processing in Action

自然语言处理技术的书推荐几本

推荐下自然语言处理的教材

计算机毕设英文参考文献

自然语言处理的学习路径，列出一些学习资源

括机器学习、深度学习、自然语言处理等方面推荐的书籍

推荐一些学习chatgpt的论文

jacob的压缩包jacob.zip

最新资源