Python与nltk：文本挖掘与关键词提取实战

需积分: 9 136 浏览量更新于2024-07-22 收藏 1.25MB PDF 举报

本资源主要围绕自然语言处理（Natural Language Processing, NLP）在Python编程中的应用展开，特别强调了nltk库在文本分析中的重要作用。首先，章节标题"LanguageProcessingandPython"明确了本书将结合Python编程技术与大量文本数据来探讨自然语言处理的可能性和实用价值。以下是章节内容的详细解读： 1. 利用简单编程与海量文本：作者提出，通过简单的编程手段与大规模文本数据的结合，可以实现诸如文本挖掘、词频统计、情感分析等任务。这涉及到了基础的数据处理和清洗，以及如何设计算法来提取有价值的信息。 2. 自动关键词提取与主题概括：章节关注如何运用NLP技术自动化地识别文档中的关键术语和短语，这些内容可以作为文本的主题概括，帮助我们快速理解文本的核心思想。这涉及到文本特征提取、聚类和信息检索等NLP核心技术。 3. Python的适用性评估：作者质疑Python是否是进行自然语言处理的理想工具，这暗示了Python强大的数据处理能力、丰富的库支持（如nltk、spaCy等）以及其易学性可能会使其成为这一领域内的热门选择。 4. 自然语言处理的挑战：深入讨论了NLP面临的挑战，如歧义性、语言结构的复杂性、文化依赖性以及处理非标准或口语化的文本等，这些都是开发者在实际应用中需要面对和解决的问题。在教学方法上，章节分为“计算机与语言”和“更深入的Python”两部分。前者侧重于使用编程技术解决问题，而后者则系统地讲解关键编程概念。这种混合的教学方式旨在提供一个真实的NLP工作环境体验，同时确保读者逐步掌握基础语言学概念和技术。这个章节为读者展示了如何通过Python和nltk库在NLP领域开展实践，包括基本技术应用、问题解决策略以及语言处理技术的潜在挑战。后续章节会更加深入地探讨这些概念，并鼓励读者在实践中不断提升技能。对于想要入门NLP或提高Python在该领域应用能力的人来说，这是一个宝贵的起点。

1.3. Computing with Language: Simple Statistics

differences between authors, genres or languages. Table 1.2 summarizes the methods deﬁned in

frequency distributions.

December 18, 2008 16 Bird, Klein & Loper

1. Language Processing and Python Natural Language Processing (DRAFT)

Example

Description

fdist = FreqDist(

samples)

create a frequency distribution containing the given samples

fdist.inc(sample) increment the count for this sample

fdist[’monstrous’] count of the number of times a given sample occurred

fdist.freq(’monstrous’) frequency of a given sample

fdist.N() total number of samples

fdist.keys() the samples sorted in order of decreasing frequency

for sample in fdist: iterate over the samples, in order of decreasing frequency

fdist.max() sample with the greatest count

fdist.tabulate() tabulate the frequency distribution

fdist.plot() graphical plot of the frequency distribution

fdist.plot(cumulative=

True)

cumulative plot of the frequency distribution

fdist1 < fdist2 samples in fdist1 occur less frequently than in fdist2

Table 1.2: Methods Deﬁned for NLTK’s Frequency Distributions

Our discussion of frequency distributions has introduced some important Python concepts, and we will

look at them systematically in Section 1.4. We’ve also touched on the topic of normalization, and we’ll

explore this in depth in Chapter 3.

1.4 Back to Python: Making Decisions and Taking Control

So far, our little programs have had some interesting qualities: (i) the ability to work with language,

and (ii) the potential to save human effort through automation. A key feature of programming is the

ability of machines to make decisions on our behalf, executing instructions when certain conditions are

met, or repeatedly looping through text data until some condition is satisﬁed. This feature is known as

control, and is the focus of this section.

Conditionals

Python supports a wide range of operators like < and >= for testing the relationship between values.

The full set of these relational operators are shown in T

able 1.3.

Operator Relationship

< less than

<= less than or equal to

== equal to (note this is two not one = sign)

!= not equal to

> greater than

>= greater than or equal to

Table 1.3: Numerical Comparison Operators

Bird, Klein & Loper 17 December 18, 2008

1.4. Back to Python: Making Decisions and Taking Control

We can use these to select different words from a sentence of news text. Here are some examples 

only the operator is changed from one line to the next. They all use sent7, the ﬁrst sentence from

text7 (Wall Street Journal).

>>> [w for w in sent7 if len(w) < 4]

[’,’, ’61’, ’old’, ’,’, ’the’, ’as’, ’a’, ’29’, ’.’]

>>> [w for w in sent7 if len(w) <= 4]

[’,’, ’61’, ’old’, ’,’, ’will’, ’join’, ’the’, ’as’, ’a’, ’Nov.’, ’29’, ’.’]

>>> [w for w in sent7 if len(w)

== 4]

[’will’, ’join’, ’Nov.’]

>>> [w for w in sent7 if len(w)

!= 4]

[’Pierre’, ’Vinken’, ’,’, ’61’, ’years’, ’old’, ’,’, ’the’, ’board’,

’as’,

’a’, ’nonexecutive’, ’director’, ’29’, ’.’]

>>>

Notice

the pattern in all of these examples: [w for w in text if condition ]. In these cases the

condition is always a numerical comparison. However, we can also test various properties of words,

using the functions listed in Table 1.4.

Function Meaning

s.startswith(t) s starts

with t

s.endswith(t) s ends

with t

t in s t is

contained inside s

s.islower() all

cased characters in s are lowercase

s.isupper() all

cased characters in s are uppercase

s.isalpha() all

characters in s are alphabetic

s.isalnum() all

characters in s are alphanumeric

s.isdigit() all

characters in s are digits

s.istitle() s is

titlecased (all words have initial capital)

able 1.4: Some Word Comparison Operators

Here are some examples of these operators being used to select words from our texts: words ending

with -ableness; words containing gnt; words having an initial capital; and words consisting entirely of

digits.

>>> sorted(w for w in set(text1) if w.endswith(’ableness’))

[’comfortableness’, ’honourableness’, ’immutableness’, ’indispensableness’, ...]

>>> sorted(term for term in set(text4) if ’gnt’ in term)

[’Sovereignty’,

’sovereignties’, ’sovereignty’]

>>> sorted(item for item in set(text6) if item.istitle())

[’A’,

’Aaaaaaaaah’, ’Aaaaaaaah’, ’Aaaaaah’, ’Aaaah’, ’Aaaaugh’, ’Aaagh’, ...]

>>> sorted(item for item in set(sent7) if item.isdigit())

[’29’,

’61’]

>>>

e can also create more complex conditions. If c is a condition, then not c is also a condition. If we

have two conditions c

and c

, then we can combine them to form a new condition using and and or:

and c

, c

or c

December

18, 2008 18 Bird, Klein & Loper

1. Language Processing and Python Natural Language Processing (DRAFT)

Your Turn: Run the following examples and try to explain what is going on in each one.

Next, try to make up some conditions of your own.

sorted(w for w in set(text7) if ’-’ in w and ’index’ in w) sorted(wd for wd in

set(text3) if wd.istitle() and len(wd) > 10) sorted(w for w in set(sent7) if not

w.islower()) sorted(t for t in set(text2) if ’cie’ in t or ’cei’ in t)

Operating on Every Element

In Section 1.3, we saw some examples of counting items other than words. Let’s take a closer look at

the notation we used:

>>> [len(w) for w in text1]

[1, 4, 4, 2, 6, 8, 4, 1, 9, 1, 1, 8, 2, 1, 4, 11, 5, 2, 1, 7, 6, 1, 3, 4, 5, 2, ...]

>>> [w.upper() for w in text1]

[’[’,

’MOBY’, ’DICK’, ’BY’, ’HERMAN’, ’MELVILLE’, ’1851’, ’]’, ’ETYMOLOGY’, ’.’, ...]

>>>

These

expressions have the form [f(w) for ...] or [w.f() for ...], where f is a function that

operates on a word to compute its length, or to convert it to uppercase. For now, you don’t need to

understand the difference between the notations f(w) and w.f(). Instead, simply learn this Python

idiom which performs the same operation on every element of a list. In the above examples, it goes

through each word in text1, assigning each one in turn to the variable w and performing the speciﬁed

operation on the variable.

The above notation is called a “list comprehension”. This is our ﬁrst example of a Python

idiom, a ﬁxed notation that we use habitually without bothering to analyze each time. Mas-

tering such idioms is an important part of becoming a ﬂuent Python programmer.

Let’s return to the question of vocabulary size, and apply the same idiom here:

>>> len(text1)

260819

>>> len(set(text1))

19317

>>> len(set(word.lower() for word in text1))

17231

>>>

w that we are not double-counting words like This and this, which differ only in capitalization, we’ve

wiped 2,000 off the vocabulary count! We can go a step further and eliminate numbers and punctuation

from the vocabulary count, by ﬁltering out any non-alphabetic items:

>>> len(set(word.lower() for word in text1 if word.isalpha()))

16948

>>>

This

example is slightly complicated: it lowercases all the purely alphabetic items. Perhaps it would

have been simpler just to count the lowercase-only items, but this gives the incorrect result (why?).

Don’t worry if you don’t feel conﬁdent with these already. You might like to try some of the exercises

at the end of this chapter, or wait til we come back to these again in the next chapter.

Bird, Klein & Loper 19 December 18, 2008

剩余93页未读，继续阅读

chinapuma123

粉丝: 0
资源: 1

Python与nltk：文本挖掘与关键词提取实战

最新资源