Python文本分析实战：获取数据洞察力

4星 · 超过85%的资源需积分: 46 26 浏览量更新于2024-07-20 收藏 6.5MB PDF 举报

"Text.Analytics.with.Python" 是一本面向IT专业人士、分析师、开发者、语言学专家和数据科学家的书籍，旨在帮助读者通过Python学习自然语言处理（NLP）和文本分析技术，从而从文本数据中提取有用见解。书中涵盖了从基础到高级的概念，包括文本和语言的语法、结构、语义，以及文本分类、聚类、主题建模和文本摘要等算法和技巧。作者Dipanjan Sarkar采用结构化和全面的方法，适合初学者和有一定经验的读者。书中的每个章节都会从宏观和微观视角来探讨技术，既解释其应用场景，又深入解析数学概念，并提供实际案例来指导读者解决问题。例如，书中会展示如何构建文本分类系统对新闻文章进行分类，如何使用主题模型分析应用或游戏评论，以及如何对电影概要进行聚类和情感分析。本书的特点包括： 1. 完整覆盖NLP和文本分析的主要概念和技术。 2. 提供实用的现实世界案例，如新闻文章分类、电影评论情感分析等。 3. 基于Python和多个流行的开源库实现，如nltk、gensim、scikit-learn、spaCy和Pattern。 4. 内容涵盖自然语言基础、Python回顾、文本处理与理解、文本分类、文本摘要、文本相似性和聚类，以及语义和情感分析。目录如下： - 第1章：自然语言基础 - 第2章：Python复习 - 第3章：处理和理解文本 - 第4章：文本分类 - 第5章：文本摘要 - 第6章：文本相似性和聚类 - 第7章：语义和情感分析这本书对于那些希望提升在文本数据分析方面技能的人来说是一本宝贵的资源，无论你是想提升现有技能，还是初次接触这个领域，都能从中受益。通过阅读此书，读者将能够运用Python和相关工具解决实际问题，从文本数据中获取可操作的洞察。

xxi

Introduction

I have been into mathematics and statistics since high school, when numbers began to

really interest me. Analytics, data science, and more recently text analytics came much

later, perhaps around four or five years ago when the hype about Big Data and Analytics

was getting bigger and crazier. Personally I think a lot of it is over-hyped, but a lot of it is

also exciting and presents huge possibilities with regard to new jobs, new discoveries, and

solving problems that were previously deemed impossible to solve.

Natural Language Processing (NLP) has always caught my eye because the human

brain and our cognitive abilities are really fascinating. The ability to communicate

information, complex thoughts, and emotions with such little effort is staggering once

you think about trying to replicate that ability in machines. Of course, we are advancing

by leaps and bounds with regard to cognitive computing and artificial intelligence (AI),

but we are not there yet. Passing the Turing Test is perhaps not enough; can a machine

truly replicate a human in all aspects?

The ability to extract useful information and actionable insights from heaps of

unstructured and raw textual data is in great demand today with regard to applications in

NLP and text analytics. In my journey so far, I have struggled with various problems, faced

many challenges, and learned various lessons over time. This book contains a major

chunk of the knowledge I’ve gained in the world of text analytics, where building a fancy

word cloud from a bunch of text documents is not enough anymore.

Perhaps the biggest problem with regard to learning text analytics is not a lack of

information but too much information, often called information overload. There are

so many resources, documentation, papers, books, and journals containing so much

theoretical material, concepts, techniques, and algorithms that they often overwhelm

someone new to the field. What is the right technique to solve a problem? How does

text summarization really work? Which are the best frameworks to solve multi-class text

categorization? By combining mathematical and theoretical concepts with practical

implementations of real-world use-cases using Python, this book tries to address this

problem and help readers avoid the pressing issues I’ve faced in my journey so far.

This book follows a comprehensive and structured approach. First it tackles the

basics of natural language understanding and Python constructs in the initial chapters.

Once you’re familiar with the basics, it addresses interesting problems in text analytics

in each of the remaining chapters, including text classification, clustering, similarity

analysis, text summarization, and topic models. In this book we will also analyze text

structure, semantics, sentiment, and opinions. For each topic, I cover the basic concepts

and use some real-world scenarios and data to implement techniques covering each

concept. The idea of this book is to give you a flavor of the vast landscape of text analytics

and NLP and arm you with the necessary tools, techniques, and knowledge to tackle your

own problems and start solving them. I hope you find this book helpful and wish you the

very best in your journey through the world of text analytics!

D. Sarkar, Text Analytics with Python, DOI 10.1007/978-1-4842-2388-8_1

CHAPTER 1

Natural Language Basics

We have ushered in the age of Big Data where organizations and businesses are having

difficulty managing all the data generated by various systems, processes, and transactions.

However, the term Big Data is misused a lot due to the nature of its popular but vague

definition of “the 3 V’s”—volume, variety, and velocity of data. This is because sometimes

it is very difficult to exactly quantify what data is “Big.” Some might think a billion records

in a database would be Big Data, but that number seems really minute compared to the

petabytes of data being generated by various sensors or even social media. There is a large

volume of unstructured textual data present across all organizations, irrespective of their

domain. Just to take some examples, we have vast amounts of data in the form of tweets,

status updates, comments, hashtags, articles, blogs, wikis, and much more on social

media. Even retail and e-commerce stores generate a lot of textual data from new product

information and metadata with customer reviews and feedback.

The main challenges associated with textual data are twofold. The first challenge

deals with effective storage and management of this data. Usually textual data is

unstructured and does not adhere to any specific predefined data model or schema,

which is usually followed by relational databases. However, based on the data semantics,

you can store it in either SQL-based database management systems ( DBMS ) like SQL

Server or even NoSQL-based systems like MongoDB. Organizations having enormous

amounts of textual datasets often resort to file-based systems like Hadoop where they

dump all the data in the Hadoop Distributed File System (HDFS) and access it as needed,

which is one of the main principles of a data lake .

The second challenge is with regard to analyzing this data and trying to extract

meaningful patterns and useful insights that would be beneficial to the organization.

Even though we have a large number of machine learning and data analysis techniques

at our disposal, most of them are tuned to work with numerical data, hence we have

to resort to areas like natural language processing ( NLP ) and specialized techniques,

transformations, and algorithms to analyze text data, or more specifically natural

language , which is quite different from programming languages that are easily

understood by machines. Remember that textual data, being highly unstructured, does

not follow or adhere to structured or regular syntax and patterns—hence we cannot

directly use mathematical or statistical models to analyze it.

Electronic supplementary material The online version of this chapter

(doi:

10.1007/978-1-4842-2388-8_1 ) contains supplementary material, which is available

to authorized users.

CHAPTER 1 ■ NATURAL LANGUAGE BASICS

Before we dive into specific techniques and algorithms to analyze textual data, we will be

going over some of the main concepts and theoretical principles associated with the nature

of text data in this chapter. The primary intent here is to get you familiarized with concepts

and domains associated with natural language understanding , processing , and text analytics .

We will be using the Python programming language in this book primarily for accessing and

analyzing text data. The examples in this chapter will be pretty straightforward and fairly easy

to follow. However, you can quickly skim over Chapter

2 in case you want to brush up on

Python before going through this chapter. All the examples are available with this book and

also in my GithHub repository at

https://github.com/dipanjanS/text-analytics-with-

python

which includes programs, code snippets and datasets. This chapter covers concepts

relevant to natural language, linguistics, text data formats, syntax, semantics, and grammars

before moving on to more advanced topics like text corpora , NLP, and text analytics.

Natural Language

Textual data is unstructured data but it usually belongs to a specific language following

specific syntax and semantics. Any piece of text data—a simple word, sentence, or

document—relates back to some natural language most of the time. In this section, we

will be looking at the definition of natural language, the philosophy of language, language

acquisition, and the usage of language.

What Is Natural Language?

To understand text analytics and natural language processing , we need to understand

what makes a language “natural.” In simple terms, a natural language is one developed

and evolved by humans through natural use and communication , rather than

constructed and created artificially, like a computer programming language.

Human languages like English, Japanese, and Sanskrit are natural languages. Natural

languages can be communicated in different forms, including speech, writing, or even signs.

There has been a lot of scholarship and effort applied toward understanding the origins,

nature, and philosophy of language. We will discuss that briefly in the following section.

The Philosophy of Language

We now know what a natural language means. But think about the following questions.

What are the origins of a language ? What makes the English language “English”? How did

the meaning of the word fruit come into existence? How do humans communicate among

themselves with language? These are definitely some heavy philosophical questions.

The philosophy of language mainly deals with the following four problems and seeks

answers to solve them:

• The nature of meaning in a language

• The use of language

• Language cognition

• The relationship between language and reality

CHAPTER 1 ■ NATURAL LANGUAGE BASICS

• The nature of meaning in a language is concerned with the

semantics of a language and the nature of meaning itself. Here,

philosophers of language or linguistics try to find out what it

means to actually “mean” anything—that is, how the meaning of

any word or sentence originated and came into being and how

different words in a language can be synonyms of each other and

form relations. Another thing of importance here is how structure

and syntax in the language pave the way for semantics, or to be

more specific, how words, which have their own meanings, are

structured together to form meaningful sentences. Linguistics

is the scientific study of language, a special field that deals with

some of these problems we will be looking at in more detail later

on. Syntax, semantics, grammars, and parse trees are some ways

to solve these problems. The nature of meaning can be expressed

in linguistics between two human beings, notably a sender and

a receiver, as what the sender tries to express or communicate

when they send a message to a receiver, and what the receiver

ends up understanding or deducing from the context of the

received message. Also from a non-linguistic standpoint, things

like body language, prior experiences, and psychological effects

are contributors to meaning of language, where each human

being perceives or infers meaning in their own way, taking into

account some of these factors.

• The use of language is more concerned with how language is used

as an entity in various scenarios and communication between

human beings. This includes analyzing speech and the usage of

language when speaking, including the speaker’s intent, tone,

content and actions involved in expressing a message. This is often

termed as a speech act in linguistics. More advanced concepts such

as the origins of language creation and human cognitive activities

such as language acquisition which is responsible for learning and

usage of languages are also of prime interest.

• Language cognition specifically focuses on how the cognitive

functions of the human brain are responsible for understanding

and interpreting language. Considering the example of a typical

sender and receiver, there are many actions involved from

message communication to interpretation. Cognition tries to find

out how the mind works in combining and relating specific words

into sentences and then into a meaningful message and what is

the relation of language to the thought process of the sender and

receiver when they use the language to communicate messages.

• The relationship between language and reality explores the

extent of truth of expressions originating from language. Usually,

philosophers of language try to measure how factual these

expressions are and how they relate to certain affairs in our world

which are true. This relationship can be expressed in several ways,

and we will explore some of them.

CHAPTER 1 ■ NATURAL LANGUAGE BASICS

One of the most popular models is the triangle of reference , which is used to explain

how words convey meaning and ideas in the minds of the receiver and how that meaning

relates back to a real world entity or fact. The triangle of reference was proposed by

Charles Ogden and Ivor Richards in their book, The Meaning of Meaning , first published

in 1923, and is denoted in Figure

1-1 .

The triangle of reference model is also known as the meaning of meaning model,

and I have depicted the same in Figure1-1 with a real example of a couch being perceived

by a person which is present in front of him. A symbol is denoted as a linguistic symbol,

like a word or an object that evokes thought in a person’s mind. In this case, the symbol

is the couch, and this evokes thoughts like what is a couch, a piece of furniture that can

be used for sitting on or lying down and relaxing, something that gives us comfort . These

thoughts are known as a reference and through this reference the person is able to relate it

to something that exists in the real world, termed a referent. In this case the referent is the

couch which the person perceives to be present in front of him.

The second way to find out relationships between language and reality is known as

the direction of fit , and we will talk about two main directions here. The word-to-world

direction of fit talks about instances where the usage of language can reflect reality. This

indicates using words to match or relate to something that is happening or has already

happened in the real world. An example would be the sentence The Eiffel Tower is really

big, which accentuates a fact in reality. The other direction of fit, known as world-to-word ,

talks about instances where the usage of language can change reality. An example here

would be the sentence I am going to take a swim , where the person I is changing reality

by going to take a swim by representing the same in the sentence being communicated.

Figure

1-2 shows the relationship between both the directions of fits.

Figure 1-1. The triangle of reference model

剩余396页未读，继续阅读

ramissue

粉丝: 354
资源: 1487

Python文本分析实战：获取数据洞察力

Applied_Text_Analysis_with_Python

Text Analytics with Python, 2nd Edition.epub

Text Analytics with Python, 2nd Edition.pdf

Foundations.for.Analytics.with.Python.1491922532.epub

Mastering.Data.Mining.with.Python.1785889958

Text Analytics with Python

text analytics with python

text-analytics-w-python-master.zip_Text Analytics_python_tightds

text-analytics-with-python-master.zip

Text Analytics with Python new

最新资源