《数据分析与数据挖掘入门指南》：理解与实践

需积分: 9 153 浏览量更新于2024-07-18 收藏 4.87MB PDF 举报

《数据分析与数据挖掘：入门指南》是一本由Adelchi Azzalini和Bruno Scarpa合著的专业书籍，旨在为读者提供关于数据分析和数据挖掘领域的深入理解。本书适合那些希望在数据分析领域起步或对这两个主题有初步兴趣的人阅读，通过它，读者将能够获得一个全面且系统的知识框架。数据分析是现代IT行业中至关重要的组成部分，它涵盖了收集、处理、解释和从海量数据中提取有价值信息的过程。这本书详细介绍了数据预处理、探索性数据分析（EDA）、统计推断、回归分析、聚类分析、关联规则学习以及深度学习等关键概念和技术。对于新手来说，它不仅提供了理论基础，还通过实例演示展示了如何实际应用这些方法。数据挖掘则是在大数据时代背景下，对数据进行深入挖掘以发现隐藏模式、关联和预测的能力。作者在这本书中探讨了各种数据挖掘算法，如决策树、随机森林、支持向量机和神经网络，以及它们在商业智能、市场营销、客户行为分析等场景中的应用。此外，书中还会涉及如何评估模型性能，以及如何处理数据质量问题。《数据分析与数据挖掘：An Introduction》不仅适合计算机科学专业的学生，也对数据科学家、分析师、业务管理人员以及其他任何希望通过掌握数据处理技术提升业务决策能力的人士有极大的帮助。它强调了理论与实践的结合，让读者能够在理论学习的同时，逐步掌握如何在实际工作中实施和优化数据驱动的决策过程。在版权方面，本书由牛津大学出版社出版，强调了学术诚信和知识产权保护。阅读这本书不仅是获取知识的途径，也是尊重作者辛勤工作的体现。通过这本书，读者将开启通往复杂数据世界的大门，为未来在这个充满潜力的领域内取得成功打下坚实的基础。

Introduction 5

data mining represents the work of processing, graphically or

numerically, large amounts or continuous streams of data, with the

aim of extracting information useful to those who possess them.

The expression “useful information” is deliberately general: in many cases, the

point of interest is not speciﬁed a priori at all and we often search for it by mining

the data. This aspect distinguishes between data mining and other searches related

to data analysis. In particular, the approach is diametrically opposed, for example,

to clinical studies, in which it is essential to specify very precisely a priori the aims

for which data are collected and analyzed.

What might constitute useful information varies considerably and depends on

the context in which we operate and on the objectives we set. This observation

is clearly also true in many other contexts, but in the area of data mining it has

additional value. We can make a distinction between two situations: (a) in one, the

interesting aspect is the global behavior of the phenomenon examined, and the aim

is the construction of its global model, taken from the available data; (b) in the other,

it is characterization of detail or the pattern structures of the data, as we are only

interested in cases outside standard behavior. In the example of telephone company

customers, we can examine phone trafﬁc data to identify trends that allow us to

forecast customers’ behavior according to their price plans, geographical position,

and other known elements. However, we can also examine the data with the aim of

identifying behavioral anomalies in telephone usage with respect to the behavior of

the same customer in the past—perhaps to detect a fraudulent situation created by

a third party to a customer’s detriment.

Data mining is a recent discipline, lying at the intersection of various scientiﬁc

sectors, especially statistics, machine learning, and database management.

The connection with database management is implicit in that the operations

of data cleaning, the selection of portions of data, and so on, also drawn from

distributed databases, require competences and contributions from that sector.

The link with artiﬁcial intelligence reﬂects the intense activity in that ﬁeld to

make machines “learn” how to calculate general rules originating from a series

of speciﬁc examples: this is very like the aim of extracting the laws that regulate

a phenomenon from sampled observations. This, among the methods that are

presented later, explains why some of them originate from the ﬁeld of artiﬁcial

intelligence or similar ones.

In light of the foregoing, the statements of Hand et al. (2001) become clear:

Data mining is fundamentally an applied discipline … data mining

requires an understanding of both statistical and computational

issues. (p. xxviii)

The most fundamental difference between classical statistical appli-

cations and data mining is the size of the data. (p. 19)

The computational cost connected with large data sizes and dimensions obviously

has repercussions on the method of working with these data: as they increase,

6 DATA ANALYSIS AND DATA MINING

methods with high computational cost become less feasible. Clearly, in such cases,

we cannot identify an exact rule, because various factors other than those already

mentioned come into play, such as available resources for calculation and the time

needed for results. However, the effect unquestionably exists, and it prevents the

use of some tools, or at least renders them less practical, while favoring others of

lower computational cost.

It is also true that there are situations in which these aspects are of only marginal

importance, because the amount of data is not enough to inﬂuence the computing

element; this is partly thanks to the enormous increase in the power of computers.

We often see this situation with a large-scale problem, if it can be broken down into

subproblems, which make portions of the data more manageable. More traditional

methods of venerable age have not yet been put to rest. On the contrary, many of

them, which developed in a period of limited computing resources, are much less

demanding in terms of computational effort and are still valid if suitably applied.

1.1.3 SQL, OLTP, OLAP, DWH, and KDD

We have repeatedly mentioned the great availability of data, now collected in an

increasingly systematic and thorough way, as the starting point for processing.

However, the conversion of raw data to “clean” data is time-consuming and

sometimes very demanding.

We cannot presume that all the data of a complex organization can ﬁt into a

single database on which we can simply draw and develop. In the business world,

even medium-sized companies are equipped with complex IT systems made up

of various databases designed for various aims (customers and their invoices,

employees’ careers and wages, suppliers, etc.). These databases are used by various

operators, both to insert data (e.g., from outlying sales ofﬁces) and to answer

queries about single entries, necessary for daily activities—for example, to know

whether and when customer X has paid invoice Y issued on day Z. The phrase

referring to methods of querying speciﬁc information in various databases, called

operational,isOnLine Transaction Processing (OLTP). Typically, these tools are

based on Structured Query Language (SQL), the standard tool for database queries.

For decision support, in particular analysis of data for CRM, these operational

databases are not the proper sources on which to work. They were all designed for

different goals, both in the sense that they were usually created for administrative

and accounting purposes and not for data analysis, and that those goals differ. This

means that their structures are heterogeneous and very often contain inconsistent

data, sometimes even structurally, because the deﬁnitions of the recorded variables

may be similar but are not identical. Nor is it appropriate for the strategic activities

of decision support to interfere with daily work on systems designed to work on

operational databases.

For these reasons, it is appropriate to develop focused databases and tools. We

thus construct a strategic database or Data WareHouse (DWH), in which data

from different database systems merge, are “cleaned” as much as possible, and are

organized round the postprocessing phase.

The development of a DWH is complex, and it must be carefully designed for

its future aims. From a functional point of view, the most common method of

Introduction 7

construction is progressive aggregation of various data marts—that is, of ﬁnalized

databases. For example, a data mart may contain all the relevant information

for a certain marketing division. After the DWH has been constructed, the later

aggregation must achieve a coherent, homogenous structure, and the DWH must

be periodically updated with new data from various operational databases.

After completing all these programming processes (which can then progress by

means of continual maintenance), a DWH can be used in at least two ways, which

are not mutually exclusive. The ﬁrst recomposes data from the various original data

marts to create new ones: for example, if we have created a DWH by aggregating

data mart for several lines of products, we can create a new one for selling all those

products in a certain geographical area. A new data mart is therefore created for

every problem for which we want to develop quantitative analysis.

A second way of using a DWH, which ﬂanks the ﬁrst, directly generates

processing (albeit simpliﬁed) to extract certain information about the data

summary. This is called OnLine Analytical Processing (OLAP) and, as indicated

by its name, is made up of querying and processing designed in a certain way to be

a form of data analysis, although it is still raw and primarily descriptive.

For OLAP, the general support is a structure of intermediate processing,

called a hypercube. In statistical terms, this is a multiway table, in which every

dimension corresponds to a variable, and every cell at the intersection of different

levels contains a synthetic indicator, often a frequency. To give an example

of this, let us presume that the statistical units are university students. One

variable could be constructed by place of residence, another by department or

university membership, gender, and so on, and the individual cells of the cross-

table (hypercube) contain the frequencies for the various intersecting levels. This

table can be used for several forms of processing: marginalization or conditioning

with respect to one or more variables, level aggregation, and so on. They are

described in introductory statistical texts and need no mention here. Note that in

the ﬁeld of computer science, the foregoing operations have different names.

As already noted, OLAP is an initial form of the extraction of information

from the data—relatively simple, at least from a conceptual point of view—

operating from a table with predeﬁned variables and a scope of operations limited

to them. Therefore, strictly speaking, OLAP returns to data mining as deﬁned

in section 1.1.2, but limited to a form that is conceptually a very simple way

of processing. Instead, “data mining” commonly refers to the inspection of a

strategic database and is characteristically more investigative in nature, typically

involving the identiﬁcation of relations in certain signiﬁcant ways among variables

or making speciﬁc and interesting patterns of the data. The distinction between

OLAP and data mining is therefore not completely clear, but essentially—as

already noted—the former involves inspecting a small number of prespeciﬁed

variables and has a limited number of operations, and the latter refers to a

more open and more clearly focused study on extracting knowledge from the

data. For the latter type of processing, much more computational than simple

management, it is not convenient to use SQL, because SQL does not provide

simple commands for intensive statistical processing. Alternatives are discussed

later.

8 DATA ANALYSIS AND DATA MINING

We can now think of a chain of phases, starting as follows:

• one or more operational databases to construct a strategic database

(DWH): this also involves an operation in which we homogenize the

deﬁnition of variables and data cleaning operations;

• we apply OLAP tools to this new database, to highlight points of interest

on variables singled out previously;

• data mining is the most speciﬁc phase of data analysis, and aims at ﬁnding

interesting elements in speciﬁc data marts extracted from the DWH.

The term Knowledge Discovery in Databases (KDD) is used to refer to this

complex chain, but this terminology is not unanimously accepted and data mining

is sometimes used as a synonym. In this work, data mining is intended in the more

restricted sense, which regards only the ﬁnal phases of those described.

1.1.4 Complications

We have already touched on some aspects that differentiate data mining from other

areas of data analysis. We now elaborate this point.

In many cases, data were collected for reasons other than statistical analysis.

In particular, in the business sector, data are compiled primarily for accounting

purposes. This administrative requirement led to ways of organizing these data

becoming more complex; the realization that they could be used for other purposes,

that is, marketing analysis and CRM, came later.

Data, therefore, do not correspond to any sampling plan or experimental

design: they simply ‘exist’. The lack of canonical conditions for proper data

collection initially kept many statisticians away from the ﬁeld of data mining,

whereas information technology (IT) experts were more prompt in exploiting this

challenge.

Even without these problems, we must also consider data collected in spurious

forms. This naturally entails greater difﬁculties and corresponding attention to

other applicative contexts.

The ﬁrst extremely simple but useful observation in this sense has to do with

the validity of our conclusions. Because a company’s customer database does not

represent a random sample of the total population, the conclusions we may draw

from it cover at most already acquired customers, not prospective ones.

Another reason for the initial reluctance of statisticians to enter the ﬁeld of

data mining was a second element, already mentioned in section 1.1.2—that is,

research sometimes focuses on an objective that was not declared a priori. When

we research ‘anything’, we end up ﬁnding ‘something’

...even if it is not there. To

illustrate this idea intuitively, assume that we are examining a sequence of random

numbers: ultimately, it seems that there is some regularity, at least if we examine a

sequence that is not too long. At this point, we must recall an aphorism coined by

an economist, which is very fashionable among applied statisticians: “If you torture

the data long enough, Nature will always confess” (Ronald H. Coase, 1991 Nobel

Prize for Economics).

Introduction 9

This practice of “looking for something” (when we do not know exactly what

it is) is therefore misleading, and thus the associated terms data snooping or

data dredging have negative connotations. When confronted with a considerable

amount of data, the danger of false ﬁndings decreases but is not eliminated

altogether. There are, however, techniques to counter this, as we shall see in

chapter 3.

One particularity, which seems trivial, regards the so-called leaker variables,

which are essentially surrogates of the variables of interest. For example, if the

variable of interest is the amount of money spent on telephone trafﬁc by one

customer in one month, a leaker variable is given by the number of phone calls

made in that same month, as the ﬁrst variable is recorded at the same moment

as the second variable. Conceptually, the situation is trivial, but when hundreds of

variables, often of different origin, are manipulated, this eventuality is not as remote

as it may appear. It at least signals the danger of using technology blindly, inserting

whole lists of variables without worrying about what they represent. We return to

this point in section 1.3.1.

Bibliographical notes

Hand et al. (2001) depict a broad picture of data mining, its connections with

other disciplines, and its general principles, although they do not enter into detailed

technical aspects. In particular, their chapter 12 contains a more highly developed

explanation of our section 1.1.3 about relationships between data management and

some techniques, like OLAP, closer to that context.

For descriptive statistics regarding tables of frequency and their handling, there

is a vast amount of literature, which started in the early stages of statistics and is still

developing. Some classical texts are Kendall & Stuart (1969, sections 1.30–1.34),

Bishop et al. (1975), and Agresti (2002).

For a more detailed description of the role of data mining in the corporate

context, in particular its connections with business promotion, see the ﬁrst chapters

of Berry & Linoff (1997).

1.2 A

LL MODELS ARE WRONG

All models are wrong but some are useful.

—GEORGE E. P. BOX

1.2.1 What is a Model?

The term model is very fashionable in many contexts, mainly in the ﬁelds of science

and technology and also business management. Because the important attributes

of this term (which are often implicit) are so varied and often blurred, let us clarify

at once what we mean by it:

A model is a simpliﬁed representation of the phenomenon of interest,

functional for a speciﬁc objective.

www.allitebooks.com

剩余288页未读，继续阅读

渔好学

粉丝: 80
资源: 1

《数据分析与数据挖掘入门指南》：理解与实践

数据分析与数据挖掘算法：K-means和层次聚类算法详解【英文版】

骑行跑步GPS数据集：分析和数据挖掘的运动路线

"数据仓库与数据挖掘第9章：分类与预测方法综述

Making Sense of Data A Practical Guide to Exploratory Data Analysis and Data Mining.pdf

Data Mining and Analysis: Foundations and Algorithms(2014)

Data Mining and Analysis_Fundamental Concepts and Algorithms_2014

Data Mining and Analysis: Fundamental Concepts and Algorithms 中文 part4

Data Mining and Analysis: Fundamental Concepts and Algorithms 中文 part2

Data Mining and Analysis: Fundamental Concepts and Algorithms 英文，教材

Cluster Analysis for Data Mining and system identification

最新资源