数据挖掘实战：高清英文电子书指南

需积分: 10 33 浏览量更新于2024-07-18 收藏 12.9MB PDF 举报

"Making Sense of Data_ A Practical Guides and Data Mining.pdf" 这本电子书《Making Sense of Data: A Practical Guide to Data Visualization, Advanced Data Mining Methods, and Applications》由Glenn J. Myatt和Wayne P. Johnson合著，旨在帮助读者理解和应用数据挖掘技术。书中涵盖了数据可视化、高级数据挖掘方法以及它们在实际中的应用。出版于2009年，由John Wiley & Sons, Inc.发布，并同时在加拿大发行。数据挖掘是信息技术领域的一个关键分支，它涉及从大量数据中提取有用信息的过程。本书可能深入讨论了数据预处理（如清洗、转换和整合）、模式发现（包括分类、聚类、关联规则学习）以及预测模型构建等核心概念。此外，数据可视化是数据分析过程中的一个重要环节，因为它使复杂的数据更容易理解，可能书中会介绍如何利用图表、仪表板和其他视觉工具来有效地展示数据。在数据挖掘部分，读者可能会接触到诸如决策树、随机森林、支持向量机、神经网络、贝叶斯网络等机器学习算法。这些方法在预测分析、客户细分、欺诈检测、市场篮子分析等领域有广泛应用。作者可能会通过案例研究和实际项目来解释这些技术如何工作，以及如何将它们应用于业务问题中。高级数据挖掘方法可能包括非监督学习、半监督学习和强化学习，这些方法处理的是无标签数据或部分有标签数据，以及动态环境下的决策问题。此外，书中可能还探讨了大数据和云计算在现代数据挖掘中的作用，以及如何利用Hadoop、Spark等工具进行大规模数据处理。在数据可视化方面，书中可能讲解了如何选择合适的图表类型，如直方图、散点图、折线图、热力图等，以及如何使用颜色、比例和标签来增强可视化效果。此外，书中可能还会提到交互式可视化工具，如Tableau和D3.js，这些工具可以帮助用户探索和解释复杂数据集。关于版权信息，书中明确指出未经许可，不得复制、存储、传输部分内容，除非符合美国1976年版权法第107条或第108条的规定，或者通过支付适当的复制费得到Copyright Clearance Center的授权。如果需要使用书中内容，应事先向出版社申请权限。这本书为读者提供了一个全面了解和实践数据挖掘及数据可视化的平台，无论是初学者还是经验丰富的专业人士，都能从中受益。通过阅读和实践书中的方法，读者可以提升自己的数据分析能力，更好地从数据中挖掘价值并进行有效的决策。

about how the data was collected as well as the meaning of the data is invaluable.

Certain assumptions may have been built into the data, for example speciﬁc values

may have particular meanings. Or certain variables may have been derived from

others, and it will be important to understand how they were derived. Having a

thorough understanding of the subject matter pertaining to the data set helps to

explain why speciﬁc relationships are present and what these relationships mean.

(As an aside, throughout this book variables are presented in italics.)

An important initial categorization of the variables is the scale on which they

are measured. Nominal and ordinal scales refer to variables that are categorical,

that is, they have a limited number of possible values. The difference is that ordinal

variables are ordered. The variable color which could take values black, white, red,

and so on, would be an example of a nominal variable. The variable sales, whose

values are low, medium, and high, would be an example of an ordinal scale, since

there is an order to the values. Interval and ratio scales refer to variables that can

take any continuous numeric value; however, ratio scales have a natural zero value,

allowing for a calculation of a ratio. Temperature measured in Fahrenheit or Celsius

is an example of an interval scale, as it can take any continuous value within a

range. Since a zero value does not represent the absence of temperature, it is classiﬁed

as an interval scale. However, temperatures measured in degrees Kelvin would be an

exampleof a ratioscale, since zero isthe lowest temperature. In addition, a bank balance

would be an example of a ratio scale, since zero means no value.

In addition to describing the scale on which the individual variables were

measured, it is also important to understand the frequency distribution of the variable

(in the case of interval or ratio scaled variables) or the various categories that a nom-

inal or ordinal scaled variable may take. Variables are usually examined to understand

the following:

†

Central Tendency: A number of measures for the central tendency of a variable

can be calculated, including the mean or average value, the median or the

middle number based on an ordering of all values, and the mode or the most

common value. Since the mean is sensitive to outliers, the trimmed mean

may be considered which refers to a mean calculated after excluding extreme

values. In addition, median values are often used to best represent a central

value in situations involving outliers or skewed data.

†

Variation: Different numbers show the variation of the data set’s distribution.

The minimum and maximum values describe the entire range of the variable.

Calculating the values for the different quartiles is helpful, and the calculation

determines the points at which 25% (Q1), 50% (Q2), and 75% (Q3) are found

in the ordered values. The variance and standard deviation are usually calculated

to quantify the data distribution. Assuming a normal distribution, in the case of

standard deviation, approximately 68% of all observations fall within one stan-

dard deviation of the mean, and approximately 95% of all observations fall

within two standard deviations of the mean.

†

Shape: There are a number of metrics that deﬁne the shape and symmetry

of the frequency distribution, including skewness, a measure of whether a vari-

able is skewed to the left or right, and kurtosis, a measure of whether a variable

has a ﬂat or pointed central peak.

CHAPTER 1 INTRODUCTION

Graphs help to visualize the central tendency, the distribution, and the shape of

the frequency distribution, as well as to identify any outliers. A number of graphs

that are useful in summarizing variables include: frequency histograms, bar charts,

frequency polygrams, and box plots. These visualizations are covered in detail in

the section on univariate visualizations in Chapter 2.

Figure 1.1 illustrates a series of statistics calculated for a particular variable

( percentage body fat). In this example, the variable contains 251 observations, and

the most commonly occurring value is 20.4 (mode), the median is 19.2, and the aver-

age or mean value is 19.1. The variable ranges from 0 to 47.5, with the point at which

25% of the ordered values occurring at 12.4, 50% at 19.2 (or median), and 75%

at 25.3. The variance is calculated to be 69.6, and the standard deviation at 8.34,

that is, approximately 68% of observations occur +8.34 from the mean (10.76–

28.44), and approximately 95% of observations occur +16.68 from the mean

(2.42–35.78).

At this point it is worthwhile taking a digression to explain terms used for

the different roles variables play in building a prediction model. The response vari-

able, also referred to as the dependent variable, the outcome,ory-variable, is the vari-

able any model will attempt to predict. Independent variables, also referred to as

descriptors, predictors,orx-variables, are the ﬁelds that will be used in building

the model. Labels, also referred to as record identiﬁcation,orprimary key,isa

unique value corresponding to each individual row in the table. Other variables

may be present in the table that will not be used in any model, but which can still

be used in explanations.

During this stage it is also helpful to begin exploring the data to better under-

stand its features. Summary tables, matrices of different graphs, along with interactive

techniques such as brushing, are critical data exploration tools. These tools are

described in Chapter 2 on data visualization. Grouping the data is also helpful to

understand the general categories of observations present in the set. The visualization

of groups is presented in Chapter 2, and an in-depth discussion of clustering and

grouping methods is provided in Chapter 3.

Figure 1.1 Descriptive statistics and a histogram

1.3 PREPARATION 5

1.3.5 Data Cleaning

Having extracted a table containing observations (represented as rows) and variables

(represented as columns), the next step is to clean the data table, which often takes a

considerable amount of time. Some common cleaning operations include identifying

(1) errors, (2) entries with no data, and (3) entries with missing data. Errors and

missing values may be attributable to the original collection, the transmission of

the information, or the result of the preparation process.

Values are often missing from the data table, but a data mining approach cannot

proceed until this issue is resolved. There are ﬁve options: (1) remove the entire

observation from the data table; (2) remove the variable (containing the missing

values) from the data table; (3) replace the missing value manually; (4) replace the

value with a computed value, for example, the variable’s mean or mode value; and

(5) replace the entry with a predicted value based on a generated model using

other ﬁelds in the data table. Different approaches for generating predictions are

described in Chapter 4 on Predictive Analytics. The choice depends on the data set

and the problem being addressed. For example, if most of the missing values are

found in a single variable, then removing this variable may be a better option than

removing the individual observations.

A similar situation to missing values occurs when a variable that is intended to

be treated as a numeric variable contains text values, or speciﬁc numbers that have

speciﬁc meanings. Again, the ﬁve choices previously outlined above may be used;

however, the text or the speciﬁc number value may suggest numeric values to replace

them with. Another example is a numeric variable where values below a threshold

value are assigned a text string such as “,10



29.” A solution for this case

might be to replace the string with the number 0.000000001.

Another problem occurs when values within the data tables are incorrect. The

value may be problematic as a result of an equipment malfunction or a data entry

error. There are a number of ways to help identify errors in the data. Outliers in

the data may be errors and can be found using a variety of methods based on the vari-

able, for example, calculating a z-score for each value that represents the number of

standard deviations the value is away from the mean. Values greater than plus or

minus three may be considered outliers. In addition, plotting the data using a box

plot or a frequency histogram can often identify data values that signiﬁcantly deviate

from the mean. For variables that are particularly noisy, that is they contain some

degree of errors, replacing the variable with a binned version that more accurately rep-

resents the variation of the data may be necessary. This process is called data smooth-

ing. Other methods, such as data visualization, clustering, and regression models

(described in Chapters 2–4) can also be useful to identify anomalous observations

that do not look similar to other observations or that do not ﬁt a trend observed for

the majority of the variable’s observations.

Looking for values that deviate from the mean works well for numeric variables;

however, a different strategy is required to handle categorical data, especially where all

datavalues are nonnumeric. Looking at the list of all possible values a variable can take

helps to eliminate and/or consolidate values where more than one value has the same

meaning, which might happen, for example, in a categorical variable. Even though a

CHAPTER 1 INTRODUCTION

data value may look different from other values in the variable, the data may, in fact, be

correct, so it is important to consult with an expert.

Problems can also arise when data from multiple sources is integrated and incon-

sistencies are introduced. Different sources may have values for the same variables;

however, the values may have been recorded using different units of measurement

and hence must be standardized to a single unit of measurement. Different sources

of data may contain the same observation. Where the same observation has the same

values for all variables, removing one of the observations is the most straightforward

approach. Where the observations have different values, choosing which observation

to keep is more challenging and best decided bysomeone who is able to assess the most

trusted source. Other common problems when dealing with integrated data concern

assessing how up-to-date the observations are and whether the quality is the same

across different sources of data. Where observations are taken from different sources,

retaining information on the source for future reference is prudent.

1.3.6 Transformation

In many situations, it is necessary to create new variables from existing columns of

data to reﬂect more closely the purpose of the project or to enhance the quality of

the predictions. For example, creating a new column age from an existing column

date of birth, or computing an average from a series of experimental runs might be

helpful. The data may also need to be transformed in order to be used with a particular

analysis technique. There are six common transformations:

1. Creating Dummy Variables: A variable measured on a nominal or ordinal scale

is usually converted into a series of dummy variables for use within data mining

methods that require numbers. Each category is usually converted to a variable

with one of two values: a one when the value is present in the observation and

a zero when it is absent. Since this method would generate a new variable for

each category, care should be taken when using all these columns with various

methods, such as multiple linear regression or logistic regression (discussed in

Chapter 4). These methods are sensitive to issues relating to colinearity (a

high degree of correlation between variables), and hence including all variables

would introduce a problem for these methods. When a ﬁnal variable can be

deduced from the other variables, there is no need to include the ﬁnal variable.

For example, the variable color whose values are black, white, and red could

be translated into three dummy variables, one for each of the three values.

Each observation would have a value one for the color corresponding to the

row, and zero corresponding to the other two colors. Since the red column can

be derived from the other two columns, only black and white columns are

needed. The use of dummy variables is illustrated in the case studies in

Chapter 5.

2. Reducing the Number of Categories: A categorical variable may be comprised

of many different values, and using the variable directly may not draw any

meaningful conclusions; however, generalizing the values may generate

useful conclusions. This can be achieved through a manual deﬁnition of a

1.3 PREPARATION 7

concept hierarchy or assisted using automated approaches. References in the

further readings section of this chapter discuss this further, along with

Appendix B (Software). For example, a variable comprising street names

may be more valuable if it is generalized to the town containing those streets.

This may be achieved through the construction of a concept hierarchy, where

individual street names map on to the town names. In this case, there will be

more observations for a particular town which hopefully result in more

interesting conclusions.

3. Create Bins for Continuous Variables: To facilitate the use of a continuous

variable within methods that require categorical variables (such as the associ-

ation rules method), or to perform data smoothing, a continuous variable

could be divided into a series of contiguous ranges or bins. Each of the obser-

vation’s values would then be assigned to a speciﬁc bin, and potentially assigned

a value such as the bin’s mean. For example, a variable temperature with values

ranging from 0 to 100, may be divided into a series of bins: 0–10, 10–20, and so

on. A value could be assigned as each bin’s mid-point. There are a variety of

manual or automated approaches, and references to them are provided in the

further readings section of this chapter, as well as in cases in Chapter 5

(Applications) and Appendix B (Software).

4. Mapping the Data to a Normal Distribution: Certain modeling approaches

require that the frequency distribution of the variables approximate a normal

distribution, or a bell-shaped curve. There are a number of common transform-

ations that can be applied to a variable to achieve this. For example, a Box-Cox

transformation or a log transformation may be used to generate a new variable

where the data more closely follows the bell-shaped curve of a normal

distribution. The Further Reading section, as well as Appendix B, provide

more details related to this subject.

5. Standardizing the Variables to a Consistent Range: In order to treat different

variables with the same weight, a scheme for normalizing the variables to the

same range is often used, such as between zero and one. Min–max, z-score,

and decimal scaling are examples of approachesto normalizing datato a speciﬁc,

common range. As an example, a data set containing the variables age and bank

account balance may be standardized using the min–max normalization to a

consistent range of zero to one. These new variables make possible the consistent

treatment of variables within methods, such as clustering, which utilizes dis-

tances between variables. If these two variables were not on a standard range,

the bank account balance variable would, for the most part, be more inﬂuential

than the age variable.

6. Calculating Terms to Enhance Prediction: To improve prediction, certain vari-

ables may be combined, or the variables may be transformed using some sort of

mathematical operation. This may, for example, allow the more accurate mod-

eling of nonlinear relationships. Some commonly used mathematical operations

include square, cube, and square root. Appendix B and the Further Reading sec-

tion of this chapter provide more details and references on this subject.

CHAPTER 1 INTRODUCTION

剩余297页未读，继续阅读

葉葉笙歌

粉丝: 17
资源: 80

数据挖掘实战：高清英文电子书指南

"北京大学研究生文本挖掘全套PPT教程：特征提取至应用技术详解

"数据挖掘教案：概念描述与数据泛化比较

"英文数据结构教学课件：03_linear list_01.pdf

Making Sense of Data A Practical Guide to Exploratory Data Analysis and Data Mining.pdf

Encyclopedia of Machine Learning and Data Mining 2nd.Edition.pdf

ccd6161_data_sheet_january_2013.pdf

Data Assimilation_Making Sense of Observations

Practical Web Scraping for Data Science.pdf

SQL_Server_2017_Licensing_Datasheet.pdf

CRC.Data.Mining.with.R.Learning.with.Case.Studies.2nd.Edition

最新资源