大数据挖掘：处理海量数据的实践与方法

需积分: 10 108 浏览量更新于2024-07-26 收藏 2.4MB PDF 举报

《大规模数据挖掘》(Mining of Massive Datasets)是一本由Anand Rajaraman、Jure Leskovec和Jeffrey D. Ullman合著的专业书籍，它源于斯坦福大学开设的一门高级研究生课程——"Web Mining"，该课程最初面向的是硕士生，但后来由于其深度和广度吸引了许多高年级本科生的兴趣。Jure Leskovec加入斯坦福后，对课程内容进行了重大重组，将网络分析纳入新的课程CS224W，并对原有CS345A（后更名为CS246）进行了扩展。本书的核心关注点在于处理大规模的数据挖掘，这里的“大规模”意味着数据量如此之大，以至于无法一次性加载到常规的内存中。因此，书中的许多实例都围绕互联网及其衍生数据展开，这反映了大数据时代的现实挑战。作者们在教学中不仅涵盖了理论知识，还引入了一门名为CS341的大规模数据挖掘项目课程，使得内容更加全面和实用。书中涉及的主题广泛，包括但不限于：数据挖掘的基本原理和技术，如频繁模式挖掘、关联规则学习、聚类分析、分类和预测等；如何设计和实现针对大规模数据集的有效算法；以及如何利用分布式计算、云计算和MapReduce等技术来处理海量数据。此外，书中还会探讨如何从互联网日志、社交网络、搜索引擎结果等复杂数据源中提取有价值的信息，以支持商业决策、用户行为分析和网络社会的研究。本书的目的是帮助读者理解和掌握如何在当今的数字世界中有效地处理、分析和提取隐藏在海量数据背后的有价值信息，从而在大数据时代发挥关键作用。无论是对数据科学专业学生，还是对需要处理大量数据的IT从业者来说，这本书都是一个宝贵的资源，提供了深入理解大规模数据挖掘实践的坚实基础。

xvi CONTENTS

10.6.5 Using Fewer Reduce Tasks . . . . . . . . . . . . . . . . . . 371

10.6.6 Exercises for Section 1 0.6 . . . . . . . . . . . . . . . . . . 371

10.7 Neighborhood Pro perties of Graphs . . . . . . . . . . . . . . . . . 373

10.7.1 Directed Gra phs and Neighborho ods . . . . . . . . . . . . 37 3

10.7.2 The Diameter of a Graph . . . . . . . . . . . . . . . . . . 374

10.7.3 Transitive Closure and Reachability . . . . . . . . . . . . 375

10.7.4 Transitive Closure Via Map-Reduce . . . . . . . . . . . . 376

10.7.5 Smart Transitive Closure . . . . . . . . . . . . . . . . . . 378

10.7.6 Transitive Closure by Graph Reduction . . . . . . . . . . 380

10.7.7 Approximating the Sizes of Neighborhoods . . . . . . . . 382

10.7.8 Exercises for Section 1 0.7 . . . . . . . . . . . . . . . . . . 383

10.8 Summary of Chapter 10 . . . . . . . . . . . . . . . . . . . . . . . 384

10.9 References for Chapter 1 0 . . . . . . . . . . . . . . . . . . . . . . 387

2 CHAPTER 1. DATA MINING

and standard deviation of this Gaussian distribution completely characterize the

distribution and would become the model of the data. 2

1.1.2 Machine Learning

There are some who regard data mining as s ynonymous with machine learning.

There is no question that some data mining appropriately uses algorithms from

machine learning. Machine-learning practitioners use the data as a training set,

to train an algorithm of one of the many types used by machine-learning prac-

titioners, such a s Bayes nets, suppo rt-vector machines, decision tree s, hidden

Markov models, and many others.

There are situations where using da ta in this way makes sense. The typical

case where machine learning is a good approach is when we have little idea of

what we are looking for in the data. For example, it is rather unclear what

it is about movies that makes certain movie-goers like or dislike it. Thus,

in answering the “Netﬂix challenge” to devise an algorithm that predicts the

ratings of movies by users, based on a sample of their responses, machine-

learning algorithms have proved quite successful. We shall discuss a simple

form of this type of algorithm in Section 9.4.

On the o ther hand, machine learning has not proved successful in situations

where we can describe the goals of the mining more directly. An interesting

case in point is the a ttempt by WhizBang! Labs

to use machine learning to

locate people’s resumes on the Web. It was not able to do b etter than algorithms

in the typical resume. Since everyone who has looked at or written a resume has

a pretty good idea of what resumes contain, there was no mystery about what

makes a Web page a resume. Thus, there was no advantage to machine-learning

over the direct design of an algorithm to discover resumes.

1.1.3 Computational Approaches to Modeling

More recently, computer scientists have looked at data mining as an algorithmic

problem. In this case, the model of the data is simply the answer to a complex

query about it. For instance, given the set of numbers of Ex ample 1.1, we might

compute their average and standar d deviation. Note tha t these values might

not be the parameters of the Gaussian that best ﬁts the data, although they

will almost certainly be very close if the size of the data is lar ge.

There are many diﬀerent approaches to modeling data. We have already

mentioned the possibility of constructing a statistical process whereby the data

could have been generated. Most other approaches to mo deling can be described

as either

1. Summarizing the data succinctly and approximately, or

This startup attempted to use machine learning to mine large-scale data, and hired many

of the top machine-learning people to do so. Unfortunately, it was not able to survive.

4 CHAPTER 1. DATA MINING

The cases clus tered around some of the intersections of roads. These inter-

sections were the locatio ns of wells that had become contaminated; people who

lived nearest these wells got sick, while people who lived neare r to wells that

had not been contaminated did not get sick. Without the ability to cluster the

data, the cause of Cholera would not have been discovered. 2

1.1.5 Feature Extraction

The typical fea tur e -based model looks for the most extre me examples of a phe-

nomenon and represents the data by these ex amples. If you are familiar with

Bayes nets, a branch of machine learning and a topic we do not cover in this

book, you know how a complex relationship b etween objects is represented by

ﬁnding the stronge st statistical dependencies among these objects and using

only those in representing all statistical connections. Some of the important

kinds of feature extraction from large-scale data that we shall study are:

1. Frequent Itemsets. This model makes sense for data that consists of “bas-

kets” of small sets of items, a s in the mar ket-basket problem that we shall

discuss in Chapter 6. We look for sma ll se ts of items that appear toge ther

in ma ny baskets, and these “frequent itemsets” are the characterization of

the data that we seek. The original application of this sort of mining was

true market baskets: the sets of items, such as hamburger and ketchup,

that people tend to buy together when checking out at the cash register

of a sto re or s uper market.

2. Similar Items. Often, your data looks like a collection of sets, and the

objective is to ﬁnd pairs of sets that have a relatively large fraction of

their elements in common. An example is trea ting customers at an on-

line store like Amazon as the set of items they have bought. In order

for Amazon to recommend something else they might like, Amazon can

look for “simila r” customers and recommend something many of these

customers have bought. This process is called “collaborative ﬁltering.”

If customers were single -minded, that is, they bought only one kind of

thing, then clustering customers might work. However, since customers

tend to have interests in many diﬀerent things, it is more useful to ﬁnd,

for each customer, a small number of other customers who are similar

in their tastes, a nd represent the da ta by these connections. We discuss

similarity in Chapter 3.

1.2 Statistical Limits on Data Mining

A common sort of data-mining problem involves discovering unusual events

hidden within massive amounts of data. This section is a discussion of the

problem, including “Bonferroni’s Principle,” a warning against overzealous use

of data mining.

剩余414页未读，继续阅读

ddly2

粉丝: 0
资源: 4

大数据挖掘：处理海量数据的实践与方法

斯坦福大学book-Mining of Massive Datasets

Anand.Rajaraman-Mining of Massive Datasets

飞思卡尔智能车平台，基于 python+CNN实现识别道路图像类型，图像处理计算轨迹线

计算机图形学之动画和模拟算法：Procedural Animation：布料模拟算法.docx

毕设项目：基于Node+VUE后台管理系统

【上交所-2024研报】精达股份2024年第三季度报告.pdf

【上交所-2024研报】宁波舟山港股份有限公司2024年第三季度报告.pdf

springboot021基于Springboot+Vue校园周边美食探索及分享平台毕业源码案例设计.zip

python021基于Python餐厅点餐系统vue前端分离毕业源码案例设计.zip

计算机视觉领域中的卷积神经网络：从基础到实践

最新资源