大数据挖掘：Web海量数据处理与网络分析

需积分: 10 87 浏览量更新于2024-07-25 收藏 2.31MB PDF 举报

《大规模数据挖掘：互联网上的分布式处理与实践》这本书由Anand Rajaraman、Jure Leskovec和Jeffrey D. Ullman三位作者共同编著，版权于2010年至2012年，主要源于斯坦福大学多年来的课程开发。起初，这门课程CS345A名为"Web Mining"，是一门高级研究生课程，但因其深度和广泛性，吸引了许多高级本科生的兴趣。随着Jure Leskovec的加入，课程内容得到了进一步深化和整合。书中核心内容聚焦于大规模数据挖掘，即处理那些超出传统内存容量的数据集。由于其重点在于大数据，书中的实例多涉及互联网及相关数据源，如网页抓取的数据。作者们将课程扩展至多个层次，包括网络分析（在课程CS224W中），以及大规模数据挖掘项目课程CS341。因此，本书的内容涵盖了数据挖掘的基本原理、技术以及在实际场景中如何处理海量数据的实践经验。书中详细阐述了以下几个关键知识点： 1. 数据挖掘基础：介绍了数据挖掘的基本概念，包括其定义、目标和常用的方法论，如关联规则学习、分类、聚类和预测等。 2. 大规模数据处理技术：讨论了如何在分布式环境中有效地处理大规模数据，涉及Hadoop、Spark等分布式计算框架，以及MapReduce编程模型的应用。 3. Web数据获取与预处理：讲解如何通过网络爬虫技术收集数据，并对原始数据进行清洗、格式化和转换，以便后续分析。 4. 网络分析：深入探讨了社交网络、网页链接结构等网络数据的特性和分析方法，如社区检测、PageRank算法等。 5. 实时和流数据挖掘：针对不断增长的数据流，介绍了如何实现实时数据处理和在线学习的策略。 6. 案例研究与项目实践：书中提供了丰富的案例研究，展示了如何将理论应用于实际问题，如电子商务推荐系统、搜索引擎优化等。 7. 伦理与社会影响：讨论了数据挖掘的潜在风险和伦理问题，强调了隐私保护和数据使用的透明度。《大规模数据挖掘：互联网大规模数据挖掘与分布式处理》是一本深入浅出的教材，不仅适合研究生学习，也对大数据领域的专业人士具有很高的参考价值，帮助读者掌握处理和挖掘海量数据的实用技巧和策略。

xvi CONTENTS

10.7.7 Approximating the Sizes of Neighborhoods . . . . . . . . 366

10.7.8 Exercises for Section 1 0.7 . . . . . . . . . . . . . . . . . . 367

10.8 Summary of Chapter 10 . . . . . . . . . . . . . . . . . . . . . . . 368

10.9 References for Chapter 1 0 . . . . . . . . . . . . . . . . . . . . . . 371

2 CHAPTER 1. DATA MINING

and standard deviation of this Gaussian distribution completely characterize the

distribution and would become the model of the data. 2

1.1.2 Machine Learning

There are some who regard data mining as s ynonymous with machine learning.

There is no question that some data mining appropriately uses algorithms from

machine learning. Machine-learning practitioners use the data as a training set,

to train an algorithm of one of the many types used by machine-learning prac-

titioners, such a s Bayes nets, suppo rt-vector machines, decision tree s, hidden

Markov models, and many others.

There are situations where using da ta in this way makes sense. The typical

case where machine learning is a good approach is when we have little idea of

what we are looking for in the data. For example, it is rather unclear what

it is about movies that makes certain movie-goers like or dislike it. Thus,

in answering the “Netﬂix challenge” to devise an algorithm that predicts the

ratings of movies by users, based on a sample of their responses, machine-

learning algorithms have proved quite successful. We shall discuss a simple

form of this type of algorithm in Section 9.4.

On the o ther hand, machine learning has not proved successful in situations

where we can describe the goals of the mining more directly. An interesting

case in point is the a ttempt by WhizBang! Labs

to use machine learning to

locate people’s resumes on the Web. It was not able to do b etter than algorithms

in the typical resume. Since everyone who has looked at or written a resume has

a pretty good idea of what resumes contain, there was no mystery about what

makes a Web page a resume. Thus, there was no advantage to machine-learning

over the direct design of an algorithm to discover resumes.

1.1.3 Computational Approaches to Modeling

More recently, computer scientists have looked at data mining as an algorithmic

problem. In this case, the model of the data is simply the answer to a complex

query about it. For instance, given the set of numbers of Ex ample 1.1, we might

compute their average and standar d deviation. Note tha t these values might

not be the parameters of the Gaussian that best ﬁts the data, although they

will almost certainly be very close if the size of the data is lar ge.

There are many diﬀerent approaches to modeling data. We have already

mentioned the possibility of constructing a statistical process whereby the data

could have been generated. Most other approaches to mo deling can be described

as either

1. Summarizing the data succinctly and approximately, or

This startup attempted to use machine learning to mine large-scale data, and hired many

of the top machine-learning people to do so. Unfortunately, it was not able to survive.

4 CHAPTER 1. DATA MINING

The cases clus tered around some of the intersections of roads. These inter-

sections were the locatio ns of wells that had become contaminated; people who

lived nearest these wells got sick, while people who lived neare r to wells that

had not been contaminated did not get sick. Without the ability to cluster the

data, the cause of Cholera would not have been discovered. 2

1.1.5 Feature Extraction

The typical fea tur e -based model looks for the most extre me examples of a phe-

nomenon and represents the data by these ex amples. If you are familiar with

Bayes nets, a branch of machine learning and a topic we do not cover in this

book, you know how a complex relationship b etween objects is represented by

ﬁnding the stronge st statistical dependencies among these objects and using

only those in representing all statistical connections. Some of the important

kinds of feature extraction from large-scale data that we shall study are:

1. Frequent Itemsets. This model makes sense for data that consists of “bas-

kets” of small sets of items, a s in the mar ket-basket problem that we shall

discuss in Chapter 6. We look for sma ll se ts of items that appear toge ther

in ma ny baskets, and these “frequent itemsets” are the characterization of

the data that we seek. The original application of this sort of mining was

true market baskets: the sets of items, such as hamburger and ketchup,

that people tend to buy together when checking out at the cash register

of a sto re or s uper market.

2. Similar Items. Often, your data looks like a collection of sets, and the

objective is to ﬁnd pairs of sets that have a relatively large fraction of

their elements in common. An example is trea ting customers at an on-

line store like Amazon as the set of items they have bought. In order

for Amazon to recommend something else they might like, Amazon can

look for “simila r” customers and recommend something many of these

customers have bought. This process is called “collaborative ﬁltering.”

If customers were single -minded, that is, they bought only one kind of

thing, then clustering customers might work. However, since customers

tend to have interests in many diﬀerent things, it is more useful to ﬁnd,

for each customer, a small number of other customers who are similar

in their tastes, a nd represent the da ta by these connections. We discuss

similarity in Chapter 3.

1.2 Statistical Limits on Data Mining

A common sort of data-mining problem involves discovering unusual events

hidden within massive amounts of data. This section is a discussion of the

problem, including “Bonferroni’s Principle,” a warning against overzealous use

of data mining.

剩余397页未读，继续阅读

x41_zxs

粉丝: 0
资源: 5

大数据挖掘：Web海量数据处理与网络分析

SB_release1.rar_SB_稀疏分类_稀疏贝叶斯_稀疏贝叶斯SB

PMC-SB.rar_PLC_PMC SB7_PMC-RA_pmc_斗山DB130

sb165.zip_Lyapunov_典型相关分析

SB Hacks Contact Finder-crx插件

Scratch 探险游戏：深层发掘.sb3

sb_all_smartESB新特性介绍_源码.zip

安卓逆向助手软件(资源名称不得少于十一字sbCSDN)

化探数据处理方法及应用效果对比研究.pdf

数据分析Matlab实现粗糙集集约简的算法.zip

旅行行为分析：基于采样密度的聚类算法SB-DBSCAN

最新资源