大规模数据挖掘：网络分析与项目实践

4星 · 超过85%的资源需积分: 10 128 浏览量更新于2024-07-27 收藏 2.31MB PDF 举报

"大规模数据挖掘" 本书《大规模数据挖掘》由Anand Rajaraman、Jure Leskovec和Jeﬀrey D. Ullman合著，源自他们在斯坦福大学多年教学的经验，主要针对的是CS345A（后来改为CS246）“网络挖掘”课程的内容。随着Jure Leskovec加入斯坦福，课程内容进一步扩展，加入了CS224W网络分析的内容，并增设了大规模数据挖掘项目课程CS341。这三门课程的知识被整合到本书中，使得它涵盖了广泛的数据挖掘主题。本书的核心是关于数据挖掘，特别是针对极其庞大的数据集进行挖掘。由于关注数据的规模，书中很多实例都与互联网或源自互联网的数据相关。在处理如此大规模的数据时，内存管理成为一个关键问题，因此书中也探讨了如何在内存受限的情况下有效地处理和分析数据。书中涵盖的要点可能包括以下几个方面： 1. 数据预处理：在大规模数据挖掘中，数据清洗、转换和归一化是必不可少的步骤。这涉及到处理缺失值、异常值以及数据标准化，以确保后续分析的有效性。 2. 数据存储和索引：对于大数据，传统的数据库管理系统可能不再适用。书中可能会介绍分布式文件系统（如Hadoop的HDFS）和列式存储结构，以及如何构建高效的索引来加速查询。 3. 数据采样和近似算法：由于数据量过大，无法一次性加载到内存，采样技术用于获取数据的代表样本，而近似算法则用于在有限计算资源下获得可接受的解决方案。 4. 网络分析：Jure Leskovec的贡献可能包括社交网络分析、链接预测和社区检测等，这些都是理解互联网结构和行为的关键工具。 5. 数据挖掘算法：书中会涉及聚类、分类、关联规则学习等经典数据挖掘算法，但会特别考虑如何在大规模数据上优化这些算法。 6. 实时和流式数据处理：随着数据的不断生成，实时分析和流处理技术变得重要。Apache Storm、Spark Streaming等工具可能是讨论的话题。 7. 大规模机器学习：深度学习和分布式机器学习算法在处理大数据时的作用不容忽视，如Google的TensorFlow和Facebook的PyTorch。 8. 实战项目：书中可能包含实际的大规模数据挖掘项目，帮助读者应用理论知识解决真实世界的问题。 9. 隐私和伦理：在处理大量个人数据时，隐私保护和伦理问题也需考虑。书里可能会讨论相关的法规和最佳实践。这本书不仅仅是理论性的介绍，还强调实践应用，通过实例和案例研究来阐述如何在现实场景中运用数据挖掘技术。无论是对研究生还是高级本科生，都将是一本有价值的参考书。

xvi CONTENTS

10.7.7 Approximating the Sizes of Neighborhoods . . . . . . . . 366

10.7.8 Exercises for Section 1 0.7 . . . . . . . . . . . . . . . . . . 367

10.8 Summary of Chapter 10 . . . . . . . . . . . . . . . . . . . . . . . 368

10.9 References for Chapter 1 0 . . . . . . . . . . . . . . . . . . . . . . 371

2 CHAPTER 1. DATA MINING

and standard deviation of this Gaussian distribution completely characterize the

distribution and would become the model of the data. 2

1.1.2 Machine Learning

There are some who regard data mining as s ynonymous with machine learning.

There is no question that some data mining appropriately uses algorithms from

machine learning. Machine-learning practitioners use the data as a training set,

to train an algorithm of one of the many types used by machine-learning prac-

titioners, such a s Bayes nets, suppo rt-vector machines, decision tree s, hidden

Markov models, and many others.

There are situations where using da ta in this way makes sense. The typical

case where machine learning is a good approach is when we have little idea of

what we are looking for in the data. For example, it is rather unclear what

it is about movies that makes certain movie-goers like or dislike it. Thus,

in answering the “Netﬂix challenge” to devise an algorithm that predicts the

ratings of movies by users, based on a sample of their responses, machine-

learning algorithms have proved quite successful. We shall discuss a simple

form of this type of algorithm in Section 9.4.

On the o ther hand, machine learning has not proved successful in situations

where we can describe the goals of the mining more directly. An interesting

case in point is the a ttempt by WhizBang! Labs

to use machine learning to

locate people’s resumes on the Web. It was not able to do b etter than algorithms

in the typical resume. Since everyone who has looked at or written a resume has

a pretty good idea of what resumes contain, there was no mystery about what

makes a Web page a resume. Thus, there was no advantage to machine-learning

over the direct design of an algorithm to discover resumes.

1.1.3 Computational Approaches to Modeling

More recently, computer scientists have looked at data mining as an algorithmic

problem. In this case, the model of the data is simply the answer to a complex

query about it. For instance, given the set of numbers of Ex ample 1.1, we might

compute their average and standar d deviation. Note tha t these values might

not be the parameters of the Gaussian that best ﬁts the data, although they

will almost certainly be very close if the size of the data is lar ge.

There are many diﬀerent approaches to modeling data. We have already

mentioned the possibility of constructing a statistical process whereby the data

could have been generated. Most other approaches to mo deling can be described

as either

1. Summarizing the data succinctly and approximately, or

This startup attempted to use machine learning to mine large-scale data, and hired many

of the top machine-learning people to do so. Unfortunately, it was not able to survive.

4 CHAPTER 1. DATA MINING

The cases clus tered around some of the intersections of roads. These inter-

sections were the locatio ns of wells that had become contaminated; people who

lived nearest these wells got sick, while people who lived neare r to wells that

had not been contaminated did not get sick. Without the ability to cluster the

data, the cause of Cholera would not have been discovered. 2

1.1.5 Feature Extraction

The typical fea tur e -based model looks for the most extre me examples of a phe-

nomenon and represents the data by these ex amples. If you are familiar with

Bayes nets, a branch of machine learning and a topic we do not cover in this

book, you know how a complex relationship b etween objects is represented by

ﬁnding the stronge st statistical dependencies among these objects and using

only those in representing all statistical connections. Some of the important

kinds of feature extraction from large-scale data that we shall study are:

1. Frequent Itemsets. This model makes sense for data that consists of “bas-

kets” of small sets of items, a s in the mar ket-basket problem that we shall

discuss in Chapter 6. We look for sma ll se ts of items that appear toge ther

in ma ny baskets, and these “frequent itemsets” are the characterization of

the data that we seek. The orignal a pplication of this sort of mining was

true market baskets: the sets of items, such as hamburger and ketchup,

that people tend to buy together when checking out at the cash register

of a sto re or s uper market.

2. Similar Items. Often, your data looks like a collection of sets, and the

objective is to ﬁnd pairs of sets that have a relatively large fraction of

their elements in common. An example is trea ting customers at an on-

line store like Amazon as the set of items they have bought. In order

for Amazon to recommend something else they might like, Amazon can

look for “simila r” customers and recommend something many of these

customers have bought. This process is called “collaborative ﬁltering.”

If customers were single -minded, that is, they bought only one kind of

thing, then clustering customers might work. However, since customers

tend to have interests in many diﬀerent things, it is more useful to ﬁnd,

for each customer, a small number of other customers who are similar

in their tastes, a nd represent the da ta by these connections. We discuss

similarity in Chapter 3.

1.2 Statistical Limits on Data Mining

A common sort of data-mining problem involves discovering unusual events

hidden within massive amounts of data. This section is a discussion of the

problem, including “Bonferroni’s Principle,” a warning against overzealous use

of data mining.

剩余397页未读，继续阅读

sunxiaojun2011

粉丝: 0
资源: 3

大规模数据挖掘：网络分析与项目实践

mining of massive datasets

Mining of massive dataset

Mining of Massive Datasets

mining_of_massive_datasets:CS 246斯坦福大学和CS 547华盛顿

Mining of massive datasets

mining of massive datasets-ch01-intro.pdf

mining-massive-datasets:此 Coursera MOOC 的作业代码

基于freeRTOS和STM32F103x的手机远程控制浴室温度系统设计源码

LABVIEW程序实例-web写数据.zip

LABVIEW程序实例-前面板对象常用属性.zip

最新资源