互联网挖掘技术：从超文本数据中发现知识

需积分: 10 125 浏览量更新于2024-08-02 收藏 1.72MB PDF 举报

"Mining_the_Web.pdf 是一本关于数据挖掘在超文本数据中的应用的英文资料，作者Soumen Chakrabarti。这本书属于Morgan Kaufmann Series in Data Management Systems系列，由Jim Gray编辑。书中深入探讨了如何从互联网的海量信息中发现知识，涉及搜索引擎、互联网数据挖掘等相关技术。" 《Mining the Web》一书全面覆盖了互联网数据挖掘的关键概念和技术，旨在从网络的超文本数据中提取有价值的信息和知识。以下是对书中部分内容的详细阐述： 1. **搜索引擎**：书中详细介绍了搜索引擎的工作原理，包括网页爬取（网络抓取）、索引构建、查询处理和排序算法等。搜索引擎通过爬虫技术遍历互联网，抓取网页并存储在服务器上。索引构建是关键步骤，它允许快速定位相关文档，而查询处理和排序算法则确保用户得到最相关的搜索结果。 2. **数据挖掘**：数据挖掘是本书的核心，它包括预处理、模式发现、评估和知识表示等步骤。预处理涉及数据清洗、转换和归一化，以去除噪声和不一致性。模式发现使用各种算法（如关联规则学习、聚类、分类和序列模式挖掘）来揭示隐藏的结构和规律。评估确保发现的模式具有统计显著性和实际意义，而知识表示则将这些模式转化为人类可理解的形式。 3. **互联网数据的特点**：互联网数据具有大规模、异构性、动态性和不确定性等特点。作者讨论了如何适应这些特点，设计有效的数据挖掘方法，如分布式计算、流式数据处理和半结构化数据处理。 4. **Web结构分析**：书中可能涵盖网页链接分析，如PageRank算法，这是Google搜索引擎的重要组成部分。PageRank通过计算网页间的链接关系来评估其重要性，为搜索结果的排序提供依据。 5. **文本挖掘与自然语言处理**：文本挖掘是互联网数据挖掘的重要部分，涉及到词汇分析、主题建模和情感分析等。自然语言处理技术用于理解和解析人类语言，帮助机器理解网页内容。 6. **应用案例**：书中可能包含实际应用案例，如推荐系统、社交网络分析和网络行为预测，展示数据挖掘技术如何在现实世界中解决问题。 7. **未来趋势**：作者还可能讨论了数据挖掘和互联网技术的未来发展方向，如深度学习在搜索引擎优化中的应用、大数据时代的挑战以及隐私保护和安全问题。通过阅读《Mining the Web》，读者不仅可以了解数据挖掘的基础理论，还能掌握实际操作技巧，从而在面对互联网的海量信息时，能够有效地提取、分析并利用这些知识。

PREFACE

This book is about ﬁnding signiﬁcant statistical patterns relating hypertext docu-

ments, topics, hyperlinks, and queries and using these patterns to connect users to

information they seek. The Web has become a vast storehouse of knowledge, built

in a decentralized yet collaborative manner. It is a living, growing, populist, and

participatory medium of expression with no central editorship. This has positive

and negative implications. On the positive side, there is widespread participation

in authoring content. Compared to print or broadcast media, the ratio of content

creators to the audience is more equitable. On the negative side, the heterogeneity

and lack of structure makes it hard to frame queries and satisfy information needs.

For many queries posed with the help of words and phrases, there are thousands

of apparently relevant responses, but on closer inspection these turn out to be

disappointing for all but the simplest queries. Queries involving nouns and noun

phrases, where the information need is to ﬁnd out about the named entity, are the

simplest sort of information-hunting tasks. Only sophisticated users succeed with

more complex queries—for instance, those that involve articles and prepositions

to relate named objects, actions, and agents. If you are a regular seeker and user

of Web information, this state of affairs needs no further description.

Detecting and exploiting statistical dependencies between terms, Web pages, and

hyperlinks will be the central theme in this book. Such dependencies are also called

patterns, and the act of searching for such patterns is called machine learning,ordata

mining. Here are some examples of machine learning for Web applications. Given

a crawl of a substantial portion of the Web, we may be interested in constructing

a topic directory like Yahoo!, perhaps detecting the emergence and decline of

prominent topics with passing time. Once a topic directory is available, we may

wish to assign freshly crawled pages and sites to suitable positions in the directory.

In this book, the data that we will “mine” will be very rich, comprising text,

hypertext markup, hyperlinks, sites, and topic directories. This distinguishes the

area of Web mining as a new and exciting ﬁeld, although it also borrows liberally

from traditional data analysis. As we shall see, useful information on the Web is

accompanied by incredible levels of noise, but thankfully, the law of large numbers

kicks in often enough that statistical analysis can make sense of the confusion. Our

xvi Preface

goal is to provide both the technical background and tools and tricks of the trade

of Web content mining, which was developed roughly between 1995 and 2002,

although it continues to advance. This book is addressed to those who are, or

would like to become, researchers and innovative developers in this area.

Prerequisites and Contents

The contents of this book are targeted at fresh graduate students but are also

quite suitable for senior undergraduates. The book is partly based on tutorials at

SIGMOD 1999 and KDD 2000, a survey article in SIGKDD Explorations, invited

lectures at ACL 1999 and ICDT 2001, and teaching a graduate elective at IIT

Bombay in the spring of 2001. The general style is a mix of scientiﬁc and statistical

programming with system engineering and optimizations. A background in

elementary undergraduate statistics, algorithms, and networking should sufﬁce

to follow the material. The exposition also assumes that the reader is a regular

user of search engines, topic directories, and Web content in general, and has

some appreciation for the limitations of basic Web access based on clicking on

links and typing keyword queries.

The chapters fall into three major parts. For concreteness, we start with some

engineering issues: crawling, indexing, and keyword search. This part also gives

us some basic know-how for efﬁciently representing, manipulating, and analyzing

hypertext documents with computer programs. In the second part, which is the

bulk of the book, we focus on machine learning for hypertext: the art of creating

programs that seek out statistical relations between attributes extracted from Web

documents. Such relations can be used to discover topic-based clusters from a

collection of Web pages, assign a Web page to a predeﬁned topic, or match a

user’s interest to Web sites. The third part is a collection of applications that draw

upon the techniques discussed in the ﬁrst two parts.

To make the presentation concrete, speciﬁc URLs are indicated throughout,

but there is no saying how long they will remain accessible on the Web. Luckily,

the Internet Archive will let you view old versions of pages at www.archive.org/,

provided this URL does not get dated.

Omissions

The ﬁeld of research underlying this book is in rapid ﬂux. A book written at this

juncture is guaranteed to miss out on important areas. At some point a snapshot

Acknowledgments xvii

must be taken to complete the project. A few omissions, however, are deliberate.

Beyond bare necessities, I have not engaged in a study of protocols for representing

and transferring content on the Internet and the Web. Readers are assumed to be

reasonably familiar with HTML. For the purposes of this book, you do not need

to understand the XML (Extensible Markup Language) standard much more deeply

than HTML. There is also no treatment of Web application services, dynamic site

management, or associated networking and data-processing technology.

I make no attempt to cover natural language (NL) processing, natural lan-

guage understanding, or knowledge representation. This is largely because I do

not know enough about natural language processing. NL techniques can now

parse relatively well-formed sentences in many languages, disambiguate polyse-

mous words with high accuracy, tag words in running text with part-of-speech

information, represent NL documents in a canonical machine-usable form, and

perform NL translation. Web search engines have been slow to embrace NL pro-

cessing except as an explicit translation service. In this book, I will make occasional

references to what has been called “ankle-deep semantics”—techniques that lever-

age semantic databases (e.g., as a dictionary or thesaurus) in shallow, efﬁcient ways

to improve keyword search.

Another missing area is Web usage mining. Optimizing large, high-ﬂux Web

sites to be visitor-friendly is nontrivial. Monitoring and analyzing the behavior of

visitors in the past may lead to valuable insights into their information needs, and

help in continually adapting the design of the site. Several companies have built

systems integrated with Web servers, especially the kind that hosts e-commerce

sites, to monitor and analyze trafﬁc and propose site organization strategies. The

array of techniques brought to bear on usage mining has a large overlap with

traditional data mining in the relational data-warehousing scenario, for which

excellent texts already exist.

Acknowledgments

I am grateful to many people for making this work possible. I was fortunate to

associate with Byron Dom, Inderjit Dhillon, Dharmendra Modha, David Gibson,

Dimitrios Gunopulos, Jon Kleinberg, Kevin McCurley, Nimrod Megiddo, and

Prabhakar Raghavan at IBM Almaden Research Center, where some of the

inventions described in this book were made between 1996 and 1999. I also

acknowledge the extremely stimulating discussions I have had with researchers

at the then Digital System Research Center in Palo Alto, California: Krishna

剩余363页未读，继续阅读

xinheblue

粉丝: 15
资源: 8

互联网挖掘技术：从超文本数据中发现知识

[Mining.the.Social.Web(2nd,2013.10)].Matthew.A.Russell.文字版.pdf

Mining of Massive Datasets.pdf

Data Mining The Web Uncovering Patterns In Web Content Structure And Usage

Mining the Web-Discovering Knowledge from Hypertext Data

社交网站数据挖掘与分析[Mining the Social Web](英文完整+中文样章+源代码

Web and Big Data_First International Joint Conference, Part I-Springer(2017).pdf

Professional Mobile Web Development with WordPress, Joomla! and Drupal.pdf

Python Web Scraping Cookbook-Packt Publishing(2018).pdf )

Data Mining: The Textbook (Springer 原版超清)

Data Mining: The Textbook (Springer 2015原版超清)

最新资源