Hadoop权威指南第三版：深入解析

需积分: 6 103 浏览量更新于2024-07-18 收藏 10.72MB PDF 举报

"Hadoop权威指南第三版.pdf" 《Hadoop权威指南》第三版是由Tom White编写的关于Hadoop生态系统的全面教程。这本书是Hadoop学习者的必备参考书，旨在深入理解并有效地使用Hadoop框架。出版于2012年，由O'Reilly Media, Inc.发行。 Hadoop是一个开源的分布式计算框架，它允许在大规模集群上处理海量数据。这个框架的核心由两个主要部分组成：Hadoop Distributed File System (HDFS) 和 MapReduce。HDFS提供了高容错性的分布式存储，而MapReduce则用于大规模数据集的并行处理。在本书中，Tom White详细介绍了Hadoop的各个组件和相关技术，包括： 1. **Hadoop安装与配置** - 书中详细阐述了如何在不同的环境中安装和配置Hadoop，涵盖了单机模式、伪分布式模式以及完全分布式模式。 2. **HDFS** - 深入探讨了HDFS的工作原理，包括数据块、数据节点、名称节点、副本策略以及故障恢复机制。 3. **MapReduce** - 详细解释了MapReduce编程模型，包括map函数、reduce函数、shuffle和sort阶段，以及优化MapReduce作业的方法。 4. **Hadoop生态系统的扩展** - 除了HDFS和MapReduce，书中还介绍了如Hadoop流、Hadoop二次排序、Pig、Hive、HBase、Cassandra、Mahout等与Hadoop生态系统相关的工具和技术。 5. **YARN** - YARN（Yet Another Resource Negotiator）是Hadoop 2.x版本中的资源管理器，它将资源管理和任务调度分离，提高了系统效率和可扩展性，书中对其进行了详细介绍。 6. **Hadoop安全** - 讨论了Hadoop的安全特性，如 Kerberos 身份验证、访问控制列表（ACLs）以及数据加密。 7. **数据处理的最佳实践** - 提供了在Hadoop上进行大数据处理时的策略和技巧，包括数据预处理、性能优化和故障排查。 8. **实时数据分析** - 阐述了如何利用Hadoop与其他实时数据分析工具（如Spark）结合，实现快速响应的数据处理需求。 9. **案例研究** - 通过实际案例展示了Hadoop在不同行业的应用，包括互联网分析、生物信息学、金融领域等。 10. **社区与未来** - 介绍了Hadoop的社区动态，包括开发进度、版本更新，以及Hadoop可能的发展趋势。此外，书中还提供了大量的示例代码和实战指导，帮助读者更好地理解和应用Hadoop。对于想要深入了解和使用Hadoop的人来说，《Hadoop权威指南》第三版无疑是一本极具价值的参考资料。

Foreword

Hadoop got its start in Nutch. A few of us were attempting to build an open source

web search engine and having trouble managing computations running on even a

handful of computers. Once Google published its GFS and MapReduce papers, the

route became clear. They’d devised systems to solve precisely the problems we were

having with Nutch. So we started, two of us, half-time, to try to re-create these systems

as a part of Nutch.

We managed to get Nutch limping along on 20 machines, but it soon became clear that

to handle the Web’s massive scale, we’d need to run it on thousands of machines and,

moreover, that the job was bigger than two half-time developers could handle.

Around that time, Yahoo! got interested, and quickly put together a team that I joined.

We split off the distributed computing part of Nutch, naming it Hadoop. With the help

of Yahoo!, Hadoop soon grew into a technology that could truly scale to the Web.

In 2006, Tom White started contributing to Hadoop. I already knew Tom through an

excellent article he’d written about Nutch, so I knew he could present complex ideas

in clear prose. I soon learned that he could also develop software that was as pleasant

to read as his prose.

From the beginning, Tom’s contributions to Hadoop showed his concern for users and

for the project. Unlike most open source contributors, Tom is not primarily interested

in tweaking the system to better meet his own needs, but rather in making it easier for

anyone to use.

Initially, Tom specialized in making Hadoop run well on Amazon’s EC2 and S3 serv-

ices. Then he moved on to tackle a wide variety of problems, including improving the

MapReduce APIs, enhancing the website, and devising an object serialization frame-

work. In all cases, Tom presented his ideas precisely. In short order, Tom earned the

role of Hadoop committer and soon thereafter became a member of the Hadoop Project

Management Committee.

Tom is now a respected senior member of the Hadoop developer community. Though

he’s an expert in many technical corners of the project, his specialty is making Hadoop

easier to use and understand.

Preface

Martin Gardner, the mathematics and science writer, once said in an interview:

Beyond calculus, I am lost. That was the secret of my column’s success. It took me so

long to understand what I was writing about that I knew how to write in a way most

readers would understand.

In many ways, this is how I feel about Hadoop. Its inner workings are complex, resting

as they do on a mixture of distributed systems theory, practical engineering, and com-

mon sense. And to the uninitiated, Hadoop can appear alien.

But it doesn’t need to be like this. Stripped to its core, the tools that Hadoop provides

for building distributed systems—for data storage, data analysis, and coordination—

are simple. If there’s a common theme, it is about raising the level of abstraction—to

create building blocks for programmers who just happen to have lots of data to store,

or lots of data to analyze, or lots of machines to coordinate, and who don’t have the

time, the skill, or the inclination to become distributed systems experts to build the

infrastructure to handle it.

With such a simple and generally applicable feature set, it seemed obvious to me when

I started using it that Hadoop deserved to be widely used. However, at the time (in

early 2006), setting up, configuring, and writing programs to use Hadoop was an art.

Things have certainly improved since then: there is more documentation, there are

more examples, and there are thriving mailing lists to go to when you have questions.

And yet the biggest hurdle for newcomers is understanding what this technology is

capable of, where it excels, and how to use it. That is why I wrote this book.

The Apache Hadoop community has come a long way. Over the course of three years,

the Hadoop project has blossomed and spun off half a dozen subprojects. In this time,

the software has made great leaps in performance, reliability, scalability, and manage-

ability. To gain even wider adoption, however, I believe we need to make Hadoop even

easier to use. This will involve writing more tools; integrating with more systems; and

1. “The science of fun,” Alex Bellos, The Guardian, May 31, 2008, http://www.guardian.co.uk/science/

2008/may/31/maths.science.

xvii

writing new, improved APIs. I’m looking forward to being a part of this, and I hope

this book will encourage and enable others to do so, too.

Administrative Notes

During discussion of a particular Java class in the text, I often omit its package name

to reduce clutter. If you need to know which package a class is in, you can easily look

it up in Hadoop’s Java API documentation for the relevant subproject, linked to from

the Apache Hadoop home page at http://hadoop.apache.org/. Or if you’re using an IDE,

it can help using its auto-complete mechanism.

Similarly, although it deviates from usual style guidelines, program listings that import

multiple classes from the same package may use the asterisk wildcard character to save

space (for example, import org.apache.hadoop.io.*).

The sample programs in this book are available for download from the website that

accompanies this book: http://www.hadoopbook.com/. You will also find instructions

there for obtaining the datasets that are used in examples throughout the book, as well

as further notes for running the programs in the book, and links to updates, additional

resources, and my blog.

What’s in This Book?

The rest of this book is organized as follows. Chapter 1 emphasizes the need for Hadoop

and sketches the history of the project. Chapter 2 provides an introduction to

MapReduce. Chapter 3 looks at Hadoop filesystems, and in particular HDFS, in depth.

Chapter 4 covers the fundamentals of I/O in Hadoop: data integrity, compression,

serialization, and file-based data structures.

The next four chapters cover MapReduce in depth. Chapter 5 goes through the practical

steps needed to develop a MapReduce application. Chapter 6 looks at how MapReduce

is implemented in Hadoop, from the point of view of a user. Chapter 7 is about the

MapReduce programming model and the various data formats that MapReduce can

work with. Chapter 8 is on advanced MapReduce topics, including sorting and joining

data.

Chapters 9 and 10 are for Hadoop administrators and describe how to set up and

maintain a Hadoop cluster running HDFS and MapReduce.

Later chapters are dedicated to projects that build on Hadoop or are related to it.

Chapters 11 and 12 present Pig and Hive, which are analytics platforms built on HDFS

and MapReduce, whereas Chapters 13, 14, and 15 cover HBase, ZooKeeper, and

Sqoop, respectively.

Finally, Chapter 16 is a collection of case studies contributed by members of the Apache

Hadoop community.

xviii | Preface

What’s New in the Second Edition?

The second edition has two new chapters on Hive and Sqoop (Chapters 12 and 15), a

new section covering Avro (in Chapter 4), an introduction to the new security features

in Hadoop (in Chapter 9), and a new case study on analyzing massive network graphs

using Hadoop (in Chapter 16).

This edition continues to describe the 0.20 release series of Apache Hadoop because

this was the latest stable release at the time of writing. New features from later releases

are occasionally mentioned in the text, however, with reference to the version that they

were introduced in.

What’s New in the Third Edition?

The third edition covers the 1.x (formerly 0.20) release series of Apache Hadoop, as

well as the newer 0.22 and 2.x (formerly 0.23) series. With a few exceptions, which are

noted in the text, all the examples in this book run against these versions. The features

in each release series are described at a high level in “Hadoop Releases”

on page 13.

This edition uses the new MapReduce API for most of the examples. Because the old

API is still in widespread use, it continues to be discussed in the text alongside the new

API, and the equivalent code using the old API can be found on the book’s website.

The major change in Hadoop 2.0 is the new MapReduce runtime, MapReduce 2, which

is built on a new distributed resource management system called YARN. This edition

includes new sections covering MapReduce on YARN: how it works (Chapter 6) and

how to run it (Chapter 9).

There is more MapReduce material, too, including development practices such as

packaging MapReduce jobs with Maven, setting the user’s Java classpath, and writing

tests with MRUnit (all in Chapter 5); and more depth on features such as output com-

mitters, the distributed cache (both in Chapter 8), and task memory monitoring (Chap-

ter 9). There is a new section on writing MapReduce jobs to process Avro data (Chapter

4), and one on running a simple MapReduce workflow in Oozie (Chapter 5).

The chapter on HDFS (Chapter 3) now has introductions to high availability, federa-

tion, and the new WebHDFS and HttpFS filesystems.

The chapters on Pig, Hive, Sqoop, and ZooKeeper have all been expanded to cover the

new features and changes in their latest releases.

In addition, numerous corrections and improvements have been made throughout the

book.

Preface | xix

剩余684页未读，继续阅读

笨小孩Andy

粉丝: 1
资源: 6

Hadoop权威指南第三版：深入解析

Hadoop权威指南第三版：全面修订版

Hadoop权威指南第三版：应对海量数据分析挑战详解

Hadoop权威指南第三版：探索大数据分析

HADOOP权威指南第三版.pdf

Hadoop权威指南 第3版.pdf

hadoop权威指南第三版中文pdf

hadoop权威指南第三版 中文 pdf

hadoop权威指南第三版(英文).pdf

Hadoop权威指南（第三版）英文版.pdf

[Hadoop权威指南(第三版)].Tom.White.文字版.pdf

最新资源

Hadoop权威指南第3版.pdf

hadoop权威指南第三版中文 pdf