Hadoop权威指南第三版英文原版

下载需积分: 5 | PDF格式 | 13.17MB | 更新于2024-07-18 | 72 浏览量 | 举报

"Hadoop权威指南第三版是Tom White撰写的一本详细介绍Hadoop生态系统的英文原版书籍，适合具有英文阅读能力的读者。本书由O'Reilly Media出版，并提供了丰富的在线资源支持。" 在《Hadoop权威指南第三版》中，作者Tom White深入浅出地阐述了Hadoop的核心概念和技术细节，旨在帮助读者全面理解并掌握这个分布式计算框架。以下是书中涵盖的一些关键知识点： 1. **Hadoop简介**：介绍Hadoop的历史、设计目标和架构，包括其分布式文件系统HDFS（Hadoop Distributed File System）和MapReduce计算模型。 2. **HDFS**：详述HDFS的设计原理，如数据块、副本策略、命名节点（NameNode）和数据节点（DataNode）的角色，以及如何确保高可用性和容错性。 3. **MapReduce**：讲解MapReduce的工作流程，包括map、reduce阶段以及shuffle和sort过程，同时涵盖了JobTracker和TaskTracker的调度机制。 4. **YARN**：介绍第二代资源管理系统YARN（Yet Another Resource Negotiator），它替代了原有的JobTracker，实现更精细的资源管理和任务调度。 5. **Hadoop生态**：探讨Hadoop生态系统中的其他重要组件，如HBase（分布式数据库）、Hive（数据仓库工具）、Pig（数据分析平台）、Sqoop（数据导入导出工具）和Oozie（工作流调度系统）等。 6. **开发与调试**：提供编写Hadoop程序的指导，包括Java API的使用，以及如何进行单元测试和性能调优。 7. **数据存储与处理**：讲解数据的预处理、加载和清洗，以及使用Hadoop进行大规模数据处理的方法。 8. **安全性与管理**：讨论Hadoop集群的安全机制，如Kerberos认证，以及监控、维护和故障排查的实践。 9. **案例研究**：通过实际应用案例，展示Hadoop在各种行业和场景中的应用，如日志分析、推荐系统和基因组学研究。 10. **未来展望**：对Hadoop的未来发展和社区动态进行了展望，包括新出现的技术如Spark、Tez和Flink等。这本书不仅适合初学者了解Hadoop的基本概念，也对有经验的开发者提供了深入技术细节和实践经验，是一本全面且权威的Hadoop学习参考资料。通过阅读本书，读者能够掌握Hadoop的精髓，从而在大数据处理领域中游刃有余。

Operations 593
Taps, Schemes, and Flows 594
Cascading in Practice 595
Flexibility 598
Hadoop and Cascading at ShareThis 599
Summary 603
TeraByte Sort on Apache Hadoop 603
Using Pig and Wukong to Explore Billion-edge Network Graphs 607
Measuring Community 609
Everybody’s Talkin’ at Me: The Twitter Reply Graph 609
Symmetric Links 612
Community Extraction 613
A. Installing Apache Hadoop  .............................................. 617
B. Cloudera’s Distribution Including Apache Hadoop  .......................... 623
C. Preparing the NCDC Weather Data  ....................................... 625
Index  ..................................................................... 629
xiv | Table of Contents

Foreword

Hadoop got its start in Nutch. A few of us were attempting to build an open source

web search engine and having trouble managing computations running on even a

handful of computers. Once Google published its GFS and MapReduce papers, the

route became clear. They’d devised systems to solve precisely the problems we were

having with Nutch. So we started, two of us, half-time, to try to re-create these systems

as a part of Nutch.

We managed to get Nutch limping along on 20 machines, but it soon became clear that

to handle the Web’s massive scale, we’d need to run it on thousands of machines and,

moreover, that the job was bigger than two half-time developers could handle.

Around that time, Yahoo! got interested, and quickly put together a team that I joined.

We split off the distributed computing part of Nutch, naming it Hadoop. With the help

of Yahoo!, Hadoop soon grew into a technology that could truly scale to the Web.

In 2006, Tom White started contributing to Hadoop. I already knew Tom through an

excellent article he’d written about Nutch, so I knew he could present complex ideas

in clear prose. I soon learned that he could also develop software that was as pleasant

to read as his prose.

From the beginning, Tom’s contributions to Hadoop showed his concern for users and

for the project. Unlike most open source contributors, Tom is not primarily interested

in tweaking the system to better meet his own needs, but rather in making it easier for

anyone to use.

Initially, Tom specialized in making Hadoop run well on Amazon’s EC2 and S3 serv-

ices. Then he moved on to tackle a wide variety of problems, including improving the

MapReduce APIs, enhancing the website, and devising an object serialization frame-

work. In all cases, Tom presented his ideas precisely. In short order, Tom earned the

role of Hadoop committer and soon thereafter became a member of the Hadoop Project

Management Committee.

Tom is now a respected senior member of the Hadoop developer community. Though

he’s an expert in many technical corners of the project, his specialty is making Hadoop

easier to use and understand.

Preface

Martin Gardner, the mathematics and science writer, once said in an interview:

Beyond calculus, I am lost. That was the secret of my column’s success. It took me so

long to understand what I was writing about that I knew how to write in a way most

readers would understand.

In many ways, this is how I feel about Hadoop. Its inner workings are complex, resting

as they do on a mixture of distributed systems theory, practical engineering, and com-

mon sense. And to the uninitiated, Hadoop can appear alien.

But it doesn’t need to be like this. Stripped to its core, the tools that Hadoop provides

for building distributed systems—for data storage, data analysis, and coordination—

are simple. If there’s a common theme, it is about raising the level of abstraction—to

create building blocks for programmers who just happen to have lots of data to store,

or lots of data to analyze, or lots of machines to coordinate, and who don’t have the

time, the skill, or the inclination to become distributed systems experts to build the

infrastructure to handle it.

With such a simple and generally applicable feature set, it seemed obvious to me when

I started using it that Hadoop deserved to be widely used. However, at the time (in

early 2006), setting up, configuring, and writing programs to use Hadoop was an art.

Things have certainly improved since then: there is more documentation, there are

more examples, and there are thriving mailing lists to go to when you have questions.

And yet the biggest hurdle for newcomers is understanding what this technology is

capable of, where it excels, and how to use it. That is why I wrote this book.

The Apache Hadoop community has come a long way. Over the course of three years,

the Hadoop project has blossomed and spun off half a dozen subprojects. In this time,

the software has made great leaps in performance, reliability, scalability, and manage-

ability. To gain even wider adoption, however, I believe we need to make Hadoop even

easier to use. This will involve writing more tools; integrating with more systems; and

1. “The science of fun,” Alex Bellos, The Guardian, May 31, 2008, http://www.guardian.co.uk/science/

2008/may/31/maths.science.

xvii

writing new, improved APIs. I’m looking forward to being a part of this, and I hope
this book will encourage and enable others to do so, too.
Administrative Notes
During discussion of a particular Java class in the text, I often omit its package name
to reduce clutter. If you need to know which package a class is in, you can easily look
it up in Hadoop’s Java API documentation for the relevant subproject, linked to from
the Apache Hadoop home page at http://hadoop.apache.org/. Or if you’re using an IDE,
it can help using its auto-complete mechanism.
Similarly, although it deviates from usual style guidelines, program listings that import
multiple classes from the same package may use the asterisk wildcard character to save
space (for example, import org.apache.hadoop.io.*).
The sample programs in this book are available for download from the website that
accompanies this book: http://www.hadoopbook.com/. You will also find instructions
there for obtaining the datasets that are used in examples throughout the book, as well
as further notes for running the programs in the book, and links to updates, additional
resources, and my blog.
What’s in This Book?
The rest of this book is organized as follows. Chapter 1 emphasizes the need for Hadoop
and  sketches  the  history  of  the  project.  Chapter  2  provides  an  introduction  to
MapReduce. Chapter 3 looks at Hadoop filesystems, and in particular HDFS, in depth.
Chapter 4  covers the fundamentals of  I/O in Hadoop:  data integrity, compression,
serialization, and file-based data structures.
The next four chapters cover MapReduce in depth. Chapter 5 goes through the practical
steps needed to develop a MapReduce application. Chapter 6 looks at how MapReduce
is implemented in Hadoop, from the point of view of a user. Chapter 7 is about the
MapReduce programming model and the various data formats that MapReduce can
work with. Chapter 8 is on advanced MapReduce topics, including sorting and joining
data.
Chapters 9  and 10  are for  Hadoop administrators  and describe how  to set  up and
maintain a Hadoop cluster running HDFS and MapReduce.
Later  chapters are  dedicated  to projects  that build on  Hadoop or  are  related to  it.
Chapters 11 and 12 present Pig and Hive, which are analytics platforms built on HDFS
and  MapReduce,  whereas  Chapters  13,  14,  and  15  cover  HBase,  ZooKeeper,  and
Sqoop, respectively.
Finally, Chapter 16 is a collection of case studies contributed by members of the Apache
Hadoop community.
xviii | Preface