Hadoop权威指南：深入解析第三版

4星 · 超过85%的资源需积分: 16 140 浏览量更新于2024-07-26 收藏 15.93MB PDF 举报

"Hadoop权威指南第三版英文版是学习Hadoop的重要书籍，受到广大读者好评，豆瓣评分高达8.5。此版本为最新更新的第三版，深入探讨了Hadoop及其相关技术在大数据处理和分布式计算领域的应用。" 《Hadoop权威指南》第三版由Tom White撰写，该书详细介绍了Hadoop生态系统，是理解和掌握这一开源大数据处理框架的关键资源。Hadoop作为分布式存储和计算的基础，是大数据处理的核心工具，尤其在云计算领域有着广泛的应用。本书涵盖了以下主要知识点： 1. **Hadoop核心组件**：包括HDFS（Hadoop Distributed File System）和MapReduce。HDFS提供了高容错性的分布式文件系统，MapReduce则是一种用于大规模数据集并行处理的编程模型。 2. **YARN（Yet Another Resource Negotiator）**：在第三版中，YARN的引入作为资源管理和调度器，提高了Hadoop集群的效率和灵活性，使得多任务并行执行成为可能。 3. **Hadoop生态扩展**：如HBase（分布式数据库）、Hive（数据仓库工具）、Pig（高级数据流语言）、Spark（快速、通用的大数据处理引擎）等，这些工具与Hadoop的集成，使得大数据分析更加强大和高效。 4. **数据输入与输出**：包括各种输入格式、分割器、记录阅读器和输出格式，以及数据清洗和转换的策略。 5. **故障恢复与数据安全**：书中详细讨论了Hadoop中的容错机制、数据备份与恢复，以及如何确保数据在分布式环境中的安全性。 6. **优化与性能调优**：如何通过配置参数、数据分布策略等手段提升Hadoop集群的性能。 7. **实时处理与流计算**：随着实时数据分析的需求增加，书中也介绍了与Hadoop相关的实时处理框架，如Storm和Flume。 8. **案例研究**：提供实际应用场景，帮助读者理解Hadoop如何在企业级项目中解决问题。 9. **开发与部署**：涵盖开发Hadoop应用程序的最佳实践，以及集群部署和管理的指导。 10. **未来趋势**：讨论了Hadoop的最新发展，以及大数据处理领域的未来走向。《Hadoop权威指南》第三版不仅是初学者入门的绝佳教材，也是经验丰富的开发人员和数据工程师的参考手册。它深入浅出地解释了Hadoop的原理和应用，无论你是要了解Hadoop的基本概念，还是寻求解决特定问题的方案，都能从中受益匪浅。通过这本书，读者可以全面地学习到Hadoop技术，并能够将其应用于实际的大数据项目中。

Operations 593

Taps, Schemes, and Flows 594

Cascading in Practice 595

Flexibility 598

Hadoop and Cascading at ShareThis 599

Summary 603

TeraByte Sort on Apache Hadoop 603

Using Pig and Wukong to Explore Billion-edge Network Graphs 607

Measuring Community 609

Everybody’s Talkin’ at Me: The Twitter Reply Graph 609

Symmetric Links 612

Community Extraction 613

A. Installing Apache Hadoop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 617

B. Cloudera’s Distribution Including Apache Hadoop . . . . . . . . . . . . . . . . . . . . . . . . . . 623

C. Preparing the NCDC Weather Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 625

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 629

xiv | Table of Contents

Foreword

Hadoop got its start in Nutch. A few of us were attempting to build an open source

web search engine and having trouble managing computations running on even a

handful of computers. Once Google published its GFS and MapReduce papers, the

route became clear. They’d devised systems to solve precisely the problems we were

having with Nutch. So we started, two of us, half-time, to try to re-create these systems

as a part of Nutch.

We managed to get Nutch limping along on 20 machines, but it soon became clear that

to handle the Web’s massive scale, we’d need to run it on thousands of machines and,

moreover, that the job was bigger than two half-time developers could handle.

Around that time, Yahoo! got interested, and quickly put together a team that I joined.

We split off the distributed computing part of Nutch, naming it Hadoop. With the help

of Yahoo!, Hadoop soon grew into a technology that could truly scale to the Web.

In 2006, Tom White started contributing to Hadoop. I already knew Tom through an

excellent article he’d written about Nutch, so I knew he could present complex ideas

in clear prose. I soon learned that he could also develop software that was as pleasant

to read as his prose.

From the beginning, Tom’s contributions to Hadoop showed his concern for users and

for the project. Unlike most open source contributors, Tom is not primarily interested

in tweaking the system to better meet his own needs, but rather in making it easier for

anyone to use.

Initially, Tom specialized in making Hadoop run well on Amazon’s EC2 and S3 serv-

ices. Then he moved on to tackle a wide variety of problems, including improving the

MapReduce APIs, enhancing the website, and devising an object serialization frame-

work. In all cases, Tom presented his ideas precisely. In short order, Tom earned the

role of Hadoop committer and soon thereafter became a member of the Hadoop Project

Management Committee.

Tom is now a respected senior member of the Hadoop developer community. Though

he’s an expert in many technical corners of the project, his specialty is making Hadoop

easier to use and understand.

Preface

Martin Gardner, the mathematics and science writer, once said in an interview:

Beyond calculus, I am lost. That was the secret of my column’s success. It took me so

long to understand what I was writing about that I knew how to write in a way most

readers would understand.

In many ways, this is how I feel about Hadoop. Its inner workings are complex, resting

as they do on a mixture of distributed systems theory, practical engineering, and com-

mon sense. And to the uninitiated, Hadoop can appear alien.

But it doesn’t need to be like this. Stripped to its core, the tools that Hadoop provides

for building distributed systems—for data storage, data analysis, and coordination—

are simple. If there’s a common theme, it is about raising the level of abstraction—to

create building blocks for programmers who just happen to have lots of data to store,

or lots of data to analyze, or lots of machines to coordinate, and who don’t have the

time, the skill, or the inclination to become distributed systems experts to build the

infrastructure to handle it.

With such a simple and generally applicable feature set, it seemed obvious to me when

I started using it that Hadoop deserved to be widely used. However, at the time (in

early 2006), setting up, configuring, and writing programs to use Hadoop was an art.

Things have certainly improved since then: there is more documentation, there are

more examples, and there are thriving mailing lists to go to when you have questions.

And yet the biggest hurdle for newcomers is understanding what this technology is

capable of, where it excels, and how to use it. That is why I wrote this book.

The Apache Hadoop community has come a long way. Over the course of three years,

the Hadoop project has blossomed and spun off half a dozen subprojects. In this time,

the software has made great leaps in performance, reliability, scalability, and manage-

ability. To gain even wider adoption, however, I believe we need to make Hadoop even

easier to use. This will involve writing more tools; integrating with more systems; and

1. “The science of fun,” Alex Bellos, The Guardian, May 31, 2008, http://www.guardian.co.uk/science/

2008/may/31/maths.science.

xvii

writing new, improved APIs. I’m looking forward to being a part of this, and I hope

this book will encourage and enable others to do so, too.

Administrative Notes

During discussion of a particular Java class in the text, I often omit its package name

to reduce clutter. If you need to know which package a class is in, you can easily look

it up in Hadoop’s Java API documentation for the relevant subproject, linked to from

the Apache Hadoop home page at http://hadoop.apache.org/. Or if you’re using an IDE,

it can help using its auto-complete mechanism.

Similarly, although it deviates from usual style guidelines, program listings that import

multiple classes from the same package may use the asterisk wildcard character to save

space (for example, import org.apache.hadoop.io.*).

The sample programs in this book are available for download from the website that

accompanies this book: http://www.hadoopbook.com/. You will also find instructions

there for obtaining the datasets that are used in examples throughout the book, as well

as further notes for running the programs in the book, and links to updates, additional

resources, and my blog.

What’s in This Book?

The rest of this book is organized as follows. Chapter 1 emphasizes the need for Hadoop

and sketches the history of the project. Chapter 2 provides an introduction to

MapReduce. Chapter 3 looks at Hadoop filesystems, and in particular HDFS, in depth.

Chapter 4 covers the fundamentals of I/O in Hadoop: data integrity, compression,

serialization, and file-based data structures.

The next four chapters cover MapReduce in depth. Chapter 5 goes through the practical

steps needed to develop a MapReduce application. Chapter 6 looks at how MapReduce

is implemented in Hadoop, from the point of view of a user. Chapter 7 is about the

MapReduce programming model and the various data formats that MapReduce can

work with. Chapter 8 is on advanced MapReduce topics, including sorting and joining

data.

Chapters 9 and 10 are for Hadoop administrators and describe how to set up and

maintain a Hadoop cluster running HDFS and MapReduce.

Later chapters are dedicated to projects that build on Hadoop or are related to it.

Chapters 11 and 12 present Pig and Hive, which are analytics platforms built on HDFS

and MapReduce, whereas Chapters 13, 14, and 15 cover HBase, ZooKeeper, and

Sqoop, respectively.

Finally, Chapter 16 is a collection of case studies contributed by members of the Apache

Hadoop community.

xviii | Preface

剩余685页未读，继续阅读

manbujingxin

粉丝: 46
资源: 5

Hadoop权威指南：深入解析第三版

《hadoop权威指南》英文版第三版

Hadoop权威指南第三版(英文版)

Hadoop权威指南（第三版）英文版

hadoop权威指南 第三版 英文版

Hadoop权威指南第三版英文版详解

Hadoop权威指南第三版英文版：入门到精通

Hadoop权威指南第三版英文版：深入探索大数据处理

Hadoop权威指南第三版英文原版

Hadoop权威指南第三版英文原版详解

hadoop权威指南第二版 英文版

最新资源

hadoop权威指南第三版英文版

hadoop权威指南第二版英文版