Hadoop权威指南：架构与MapReduce详解

需积分: 1 119 浏览量更新于2024-07-28 收藏 4.98MB PDF 举报

《Hadoop权威指南》是由Tom White撰写的一本关于Apache Hadoop的详尽指南，这本书在2009年首次出版，享有Doug Cutting的序言。作为一本经典的IT技术著作，本书深入剖析了Hadoop的核心概念、架构以及其在大数据处理中的关键角色。 Hadoop的核心内容包括： 1. **架构理解**：书中详细介绍了Hadoop的整体架构，包括Hadoop Distributed File System (HDFS) 和 MapReduce框架。HDFS是一种分布式文件系统，用于存储大规模的数据集，通过冗余存储确保高可用性和容错性。MapReduce则是一种编程模型，它将复杂的计算任务分解为一系列简单的Map和Reduce操作，实现了数据并行处理。 2. **元数据管理**：Hadoop的元数据是指描述数据的数据，如文件名、属性等。书中会涉及如何管理和维护这些信息，这对于理解和优化Hadoop集群的性能至关重要。 3. **数据存储**：作者会探讨Hadoop如何设计存储系统以支持海量数据的高效读写，包括数据块的划分、副本策略以及数据压缩等优化手段。 4. **MapReduce架构详解**：读者可以在这里学习MapReduce的工作原理，包括Mapper、Reducer、Shuffle和Sort阶段，以及如何编写MapReduce作业。 5. **原理与实践**：书中的内容不仅停留在理论层面，还会结合实际案例，让读者理解如何在实际项目中应用Hadoop解决复杂的数据处理问题。 6. **最新发展**：考虑到Hadoop技术的持续发展，书中也会涵盖Hadoop生态系统的新进展，如Hadoop 2.x版本的改进，以及Hadoop生态系统的扩展组件如HBase、Hive和Pig等。 7. **版权信息**：书中强调版权保护，O'Reilly Media是该书的出版商，提供了各种购买和授权选项，同时也提到了在线版本的获取方式。《Hadoop权威指南》是Hadoop领域的权威参考书籍，对于想要深入学习Hadoop技术的开发者、数据科学家和工程师来说，它是不可或缺的参考资料。无论是初学者还是经验丰富的专业人士，都能从中获得有价值的知识和实践经验。

Given this, I was very pleased when I learned that Tom intended to write a book about

Hadoop. Who could be better qualified? Now you have the opportunity to learn about

Hadoop from a master—not only of the technology, but also of common sense and

plain talk.

—Doug Cutting

Shed in the Yard, California

xiv | Foreword

Preface

Martin Gardner, the mathematics and science writer, once said in an interview:

Beyond calculus, I am lost. That was the secret of my column’s success. It took me so

long to understand what I was writing about that I knew how to write in a way most

readers would understand.

In many ways, this is how I feel about Hadoop. Its inner workings are complex, resting

as they do on a mixture of distributed systems theory, practical engineering, and com-

mon sense. And to the uninitiated, Hadoop can appear alien.

But it doesn’t need to be like this. Stripped to its core, the tools that Hadoop provides

for building distributed systems—for data storage, data analysis, and coordination—

are simple. If there’s a common theme, it is about raising the level of abstraction—to

create building blocks for programmers who just happen to have lots of data to store,

or lots of data to analyze, or lots of machines to coordinate, and who don’t have the

time, the skill, or the inclination to become distributed systems experts to build the

infrastructure to handle it.

With such a simple and generally applicable feature set, it seemed obvious to me when

I started using it that Hadoop deserved to be widely used. However, at the time (in

early 2006), setting up, configuring, and writing programs to use Hadoop was an art.

Things have certainly improved since then: there is more documentation, there are

more examples, and there are thriving mailing lists to go to when you have questions.

And yet the biggest hurdle for newcomers is understanding what this technology is

capable of, where it excels, and how to use it. That is why I wrote this book.

The Apache Hadoop community has come a long way. Over the course of three years,

the Hadoop project has blossomed and spun off half a dozen subprojects. In this time,

the software has made great leaps in performance, reliability, scalability, and manage-

ability. To gain even wider adoption, however, I believe we need to make Hadoop even

easier to use. This will involve writing more tools; integrating with more systems; and

“The science of fun,” Alex Bellos, The Guardian, May 31, 2008, http://www.guardian.co.uk/science/

2008/may/31/maths.science.

writing new, improved APIs. I’m looking forward to being a part of this, and I hope

this book will encourage and enable others to do so, too.

Administrative Notes

During discussion of a particular Java class in the text, I often omit its package name,

to reduce clutter. If you need to know which package a class is in, you can easily look

it up in Hadoop’s Java API documentation for the relevant subproject, linked to from

the Apache Hadoop home page at http://hadoop.apache.org/. Or if you’re using an IDE,

it can help using its auto-complete mechanism.

Similarly, although it deviates from usual style guidelines, program listings that import

multiple classes from the same package may use the asterisk wildcard character to save

space (for example: import org.apache.hadoop.io.*).

The sample programs in this book are available for download from the website that

accompanies this book: http://www.hadoopbook.com/. You will also find instructions

there for obtaining the datasets that are used in examples throughout the book, as well

as further notes for running the programs in the book, and links to updates, additional

resources, and my blog.

What’s in This Book?

The rest of this book is organized as follows. Chapter 2 provides an introduction to

MapReduce. Chapter 3 looks at Hadoop filesystems, and in particular HDFS, in depth.

Chapter 4 covers the fundamentals of I/O in Hadoop: data integrity, compression,

serialization, and file-based data structures.

The next four chapters cover MapReduce in depth. Chapter 5 goes through the practical

steps needed to develop a MapReduce application. Chapter 6 looks at how MapReduce

is implemented in Hadoop, from the point of view of a user. Chapter 7 is about the

MapReduce programming model, and the various data formats that MapReduce can

work with. Chapter 8 is on advanced MapReduce topics, including sorting and joining

data.

Chapters 9 and 10 are for Hadoop administrators, and describe how to set up and

maintain a Hadoop cluster running HDFS and MapReduce.

Chapters 11, 12, and 13 present Pig, HBase, and ZooKeeper, respectively.

Finally, Chapter 14 is a collection of case studies contributed by members of the Apache

Hadoop community.

xvi | Preface

剩余525页未读，继续阅读

fqhlysliuyansen

粉丝: 0
资源: 1

Hadoop权威指南：架构与MapReduce详解

Hadoop权威指南：深入解析

Hadoop权威指南：原版英文版详解

Hadoop权威指南：高清原版详解

Hadoop权威指南原版和源码

hadoop权威指南英文原版

hadoop权威指南（原版）

hadoop权威指南《原版》

Hadoop权威指南(原版).pdf

Hadoop权威指南(英文原版)

Hadoop权威指南（英文原版）

最新资源