Hadoop实践指南：入门与进阶

需积分: 0 85 浏览量更新于2024-07-26 收藏 21.07MB PDF 举报

"Hadoop in Practice 是一本由 Alex Holmes 撰写的入门级 Hadoop 书籍，由 Manning Publications 出版。这本书旨在帮助读者理解和掌握 Hadoop 技术的实际应用。" 在《Hadoop in Practice》中，作者 Alex Holmes 详细介绍了 Hadoop 生态系统的关键组件及其在实际工作中的应用。Hadoop 是一个开源框架，主要用于处理和存储大量数据，它是大数据处理领域的基石。这本书是针对那些想要深入理解 Hadoop 并希望将其应用于实际业务场景的读者而设计的。书中涵盖了以下关键知识点： 1. **Hadoop 基础**：解释了 Hadoop 分布式文件系统 (HDFS) 和 MapReduce 模型，这两个是 Hadoop 的核心组件。HDFS 提供高容错性的分布式存储，而 MapReduce 则是用于并行处理大规模数据集的编程模型。 2. **Hadoop 安装与配置**：详细介绍了如何在本地和集群环境中设置 Hadoop，包括硬件需求、软件依赖和配置文件的调整。 3. **数据处理**：讲解了如何使用 Hadoop 进行数据清洗、转换和分析，包括使用 Pig、Hive 和 Sqoop 等工具进行数据操作。这些工具提供了更高级别的抽象，使得非 Java 开发者也能轻松处理 Hadoop 任务。 4. **实时流处理**：介绍了 Apache Flume 和 Kafka 等工具，用于处理实时数据流和构建数据管道。 5. **数据存储优化**：讨论了 HBase 和 Cassandra 等 NoSQL 数据库，它们提供对 Hadoop 存储的数据进行快速随机访问的能力。 6. **MapReduce 算法**：通过示例展示了如何编写 MapReduce 程序，包括解决常见问题的策略，如数据倾斜和性能优化。 7. **Hadoop 高级主题**：涵盖了 YARN（Yet Another Resource Negotiator），它是 Hadoop 2.x 中的资源管理器，以及 Hadoop 容错机制和安全性。 8. **生态系统工具**：简述了如 Mahout（机器学习库）和 Spark（快速通用的大数据处理引擎）等扩展 Hadoop 功能的工具。通过《Hadoop in Practice》，读者不仅可以学习到 Hadoop 的基本概念，还能了解到如何在实际项目中有效地运用这些技术。此外，书中还包含了大量实例和实践建议，帮助读者将理论知识转化为实际行动，从而在大数据处理领域提升自己的技能。

preface

I first encountered Hadoop in the fall of 2008 when I was working on an internet

crawl and analysis project at Verisign. My team was making discoveries similar to those

that Doug Cutting and others at Nutch had made several years earlier regarding how

to efficiently store and manage terabytes of crawled and analyzed data. At the time, we

were getting by with our home-grown distributed system, but the influx of a new data

stream and requirements to join that stream with our crawl data couldn’t be sup-

ported by our existing system in the required timelines.

After some research we came across the Hadoop project, which seemed to be a

perfect fit for our needs—it supported storing large volumes of data and provided a

mechanism to combine them. Within a few months we’d built and deployed a Map-

Reduce application encompassing a number of MapReduce jobs, woven together with

our own MapReduce workflow management system onto a small cluster of 18 nodes. It

was a revelation to observe our MapReduce jobs crunching through our data in min-

utes. Of course we couldn’t anticipate the amount of time that we’d spend debugging

and performance-tuning our MapReduce jobs, not to mention the new roles we took

on as production administrators—the biggest surprise in this role was the number of

disk failures we encountered during those first few months supporting production!

As our experience and comfort level with Hadoop grew, we continued to build

more of our functionality using Hadoop to help with our scaling challenges. We also

started to evangelize the use of Hadoop within our organization and helped kick-start

other projects that were also facing big data challenges.

The greatest challenge we faced when working with Hadoop (and specifically

MapReduce) was relearning how to solve problems with it. MapReduce is its own

Download from Wow! eBook <www.wowebook.com>

ABOUT THIS BOOK xix

Roadmap

This book has 13 chapters divided into five parts.

Part 1 contains a single chapter that’s the introduction to this book. It reviews

Hadoop basics and looks at how to get Hadoop up and running on a single host. It

wraps up with a walk-through on how to write and execute a MapReduce job.

Part 2, “Data logistics,” consists of two chapters that cover the techniques and

tools required to deal with data fundamentals, getting data in and out of Hadoop,

and how to work with various data formats. Getting data into Hadoop is one of the

first roadblocks commonly encountered when working with Hadoop, and chapter 2

is dedicated to looking at a variety of tools that work with common enterprise data

sources. Chapter 3 covers how to work with ubiquitous data formats such as

XML

and JSON in MapReduce, before going on to look at data formats better suited to

working with big data.

Part 3 is called “Big data patterns,” and looks at techniques to help you work effec-

tively with large volumes of data. Chapter 4 examines how to optimize MapReduce

join and sort operations, and chapter 5 covers working with a large number of small

files, and compression. Chapter 6 looks at how to debug MapReduce performance

issues, and also covers a number of techniques to help make your jobs run faster.

Part 4 is all about “Data science,” and delves into the tools and methods that help

you make sense of your data. Chapter 7 covers how to represent data such as graphs

for use with MapReduce, and looks at several algorithms that operate on graph data.

Chapter 8 describes how R, a popular statistical and data mining platform, can be inte-

grated with Hadoop. Chapter 9 describes how Mahout can be used in conjunction

with MapReduce for massively scalable predictive analytics.

Part 5 is titled “Taming the elephant,” and examines a number of technologies

that make it easier to work with MapReduce. Chapters 10 and 11 cover Hive and Pig

respectively, both of which are MapReduce domain-specific languages (DSLs) geared

at providing high-level abstractions. Chapter 12 looks at Crunch and Cascading, which

are Java libraries that offer their own MapReduce abstractions, and chapter 13 covers

techniques to help write unit tests, and to debug MapReduce problems.

The appendixes start with appendix A, which covers instructions on installing both

Hadoop and all the other related technologies covered in the book. Appendix B cov-

ers low-level Hadoop ingress/egress mechanisms that the tools covered in chapter 2

leverage. Appendix C looks at how

HDFS supports reads and writes, and appendix D

covers a couple of MapReduce join frameworks written by the author and utilized in

chapter 4.

Code conventions and downloads

All source code in listings or in text is in a

fixed-width

font

this

to separate it

from ordinary text. Code annotations accompany many of the listings, highlighting

important concepts.

Download from Wow! eBook <www.wowebook.com>

剩余536页未读，继续阅读

zhaoyue007101

粉丝: 119
资源: 9

Hadoop实践指南：入门与进阶

《Hadoop in Practice 第2版》：实战指南与核心技术

实践指南：Hadoop技术详解

Hadoop实践指南：Alex Holmes解读

Hadoop in Practice

Hadoop硬实战：Hadoop in Practice

Hadoop in Practice 2nd Edition

Hadoop in Practice(2012)

Hadoop In Practice （英文）

hadoop in practice 第二版英文版几源代码

hadoop-book, 书"Hadoop in Practice" 伴随的源代码，由曼宁出版.zip

最新资源