Hadoop实战：85个解决实际任务的案例

Hadoop

需积分: 5 109 浏览量更新于2024-07-18 收藏 15.98MB PDF 举报

身份认证购VIP最低享 7 折!

30元优惠券

资源详情

资源推荐

preface

I first encountered Hadoop in the fall of 2008 when I was working on an internet

crawl and analysis project at Verisign. My team was making discoveries similar to those

that Doug Cutting and others at Nutch had made several years earlier regarding how

to efficiently store and manage terabytes of crawled and analyzed data. At the time, we

were getting by with our home-grown distributed system, but the influx of a new data

stream and requirements to join that stream with our crawl data couldn’t be sup-

ported by our existing system in the required timelines.

After some research we came across the Hadoop project, which seemed to be a

perfect fit for our needs—it supported storing large volumes of data and provided a

mechanism to combine them. Within a few months we’d built and deployed a Map-

Reduce application encompassing a number of MapReduce jobs, woven together with

our own MapReduce workflow management system onto a small cluster of 18 nodes. It

was a revelation to observe our MapReduce jobs crunching through our data in min-

utes. Of course we couldn’t anticipate the amount of time that we’d spend debugging

and performance-tuning our MapReduce jobs, not to mention the new roles we took

on as production administrators—the biggest surprise in this role was the number of

disk failures we encountered during those first few months supporting production!

As our experience and comfort level with Hadoop grew, we continued to build

more of our functionality using Hadoop to help with our scaling challenges. We also

started to evangelize the use of Hadoop within our organization and helped kick-start

other projects that were also facing big data challenges.

The greatest challenge we faced when working with Hadoop (and specifically

MapReduce) was relearning how to solve problems with it. MapReduce is its own

ABOUT THIS BOOK xix

Roadmap

This book has 13 chapters divided into five parts.

Part 1 contains a single chapter that’s the introduction to this book. It reviews

Hadoop basics and looks at how to get Hadoop up and running on a single host. It

wraps up with a walk-through on how to write and execute a MapReduce job.

Part 2, “Data logistics,” consists of two chapters that cover the techniques and

tools required to deal with data fundamentals, getting data in and out of Hadoop,

and how to work with various data formats. Getting data into Hadoop is one of the

first roadblocks commonly encountered when working with Hadoop, and chapter 2

is dedicated to looking at a variety of tools that work with common enterprise data

sources. Chapter 3 covers how to work with ubiquitous data formats such as

XML

and JSON in MapReduce, before going on to look at data formats better suited to

working with big data.

Part 3 is called “Big data patterns,” and looks at techniques to help you work effec-

tively with large volumes of data. Chapter 4 examines how to optimize MapReduce

join and sort operations, and chapter 5 covers working with a large number of small

files, and compression. Chapter 6 looks at how to debug MapReduce performance

issues, and also covers a number of techniques to help make your jobs run faster.

Part 4 is all about “Data science,” and delves into the tools and methods that help

you make sense of your data. Chapter 7 covers how to represent data such as graphs

for use with MapReduce, and looks at several algorithms that operate on graph data.

Chapter 8 describes how R, a popular statistical and data mining platform, can be inte-

grated with Hadoop. Chapter 9 describes how Mahout can be used in conjunction

with MapReduce for massively scalable predictive analytics.

Part 5 is titled “Taming the elephant,” and examines a number of technologies

that make it easier to work with MapReduce. Chapters 10 and 11 cover Hive and Pig

respectively, both of which are MapReduce domain-specific languages (DSLs) geared

at providing high-level abstractions. Chapter 12 looks at Crunch and Cascading, which

are Java libraries that offer their own MapReduce abstractions, and chapter 13 covers

techniques to help write unit tests, and to debug MapReduce problems.

The appendixes start with appendix A, which covers instructions on installing both

Hadoop and all the other related technologies covered in the book. Appendix B cov-

ers low-level Hadoop ingress/egress mechanisms that the tools covered in chapter 2

leverage. Appendix C looks at how

HDFS supports reads and writes, and appendix D

covers a couple of MapReduce join frameworks written by the author and utilized in

chapter 4.

Code conventions and downloads

All source code in listings or in text is in a

fixed-width

font

this

to separate it

from ordinary text. Code annotations accompany many of the listings, highlighting

important concepts.

剩余536页未读，继续阅读

qq_42495277

粉丝: 2
资源: 6

Hadoop实战：85个解决实际任务的案例

Hadoop_in_Action

hadoop in practice 第二版英文版几源代码

.Hadoop.in.Practice.2nd.Edition

for host in hadoop101 hadoop102 hadoop103

Hadoop is not in the classpath/dependencies

hive Cannot find hadoop installation: $HADOOP_HOME or $HADOOP_PREFIX must be set or hadoop must be in the path

Cannot find hadoop installation: $HADOOP_HOME or $HADOOP_PREFIX must be set or hadoop must be in the path

%HADOOP_HOME%in

hadoop命令hadoop classpath

schematool -initSchema -dbType mysql -verbose Cannot find hadoop installation: $HADOOP_HOME or $HADOOP_PREFIX must be set or hadoop must be in the path

hadoop3.1.4 hadoop-core

hadoop1.0和hadoop3.0

hadoop2.0和hadoop3.0

配置Hadoop环境变量

hadoop1.0与hadoop2.0

hadoop exception in thread \main\" exitcodeexception exitcode=-1073741701:"

原生Hadoop和采购hadoop

Hadoop硬实战：Hadoop in Practice

最新资源