掌握Hadoop：分布式编程实战

需积分: 10 26 浏览量更新于2024-07-24 收藏 5.09MB PDF 举报

"Hadoop in Action" 《Hadoop实战》是一本非常适合初学者了解和学习Hadoop的书籍。这本书由Manning Publications出版，通过Manning Early Access Program（MEAP）提供，旨在详细介绍分布式编程框架Hadoop及其应用。在Part 1 "Hadoop - 分布式编程框架"中，作者首先介绍了Hadoop的基础知识。这一部分涵盖了构成Hadoop集群的硬件组件，以及如何安装和配置以创建一个可工作的系统。此外，书中深入浅出地讲解了MapReduce框架，帮助读者理解其核心概念，并指导如何编写并运行第一个MapReduce程序。第1章 "引入Hadoop" 介绍了如何构建可扩展的、分布式的数据密集型应用程序的基本原理。这部分内容可能包括Hadoop的设计哲学、它如何处理大数据以及其在处理分布式计算问题中的优势。第2章 "启动Hadoop" 详细讲述了设置Hadoop环境的步骤，包括硬件需求、软件依赖、集群配置和初始化过程，确保读者能够成功启动并运行Hadoop集群。第3章 "Hadoop的组件" 探讨了Hadoop生态系统中的关键组件，如HDFS（Hadoop分布式文件系统）和YARN（Yet Another Resource Negotiator），这些组件是Hadoop实现高效数据处理的关键。在Part 2 "Hadoop实战" 中，作者进一步深入到Hadoop的实际应用中。第4章 "编写基本的MapReduce程序" 教授如何编写简单的MapReduce作业，包括映射（map）和化简（reduce）函数，以及数据处理的基本流程。第5章 "高级MapReduce" 深入讨论更复杂的MapReduce技术，如组合器（combiner）、分区器（partitioner）和自定义排序（custom sorting），这些工具能提升MapReduce程序的效率和性能。第6章 "编程实践" 提供了一些最佳实践和技巧，帮助开发者写出更高效、更可靠的Hadoop程序。第7章 "烹饪书" 可能包含一系列实际示例和案例研究，展示如何解决特定的数据处理问题，使读者能够将所学应用于实际场景。第8章 "管理Hadoop" 介绍了监控、调试、优化和维护Hadoop集群的方法，确保系统的稳定运行和性能最大化。 Part 3 "Hadoop狂野之旅" 探索了Hadoop在更广阔领域的应用，例如：第9章 "在云端运行Hadoop" 讨论了如何在云环境中部署和管理Hadoop，如Amazon Web Services（AWS）或其他云平台。第10章 "使用Pig编程" 展示了Pig Latin语言，这是一种高级的数据流语言，用于简化Hadoop上的大数据分析。第11章 "Hive和Hadoop的牧群" 介绍了Hive，一个基于Hadoop的数据仓库工具，允许用户使用SQL（结构化查询语言）进行数据分析。第12章 "案例研究" 提供了实际企业或组织使用Hadoop解决具体问题的实例，帮助读者理解Hadoop在实际业务中的应用价值。附录部分包括HDFS文件命令，提供了Hadoop分布式文件系统的操作指南，让读者更好地管理和操作数据。《Hadoop实战》是一本全面覆盖Hadoop从基础到高级应用的教程，适合希望在大数据领域深入学习和实践的读者。通过这本书，读者将能够理解Hadoop的工作原理，编写MapReduce程序，掌握Hadoop的管理和运维，以及如何利用Hadoop生态系统中的其他工具如Pig和Hive进行数据分析。

14 C

HAPTER

1 Introducing Hadoop

for each value in values {

sum = sum + value;

}

emit ((String)token, (Integer) sum);

}

We’ve said before that the output of both map and reduce function are lists. As you

can see from the pseudo-code, in practice we use a special function in the framework

called emit() to generate the elements in the list one at a time. This emit() function

further relieves the programmer from managing a large list.

The code looks similar to what we have in section 1.5.1, except this time it will

actually work at scale. Hadoop makes building scalable distributed programs easy,

doesn’t it? Now let’s turn this pseudo-code into a Hadoop program.

1.6 Counting words with Hadoop—running your ﬁ rst program

Now that you know what the Hadoop and MapReduce framework is about, let’s get it

running. In this chapter, we’ll run Hadoop only on a single machine, which can be

your desktop or laptop computer. The next chapter will show you how to run Hadoop

over a cluster of machines, which is what you’d want for practical deployment. Run-

ning Hadoop on a single machine is mainly useful for development work.

Linux is the ofﬁ cial development and production platform for Hadoop, although

Windows is a supported development platform as well. For a Windows box, you’ll need

to install cygwin (http://www-cygwin.com/) to enable shell and Unix scripts.

NOTE

Many people have reported success in running Hadoop in development

mode on other variants of Unix, such as Solaris and Mac OS X . In fact,

MacBook Pro seems to be the laptop of choice among Hadoop developers, as

they’re ubiquitous in Hadoop conferences and user group meetings.

Running Hadoop requires Java (version 1.6 or higher). Mac users should get it from

Apple. You can download the latest JDK for other operating systems from Sun at

http://java.sun.com/javase/downloads/index.jsp. Install it and remember the root of

the Java installation, which we’ll need later.

To install Hadoop, ﬁ rst get the latest stable release at http://hadoop.apache.org/

core/releases.html. After you unpack the distribution, edit the script conf/hadoop-

env.sh to set JAVA_HOME to the root of the Java installation you have remembered

from earlier. For example, in Mac OS X, you’ll replace this line

# export JAVA_HOME=/usr/lib/j2sdk1.5-sun

with this line

export JAVA_HOME=/Library/Java/Home

You’ll be using the Hadoop script quite often. Let’s run it without any arguments to

see its usage documentation:

Counting words with Hadoop—running your ﬁ rst program 15

bin/hadoop

We get

Usage: hadoop [--conﬁ g confdir] COMMAND

where COMMAND is one of:

namenode -format format the DFS ﬁ lesystem

secondarynamenode run the DFS secondary namenode

namenode run the DFS namenode

datanode run a DFS datanode

dfsadmin run a DFS admin client

fsck run a DFS ﬁ lesystem checking utility

fs run a generic ﬁ lesystem user client

balancer run a cluster balancing utility

jobtracker run the MapReduce job Tracker node

pipes run a Pipes job

tasktracker run a MapReduce task Tracker node

job manipulate MapReduce jobs

version print the version

jar <jar> run a jar ﬁ le

distcp <srcurl> <desturl> copy ﬁ le or directories recursively

archive -archiveName NAME <src>* <dest> create a hadoop archive

daemonlog get/set the log level for each daemon

CLASSNAME run the class named CLASSNAME

Most commands print help when invoked w/o parameters.

We’ll cover the various Hadoop commands in the course of this book. For our current

purpose, we only need to know that the command to run a ( Java) Hadoop program is

bin/hadoop jar <jar>

. As the command implies, Hadoop programs written in Java

are packaged in jar ﬁ les for execution.

Fortunately for us, we don’t need to write a Hadoop program ﬁ rst; the default

installation already has several sample programs we can use. The following command

shows what is available in the examples jar ﬁ le:

bin/hadoop jar hadoop-*-examples.jar

You’ll see about a dozen example programs prepackaged with Hadoop, and one

of them is a word counting program called...

wordcount

! The important (inner)

classes of that program are shown in listing 1.2. We’ll see how this Java program

implements the word counting map and reduce functions we had in pseudo-code

in listing 1.1. We’ll modify this program to understand how to vary its behavior. For

now we’ll assume it works as expected and only follow the mechanics of executing a

Hadoop program.

Without specifying any arguments, executing

wordcount

will show its usage

information:

bin/hadoop jar hadoop-*-examples.jar wordcount

which shows the arguments list:

wordcount [-m <maps>] [-r <reduces>] <input> <output>

16 C

HAPTER

1 Introducing Hadoop

The only parameters are an input directory (

<input>

) of text documents you want to

analyze and an output directory (

) where the program will dump its output.

To execute

wordcount

, we need to ﬁ rst create an input directory:

mkdir input

and put some documents in it. You can add any text document to the directory. For

illustration, let’s put the text version of the 2002 State of the Union address, obtained

from http://www.gpoaccess.gov/sou/. We now analyze its word counts and see the

results:

bin/hadoop jar hadoop-*-examples.jar wordcount input output

more output/*

You’ll see a word count of every word used in the document, listed in alphabetical or-

der. This is not bad considering you have not written a single line of code yet! But, also

note a number of shortcomings in the included

wordcount

program. Tokenization

is based purely on whitespace characters and not punctuation marks, making States,

States., and States: separate words. The same is true for capitalization, where States and

states appear as separate words. Furthermore, we would like to leave out words that

show up in the document only once or twice.

Fortunately, the source code for

wordcount

is available and included in the

installation at src/examples/org/apache/hadoop/examples/WordCount.java. We

can modify it as per our requirements. Let’s ﬁ rst set up a directory structure for our

playground and make a copy of the program.

mkdir playground

mkdir playground/src

mkdir playground/classes

cp src/examples/org/apache/hadoop/examples/WordCount.java

➥

playground/src/WordCount.java

Before we make changes to the program, let’s go through compiling and executing

this new copy in the Hadoop framework.

javac -classpath hadoop-*-core.jar -d playground/classes

➥

playground/src/WordCount.java

jar -cvf playground/wordcount.jar -C playground/classes/ .

You’ll have to remove the output directory each time you run this Hadoop command,

because it is created automatically.

bin/hadoop jar playground/wordcount.jar

➥

org.apache.hadoop.examples.WordCount input output

Look at the ﬁ les in your output directory again. As we haven’t changed any program

code, the result should be the same as before. We’ve only compiled our own copy

rather than running the precompiled version.

Now we are ready to modify

WordCount

to add some extra features. Listing 1.2 is

a partial view of the WordCount.java program. Comments and supporting code are

stripped out.

Listing 1.2 WordCount.java

public class WordCount extends Conﬁ gured implements Tool {

public static class MapClass extends MapReduceBase

implements Mapper<LongWritable, Text, Text, IntWritable> {

private ﬁ nal static IntWritable one = new IntWritable(1);

private Text word = new Text();

public void map(LongWritable key, Text value,

OutputCollector<Text, IntWritable> output,

Reporter reporter) throws IOException {

String line = value.toString();

StringTokenizer itr = new StringTokenizer(line);

while (itr.hasMoreTokens()) {

word.set(itr.nextToken());

output.collect(word, one);

}

public static class Reduce extends MapReduceBase

implements Reducer<Text, IntWritable, Text, IntWritable> {

public void reduce(Text key, Iterator<IntWritable> values,

OutputCollector<Text, IntWritable> output,

Reporter reporter) throws IOException {

int sum = 0;

while (values.hasNext()) {

sum += values.next().get();

}

output.collect(key, new IntWritable(sum));

}

...

}

The main functional distinction between

WordCount.java

and our MapReduce pseudo-

code is that in

WordCount.java

map()

processes one line of text at a time whereas our

pseudo-code processes a document at a time. This distinction may not even be apparent

from looking at

WordCount.java

as it’s Hadoop’s default conﬁ guration.

The code in listing 1.2 is virtually identical to our pseudo-code in listing 1.1 though

the Java syntax makes it more verbose. The map and reduce functions are inside inner

classes of

WordCount

. You may notice we use special classes such as

LongWritable

IntWritable

, and

Text

instead of the more familiar

Long

Integer

, and

String

classes of Java. Consider these implementation details for now. The new classes have

additional serialization capabilities needed by Hadoop’s internal.

The changes we want to make to the program are easy to spot. We see

that

WordCount

uses Java’s

StringTokenizer

in its default setting, which tokenizes based

only on whitespaces. To ignore standard punctuation marks, we add them to the

StringTokenizer

’s list of delimiter characters:

StringTokenizer itr = new StringTokenizer(line, “ \t\n\r\f,.:;?![]`”);

Counting words with Hadoop—running your ﬁ rst program 17

剩余298页未读，继续阅读

abuali_1

粉丝: 0
资源: 7

掌握Hadoop：分布式编程实战

Hadoop In Action2

Hadoop权威指南中文版（第二版）+Hadoop in Action

hadoop in action

Hadoop In Action

Hadoop in action

036GraphTheory(图论) matlab代码.rar

026SVM用于分类时的参数优化，粒子群优化算法，用于优化核函数的c,g两个参数(SVM PSO)Matlab代码.rar

药店管理-JAVA-基于springBoot的药店管理系统的设计与实现（毕业论文+开题）

【网络】基于matlab高动态网络拓扑中OSPF网络计算【含Matlab源码 10964期】.zip

今天吴老师上课的时候说我.txt

最新资源