Spring与Hadoop集成：实战指南与MapReduce应用

3星 · 超过75%的资源需积分: 10 70 浏览量更新于2024-07-27 收藏 449KB PDF 举报

身份认证购VIP最低享 7 折!

领优惠券(最高得80元）

Spring Hadoop Reference Manual 是一本由 Costin Leau 编写的文档，针对的是将 Apache Hadoop 与 Spring 框架集成以处理大规模分布式数据处理和集群任务。该手册适用于版本 1.0.0.RC1，旨在帮助开发者理解和利用 Spring 提供的便利性，以便更轻松地在 Spring 应用程序中管理和执行 Hadoop MapReduce 任务。主要内容涵盖了以下几个关键部分： 1. 介绍：文档首先介绍了Spring Hadoop集成的目的，强调了它如何简化Hadoop配置、MapReduce作业创建和运行，以及与Hadoop分布式缓存的交互。对于那些希望在Spring环境中无缝整合Hadoop技术的开发人员来说，这是一份宝贵的指南。 2. Hadoop配置与MapReduce： - Spring for Apache Hadoop命名空间：文档详细阐述了如何使用Spring框架提供的专门命名空间来配置Hadoop，使得代码组织更加清晰。 - 配置Hadoop：讲解了如何设置Hadoop集群的各种参数和配置文件，确保应用程序能够与Hadoop环境顺利通信。 - 创建Hadoop Job：介绍了如何在Spring中创建和提交MapReduce作业，包括标准Job和Hadoop Streaming作业的创建过程。 - 运行Hadoop Job和工具：涉及实际执行作业的方法，如使用HadoopJobTasklet和HadoopToolTasklet，以及通过替换Hadoop shell命令来使用工具运行器。 - Hadoop Jar作业：展示了如何使用Hadoop Jartasklet来运行自定义的Hadoop jar包。 3. Hadoop文件系统操作： - 文件系统配置：讨论了如何配置应用程序与Hadoop文件系统（HDFS）进行交互，包括路径设置和权限管理。 - 脚本化Hadoop操作：提供了使用脚本化方法来自动化Hadoop任务执行的建议，提高了开发效率。 4. MapReduce通用选项：这部分可能包含了关于如何设置和使用MapReduce作业中的通用选项，以优化性能或定制特定的作业行为。 Spring Hadoop Reference Manual 是一个强大的资源，为那些希望在Spring应用中实现大数据处理和分布式计算的开发人员提供了解决方案和最佳实践，帮助他们更高效地利用Hadoop的潜力。通过阅读这份文档，开发者可以深入了解如何在Spring框架下更好地管理海量数据，并充分利用Hadoop集群的并行计算能力。

资源详情

资源推荐

Spring Hadoop

1.0.0.RC1

Spring Hadoop

Reference Manual 10

manually or if run-at-startup is set to true. Additionally the runner (as in fact do all runners in

SHDP) allows one or multiple pre and post actions to be specified to be executed before and after each

run. Typically other runners (such as other jobs or scripts) can be specified but any JDK Callable

can be passed in. For more information on runners, see the dedicated chapter.

Using the Hadoop Job tasklet

For Spring Batch environments, SHDP provides a dedicated tasklet to execute Hadoop jobs as a step in

a Spring Batch workflow. An example declaration is shown below:

<hdp:job-tasklet id="hadoop-tasklet" job-ref="mr-job" wait-for-job="true" />

The tasklet above references a Hadoop job definition named "mr-job". By default, wait-for-job is

true so that the tasklet will wait for the job to complete when it executes. Setting wait-for-job to false

will submit the job to the Hadoop cluster but not wait for it to complete.

2.5 Running a Hadoop Tool

It is common for Hadoop utilities and libraries to be started from the command-line (ex: hadoop jar

some.jar). SHDP offers generic support for such cases provided that the packages in question are built

on top of Hadoop standard infrastructure, namely Tool and ToolRunner classes. As oppose to the

command-line usage, Tool instances benefit from Spring's IoC features; they can be parameterized,

created and destroyed on demand and have their properties (such as the Hadoop configuration) injected.

Consider the typical jar example - invoking a class with some (two in this case) arguments (notice that

the Hadoop configuration properties are passed as well):

bin/hadoop jar -conf hadoop-site.xml -jt darwin:50020 -D property=value someJar.jar org.foo.SomeTool data/in.txt data/out.txt

Since SHDP has first-class support for configuring Hadoop, the so called generic options aren't

needed any more, even more so since typically there is only one Hadoop configuration per application.

Through tool-runner element (and its backing ToolRunner class) one typically just needs to

specify the Tool implementation and its arguments:

<hdp:tool-runner id="someTool" tool-class="org.foo.SomeTool" run-at-startup="true">

<hdp:arg value="data/in.txt"/>

<hdp:arg value="data/out.txt"/>

property=value

</hdp:tool-runner>

Additionally the runner (just like the job runner) allows one or multiple pre and post actions to be

specified to be executed before and after each run. Typically other runners (such as other jobs or scripts)

can be specified but any JDK Callable can be passed in. Do note that the runner will not run unless

triggered manually or if run-at-startup is set to true. For more information on runners, see the

dedicated chapter.

The previous example assumes the Tool dependencies (such as its class) are available in the classpath.

If that is not the case, tool-runner allows a jar to be specified:

Spring Hadoop

1.0.0.RC1

Spring Hadoop

Reference Manual 11

<hdp:tool-runner ... jar="myTool.jar">

...

</hdp:tool-runner>

The jar is used to instantiate and start the tool - in fact all its dependencies are loaded from the jar meaning

they no longer need to be part of the classpath. This mechanism provides proper isolation between tools

as each of them might depend on certain libraries with different versions; rather then adding them all

into the same app (which might be impossible due to versioning conflicts ), one can simply point to the

different jars and be on her way. Note that when using a jar, if the main class (as specified by the Main-

Class entry) is the target Tool, one can skip specifying the tool as it will picked up automatically.

Like the rest of the SHDP elements, tool-runner allows the passed Hadoop configuration (by default

hadoopConfiguration but specified in the example for clarity) to be customized accordingly; the

snippet only highlights the property initialization for simplicity but more options are available. Since

usually the Tool implementation has a default argument, one can use the tool-class attribute

however it is possible to refer to another Tool instance or declare a nested one:

<hdp:tool-runner id="someTool" run-at-startup="true">

<hdp:tool>

</hdp:tool>

</hdp:tool-runner>

This is quite convenient if the Tool class provides setters or richer constructors. Note that by default

the tool-runner does not executes the Tooluntil its definition is actually called - this behavior can

be changed through the run-at-startup attribute above.

Replacing Hadoop shell invocations with tool-runner

tool-runner is a nice way for migrating series or shell invocations or scripts into fully wired,

managed Java objects. Consider the following shell script:

hadoop jar job1.jar -files fullpath:props.properties -Dconfig=config.properties ...

hadoop jar job2.jar arg1 arg2...

...

hadoop jar job10.jar ...

Each job is fully contained in the specified jar, including all the dependencies (which might conflict

with the ones from other jobs). Additionally each invocation might provide some generic options or

arguments but for the most part all will share the same configuration (as they will execute against the

same cluster).

The script can be fully ported to SHDP, through the tool-runner element:

<hdp:tool-runner id="job1" tool-class="job1.Tool" jar="job1.jar" files="fullpath:props.properties" properties-location="config.properties"/>

<hdp:tool-runner id="job2" jar="job2.jar">

<hdp:arg value="arg1"/>

<hdp:arg value="arg2"/>

</hdp:tool-runner>

<hdp:tool-runner id="job3" jar="job3.jar"/>

...

Spring Hadoop

1.0.0.RC1

Spring Hadoop

Reference Manual 12

All the features have been explained in the previous sections but let us review what happens here. As

mentioned before, each tool gets autowired with the hadoopConfiguration; job1 goes beyond

this and uses its own properties instead. For the first jar, the Tool class is specified, however the

rest assume the jar main classes implement the Tool interface; the namespace will discover them

automatically and use it accordingly. When needed (such as with job1), additional files or libs are

provisioned in the cluster. Same thing with the job arguments.

However more things that go beyond scripting, can be applied to this configuration - each job can

have multiple properties loaded or declared inlined - not just from the local file system, but also

from the classpath or any url for that matter. In fact, the whole configuration can be externalized and

parameterized (through Spring's property placeholder and/or Environment abstraction). Moreover, each

job can be ran by itself (through the JobRunner) or as part of a workflow - either through Spring's

depends-on or the much more powerful Spring Batch and tool-tasklet.

Using the Hadoop Tool tasklet

For Spring Batch environments, SHDP provides a dedicated tasklet to execute Hadoop tasks as a step

in a Spring Batch workflow. The tasklet element supports the same configuration options as tool-runner

except for run-at-startup (which does not apply for a workflow):

<hdp:tool-tasklet id="tool-tasklet" tool-ref="some-tool" />

2.6 Running a Hadoop Jar

SHDP also provides support for executing vanilla Hadoop jars. Thus the famous WordCount example:

bin/hadoop jar hadoop-examples.jar wordcount /wordcount/input /wordcount/output

becomes

<hdp:jar-runner id="wordcount" jar="hadoop-examples.jar" run-at-startup="true">

<hdp:arg value="wordcount"/>

<hdp:arg value="/wordcount/input"/>

<hdp:arg value="/wordcount/output"/>

</hdp:jar-runner>

Note

Just like the hadoop jar command, by default the jar support reads the jar's Main-Class

if none is specified. This can be customized through the main-class attribute.

Additionally the runner (just like the job runner) allows one or multiple pre and post actions to be

specified to be executed before and after each run. Typically other runners (such as other jobs or scripts)

can be specified but any JDK Callable can be passed in. Do note that the runner will not run unless

triggered manually or if run-at-startup is set to true. For more information on runners, see the

dedicated chapter.

The jar support provides a nice and easy migration path from jar invocations from the command-

line to SHDP (note that Hadoop generic options are also supported). Especially since SHDP enables

Hadoop Configuration objects created during the jar execution, to automatically inherit the context

Spring Hadoop

1.0.0.RC1

Spring Hadoop

Reference Manual 13

Hadoop configuration. In fact, just like other SHDP elements, the jar element allows configurations

properties to be declared locally, just for the jar run. So for example, if one would use the following

declaration:

<hdp:jar-runner id="wordcount" jar="hadoop-examples.jar" run-at-startup="true">

<hdp:arg value="wordcount"/>

...

speed=fast

</hdp:jar-runner>

inside the jar code, one could do the following:

assert "fast".equals(new Configuration().get("speed"));

This enabled basic Hadoop jars to use, without changes, the enclosing application Hadoop configuration.

And while we think it is a useful feature (that is why we added it in the first place), we strongly

recommend using the tool support instead or migrate to it; there are several reasons for this mainly

because there are no contracts to use, leading to very poor embeddability caused by:

• No standard Configuration injection

While SHDP does a best effort to pass the Hadoop configuration to the jar, there is no guarantee

the jar itself does not use a special initialization mechanism, ignoring the passed properties. After

all, a vanilla Configuration is not very useful so applications tend to provide custom code to

address this.

• System.exit() calls

Most jar examples out there (including WordCount) assume they are started from the command line

and among other things, call System.exit, to shut down the JVM, whether the code is succesful

or not. SHDP prevents this from happening (otherwise the entire application context would shutdown

abruptly) but it is a clear sign of poor code collaboration.

SHDP tries to use sensible defaults to provide the best integration experience possible but at the end of

the day, without any contract in place, there are no guarantees. Hence why using the Tool interface,

is a much better alternative.

Using the Hadoop Jar tasklet

Like for the rest of its tasks, for Spring Batch environments, SHDP provides a dedicated tasklet

to execute Hadoop jars as a step in a Spring Batch workflow. The tasklet element supports the

same configuration options as jar-runner except for run-at-startup (which does not apply for a

workflow):

<hdp:jar-tasklet id="jar-tasklet" jar="some-jar.jar" />

2.7 Configuring the Hadoop DistributedCache

DistributedCache is a Hadoop facility for distributing application-specific, large, read-only files ( text,

archives, jars and so on) efficiently. Applications specify the files to be cached via urls (hdfs://)

剩余87页未读，继续阅读

online??

粉丝: 0
资源: 2

Spring与Hadoop集成：实战指南与MapReduce应用

s_--www.jrmianban.com-data-2007101915131200045804.pdf.zip

Apress - Pro Hadoop.pdf

Hadoop 2.x与Hadoop 3.x版本比较

Hadoop 2.x 和 Hadoop 3.x 的重大变革

Spring与Hadoop的集成与实践

HDFS与Elasticsearch数据索引与搜索：Elasticsearch Hadoop与ES-Hadoop Connector

大数据处理：Spring Boot与Hadoop集成

spark-3.3.0-bin-hadoop3.tg和spark-3.3.0-bin-without-hadoop.tgz

spark-3.2.0-bin-without-hadoop.tgz

spark-3.2.0.tgz和spark-3.2.0-bin-hadoop2.tar.gz的区别

flink-shaded-hadoop-3-uber.jar

在安装配置hadoop时，需要进行配置的配置文件有 A yarn-env.sh B mapred-site.xml C core-site.xml D hadoop-env.sh E mapred-env.sh F hdfs-site.xml G yarn-site.xml

介绍hadoop的core-site.xml、hadoop-env.sh、hdfs-site.xml、mapred-site.xml、 yarn-env.sh、yarn-site.xml文件1200字

介绍hadoop的core-site.xml、hadoop-env.sh、hdfs-site.xml、mapred-site.xml、workers、yarn-site.xml文件1200字

cd /root cp hadoop-2.8.3.tar.gz /home/modules/ cd /home/modules/ tar -zxvf hadoop-2.8.3.tar.gz如何分割

spark-2.2.0-bin-without-hadoop.tgz 下载

$hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/common/hadoop-lzo-0.4.20.jar com.hadoop.compression.lzo.DistributedLzoIndexer -Dmapreduce.job.queuename=hive /warehouse/$APP/ods/ods_log/dt=$do_date什么意思

分析Hadoop配置文件hadoop-env.sh、core-site.xml、hdfs-site.xml、hdfs-site.xml、yarn-site.xml中每个参数的含义及作用；

cp /opt/hadoop/hadoop-0.20.2.tar.gz /usr/local/ tar –zxvf hadoop-0.20.2.tar.gz

cd /home/modules/hadoop-2.8.3/etc/hadoop/ mv mapred-site.xml.template mapred-site.xml vim /home/modules/hadoop-2.8.3/etc/hadoop/mapred-site.xml

最新资源