24.11 HADOOP2.7.3 安装..................................................................................................................................509
24.11.1
配置
root
到本机的免登录。
..........................................................................................................512
24.11.2
格式化和启动
..................................................................................................................................512
24.12 MAPREDUCE 原理...................................................................................................................................515
24.13 MAPREDUCE 的 INPUTFORMAT................................................................................................................518
24.13.1 Mapreduce
的
Combiner..................................................................................................................519
24.13.2 Mapreduce
的
partitioner................................................................................................................520
24.14 MAPREDUCE1 基础模型..........................................................................................................................521
24.14.1
总体流程如下
..................................................................................................................................521
24.15 JOBTRACKER 和 TASKTRACKER 的作用..................................................................................................522
24.15.1 Jobtracker........................................................................................................................................522
24.15.2 Tasktracker.......................................................................................................................................522
24.16 MAP 和 REDUCE 的作用..........................................................................................................................522
24.16.1 Map..................................................................................................................................................522
24.16.2 Reduce..............................................................................................................................................522
24.17 MAPREDUCE 2.0 架构.............................................................................................................................523
24.17.1 Client................................................................................................................................................523
24.17.2 Application Master
(
MR App Mstr
)
.............................................................................................523
24.17.3 ResourceManager............................................................................................................................524
24.17.4 NodeManager..................................................................................................................................524
24.17.5 Container.........................................................................................................................................524
24.17.6 YARN
运行流程
...............................................................................................................................524
24.18 MAPREDUCE2.0 的容错性......................................................................................................................526
24.19 MAPREDUCE 的数据本地化...................................................................................................................526
24.20 MAPREDUCE 的推测执行机制...............................................................................................................527
24.21 MAPREDUCE 的任务并行执行...............................................................................................................527
24.22 MAPREDUCE 编程模型...........................................................................................................................529
24.22.1 Map
编程模型
.................................................................................................................................529
24.22.2 Reduce
编程模型
.............................................................................................................................530
24.22.3 Shuffle
的整体图
..............................................................................................................................530
24.23 WORDCOUNT 程序详细计算流程...........................................................................................................531
24.24 WORDCOUNT 编写...................................................................................................................................531
24.24.1
编写
map..........................................................................................................................................531
24.24.2
编写
reduce......................................................................................................................................532
24.24.3
编写
main
方法
................................................................................................................................533
24.24.4
测试
..................................................................................................................................................533
24.25 HADOOP 的序列号 AVRO........................................................................................................................534
24.25.1 java
原生序列原理和存在的问题
..................................................................................................534
24.25.2 avro..................................................................................................................................................534
24.25.3 Hadoop Writable..............................................................................................................................535
24.26 ECLIPSE 里配置 HADOOP-ECLIPSE-PLUGIN 插件.....................................................................................535
24.26.1
复制
jar
包
.......................................................................................................................................535
24.26.2
配置
Map/ReduceLocations.............................................................................................................535
24.27 MAPREDUCE1 的作业提交详细流程.....................................................................................................541
11 / 563