24.
分布式缓存......................................................................................................................................257
24.1.1.
缓存雪崩
.................................................................................................................................... 257
24.1.2.
缓存穿透
.................................................................................................................................... 257
24.1.3.
缓存预热
.................................................................................................................................... 257
24.1.4.
缓存更新
.................................................................................................................................... 257
24.1.5.
缓存降级
.................................................................................................................................... 257
25.
HADOOP ........................................................................................................................................259
25.1.1.
概念
............................................................................................................................................ 259
25.1.2. HDFS ......................................................................................................................................... 259
25.1.2.1.
25.1.2.2.
25.1.2.3.
Client..................................................................................................................................................259
NameNode........................................................................................................................................259
Secondary NameNode.................................................................................................................259
DataNode.........................................................................................................................................259
25.1.2.4.
25.1.3. MapReduce............................................................................................................................... 260
25.1.3.1.
25.1.3.2.
25.1.3.3.
25.1.3.4.
25.1.3.5.
Client.................................................................................................................................................260
JobTracker .......................................................................................................................................260
TaskTracker......................................................................................................................................261
Task ...................................................................................................................................................261
Reduce Task 执行过程 ...................................................................................................................261
25.1.4. Hadoop MapReduce
作业的生命周期
................................................................................... 262
1.作业提交与初始化...........................................................................................................................................262
2.任务调度与监控。...........................................................................................................................................262
3.任务运行环境准备...........................................................................................................................................262
4.任务执行 ..........................................................................................................................................................262
5.作业完成。 ......................................................................................................................................................262
26.
SPARK............................................................................................................................................263
26.1.1.
概念
............................................................................................................................................ 263
26.1.2.
核心架构
.................................................................................................................................... 263
Spark Core .........................................................................................................................................................263
Spark SQL..........................................................................................................................................................263
Spark Streaming................................................................................................................................................263
Mllib .....................................................................................................................................................................263
GraphX................................................................................................................................................................263
26.1.3.
核心组件
.................................................................................................................................... 264
Cluster Manager-制整个集群,监控 worker .................................................................................................264
Worker 节点-负责控制计算节点.......................................................................................................................264
Driver: 运行 Application 的 main()函数.........................................................................................................264
Executor:执行器,是为某个 Application 运行在 worker node 上的一个进程 ..........................................264
26.1.4. SPARK
编程模型
...................................................................................................................... 264
26.1.5. SPARK
计算模型
...................................................................................................................... 265
26.1.6. SPARK
运行流程
...................................................................................................................... 266
1.
2.
构建 Spark Application 的运行环境,启动 SparkContext....................................................................267
SparkContext 向资源管理器(可以是 Standalone,Mesos,Yarn)申请运行 Executor 资源,并启
动 StandaloneExecutorbackend, ..................................................................................................................267
3.
4.
5.
Executor 向 SparkContext 申请 Task.....................................................................................................267
SparkContext 将应用程序分发给 Executor............................................................................................267
SparkContext 构建成 DAG 图,将 DAG 图分解成 Stage、将 Taskset 发送给 Task Scheduler,最
后由 Task Scheduler 将 Task 发送给 Executor 运行.....................................................................................267
6.
Task 在 Executor 上运行,运行完释放所有资源...................................................................................267
26.1.7. SPARK RDD
流程
.................................................................................................................... 267
26.1.8. SPARK RDD............................................................................................................................. 267
(1)RDD 的创建方式...........................................................................................................................................267
(2)RDD 的两种操作算子(转换(Transformation)与行动(Action)) ..............................................268
27. STORM ...........................................................................................................................................269