24. 分布式缓存 ..........................................................................257
24.1.1.
缓存雪崩
..................................................................257
24.1.2.
缓存穿透
..................................................................257
24.1.3.
缓存预热
..................................................................257
24.1.4.
缓存更新
..................................................................257
24.1.5.
缓存降级
..................................................................257
25. HADOOP .......................................................................................................................................259
25.1.1.
概念
......................................................................259
25.1.2. HDFS ..................................................................................................................................259
25.1.2.1. Client .....................................................................................................................................259
25.1.2.2. NameNode ............................................................................................................................259
25.1.2.3. Secondary NameNode ......................................................................................................................259
25.1.2.4. DataNode................................................................................................................................................259
25.1.3.
MapReduce .........................................................................................................................26 0
25.1.3.1. Client.........................................................................................................................................................260
25.1.3.2. JobTracker .............................................................................................................................................260
25.1.3.3. TaskTracker.............................................................................................................................................261
25.1.3.4. Task ..........................................................................................................................................................261
25.1.3.5. Reduce Task 执行过程 ........................................................................................................................261
25.1.4. Hadoop MapReduce
作业的生命周期
.........................................262
1.作业提交与初始化 ......................................................................262
2.任务调度与监控。 .......................................................................262
3.任务运行环境准备 ......................................................................262
4.任务执行 .............................................................................262
5.作业完成。 ...........................................................................262
26. SPARK............................................................................................................................................263
26.1.1.
概念
......................................................................263
26.1.2.
核心架构
..................................................................263
Spark
Core ...........................................................................................................................................263
Spark SQL ............................................................................................................................................263
Spark Streaming ...................................................................................................................................263
Mllib ......................................................................................................................................................263
GraphX..................................................................................................................................................263
26.1.3.
核心组件
..................................................................264
Cluster Manager-制整个集群,监控 worker ......................................................................................................264
Worker 节点-负责控制计算节点 ............................................................................................................................264
Driver: 运行 Application
的 main()函数.....................................................264
Executor:执行器,是为某个 Application 运行在 worker node 上的一个进程 .....................264
26.1.4. SPARK
编程模型
...........................................................264
26.1.5.
SPARK
计算模型
...........................................................265
26.1.6.
SPARK
运行流程
...........................................................266
1. 构建 Spark Application 的运行环境,启动 SparkContext..............................................................267
2. SparkContext 向资源管理器(可以是 Standalone,Mesos,Yarn)申请运行 Executor 资源,并启
动 StandaloneExecutorbackend, .........................................................267
3. Executor
向 SparkContext
申请 Task ...........................................................................................267
4. SparkContext 将应用程序分发给 Executor....................................................................................267
5. SparkContext 构建成 DAG 图,将 DAG
图分解成 Stage、将 Taskset 发送给 Task Scheduler,最
后由 Task Scheduler 将 Task 发送给 Executor 运行...........................................267
6. Task 在 Executor 上运行,运行完释放所有资源..........................................267
26.1.7.
SPARK RDD
流程
..........................................................267
26.1.8. SPARK RDD .......................................................................................................................267
(1)RDD 的创建方式 ......................................................................267
(2)RDD 的两种操作算子(转换(Transformation)与行动(Action) ) .......................268
27. STORM ..........................................................................................................................................269