24.
分布式缓存..............................................................................................................................................257
24.1.1.
缓存雪崩
...............................................................................................................................257
24.1.2.
缓存穿透
...............................................................................................................................257
24.1.3.
缓存预热
...............................................................................................................................257
24.1.4.
缓存更新
...............................................................................................................................257
24.1.5.
缓存降级
...............................................................................................................................257
25.
HADOOP..................................................................................................................................259
25.1.1.
概念
.......................................................................................................................................259
25.1.2.
HDFS .............................................................................................................................259
25.1.2.1. Client .....................................................................................................................................259
25.1.2.2. NameNode ............................................................................................................................259
25.1.2.3. Secondary NameNode .......................................................................................................................259
25.1.2.4. DataNode................................................................................................................................................259
25.1.3.
MapReduce ...................................................................................................................260
25.1.3.1. Client.........................................................................................................................................................260
25.1.3.2. JobTracker...............................................................................................................................................260
25.1.3.3. TaskTracker.............................................................................................................................................261
25.1.3.4. Task ...........................................................................................................................................................261
25.1.3.5. Reduce Task 执行过程 .....................................................................................................................261
25.1.4.
Hadoop MapReduce
作业的生命周期
....................................................................................
262
1. 作业提交与初始化.............................................................................................................................................262
2. 任务调度与监控。.............................................................................................................................................262
3. 任务运行环境准备.............................................................................................................................................262
4. 任务执行.............................................................................................................................................................262
5. 作业完成。.........................................................................................................................................................262
26.
SPARK .....................................................................................................................................263
26.1.1.
概念
.......................................................................................................................................263
26.1.2.
核心架构
...............................................................................................................................263
Spark Core ............................................................................................................................................263
Spark SQL.............................................................................................................................................263
Spark Streaming....................................................................................................................................263
Mllib .......................................................................................................................................................263
GraphX ..................................................................................................................................................263
26.1.3.
核心组件
...............................................................................................................................264
Cluster Manager-制整个集群,监控 worker.......................................................................................................264
Worker 节点-负责控制计算节点 .........................................................................................................................264
Driver: 运行 Application 的 main()函数 ..........................................................................................................264
Executor:执行器,是为某个 Application 运行在 worker node 上的一个进程 ...........................................264
26.1.4.
SPARK
编程模型
........................................................................................................................
264
26.1.5.
SPARK
计算模型
........................................................................................................................
265
26.1.6.
SPARK
运行流程
........................................................................................................................
266
1. 构建 Spark Application 的运行环境,启动 SparkContext ..............................................................267
2. SparkContext 向资源管理器(可以是 Standalone,Mesos,Yarn)申请运行 Executor 资源,并启
动 StandaloneExecutorbackend,.......................................................................................................................267
3. Executor 向 SparkContext 申请 Task.............................................................................................267
4. SparkContext 将应用程序分发给 Executor ....................................................................................267
5. SparkContext 构建成 DAG 图,将 DAG 图分解成 Stage、将 Taskset 发送给 Task Scheduler,最
后由 Task Scheduler 将 Task 发送给 Executor 运行 ......................................................................................267
6. Task 在 Executor 上运行,运行完释放所有资源 ....................................................................................267
26.1.7.
SPARK RDD
流程
......................................................................................................................
267
26.1.8.
SPARK RDD..................................................................................................................267
(1)
RDD 的创建方式.............................................................................................................................................267
(2)
RDD 的两种操作算子(转换(Transformation)与行动(Action)) ...............................................268
27.
STORM.....................................................................................................................................269