6.SparkSQL（上）--SparkSQL简介.pdf

需积分: 14 0 浏览量更新于2023-05-26 评论收藏 2.61MB PDF 举报

身份认证购VIP最低享 7 折!

领优惠券(最高得80元）

资源详情

资源评论

资源推荐

第 1 页共 34 页出自石山园，博客地址：http://www.cnblogs.com/shishanyuan

SparkSQL 简介

第 2 页共 34 页出自石山园，博客地址：http://www.cnblogs.com/shishanyuan

1 SPARKSQL的发展历程 ............................................................................................................................................ 3

1.1 H

IVE AND SHARK ................................................................................................................................................. 3

1.2 S

HARK和SPARKSQL ............................................................................................................................................ 3

1.3 S

PARKSQL的性能 ............................................................................................................................................... 4

SPARKSQL运行架构 ................................................................................................................................................ 8

2.1 T

REE和RULE ....................................................................................................................................................... 9

2.1.1 Tree ................................................................................................................................................................... 9

2.1.2 Rule ................................................................................................................................................................. 10

2.2

SQLCONTEXT和HIVECONTEXT的运行过程 ...................................................................................................... 10

2.2.1 sqlContext

的运行过程

................................................................................................................................... 10

2.2.2 hiveContext

的运行过程

................................................................................................................................. 11

2.3

CATALYST优化器 ............................................................................................................................................... 12

3 SPARKSQL CLI ...................................................................................................................................................... 13

3.1 运行环境说明 ................................................................................................................................................... 13

3.1.1

硬软件环境

.................................................................................................................................................... 13

3.1.2

机器网络环境

................................................................................................................................................ 14

3.2 配置并启动 ....................................................................................................................................................... 14

3.2.1

创建并配置

hive-site.xml ................................................................................................................................ 14

3.2.2

启动

Hive ......................................................................................................................................................... 15

3.2.3

启动

Spark

集群和

Spark SQL CLI .................................................................................................................. 15

3.2.4

命令参数

........................................................................................................................................................ 16

3.3 实战S

PARK SQL CLI ......................................................................................................................................... 17

3.3.1

获取订单每年的销售单数、销售总额

........................................................................................................ 17

3.3.2

计算所有订单每年的总金额

........................................................................................................................ 18

3.3.3

计算所有订单每年最大金额订单的销售额

................................................................................................ 20

4 SPARK THRIFT SERVER ..................................................................................................................................... 21

4.1 配置并启动 ....................................................................................................................................................... 21

4.1.1

创建并配置

hive-site.xml ................................................................................................................................ 21

4.1.2

启动

Hive ......................................................................................................................................................... 23

4.1.3

启动

Spark

集群和

Thrift Server ....................................................................................................................... 23

4.1.4

命令参数

........................................................................................................................................................ 24

4.2 实战T

HRIFT SERVER .......................................................................................................................................... 25

4.2.1

远程客户端连接

............................................................................................................................................ 25

4.2.2

基本操作

........................................................................................................................................................ 26

4.2.3

计算所有订单每年的订单数

........................................................................................................................ 28

4.2.4

计算所有订单月销售额前十名

.................................................................................................................... 29

4.2.5

缓存表数据

.................................................................................................................................................... 31

4.2.6

在

IDEA

中

JDBC

访问

..................................................................................................................................... 32