配置hive on spark spark版本2.4.5 hive版本3.1.2
时间: 2023-10-08 08:07:37 浏览: 213
首先,确保你的环境中已经安装了Spark和Hive,并且你已经配置好了环境变量。接下来,按照以下步骤配置Hive on Spark:
1. 在Hive的配置文件hive-site.xml中添加以下配置:
```
<property>
<name>hive.execution.engine</name>
<value>spark</value>
<description>Execution engine to use. Set this to Spark.</description>
</property>
<property>
<name>spark.master</name>
<value>local[*]</value>
<description>Spark master URL. Leave this unset to use the default local mode.</description>
</property>
<property>
<name>spark.submit.deployMode</name>
<value>client</value>
<description>Deployment mode of Spark driver program. Set this to client.</description>
</property>
<property>
<name>spark.sql.hive.convertMetastoreParquet</name>
<value>true</value>
<description>Whether to convert Parquet tables to Spark SQL on load. Set this to true.</description>
</property>
```
2. 将以下jar包添加到Hive的lib目录下:
```
spark-core_2.11-2.4.5.jar
spark-sql_2.11-2.4.5.jar
```
3. 启动Hive命令行客户端,并执行以下命令:
```
set spark.home=<spark_home_directory>
set hive.execution.engine=spark;
```
其中,`<spark_home_directory>`是你的Spark安装目录。
现在,你已经成功地配置了Hive on Spark。可以使用Hive命令行或Spark SQL来查询数据。
阅读全文