Apache Impala (incubating)：快速交互式大数据查询系统
需积分: 9 48 浏览量
更新于2024-07-18
收藏 7.95MB PDF 举报
"Apache Impala (incubating) Guide"
Apache Impala 是一个开源的数据处理系统，它提供了快速、交互式的SQL查询功能，可以直接在Apache Hadoop的数据存储（如HDFS、HBase或Amazon S3）上运行。Impala与Hive共享统一的存储平台、元数据、SQL语法（Hive SQL）、ODBC驱动和用户界面（如Hue中的Impala查询UI），从而为实时或批处理查询提供了一个熟悉且统一的平台。
Impala并不取代基于MapReduce的批处理框架，如Hive。Hive和其他基于MapReduce的工具最适合于长时间运行的批处理作业，如ETL类型的工作。Impala的引入是为了补充大数据查询的工具集，它专注于提供更快的查询性能。
Impala的主要特性包括：
1. **快速查询**：通过避免使用MapReduce，Impala能够进行低延迟的查询，适合实时分析。
2. **与Hadoop生态系统集成**：与HDFS、HBase等紧密集成，可以无缝地读取和处理存储在Hadoop中的数据。
3. **共享元数据和SQL语法**：使用相同的元数据和Hive SQL，简化了用户的学习曲线和操作流程。
4. **组件架构**：Impala服务器由多个组件组成，包括Impala Daemon（处理查询）、Impala Statestore（维护节点状态）和Impala Catalog Service（管理元数据）。
5. **编程接口**：支持多种编程接口，方便开发Impala应用程序。
6. **适应性**：Impala与Hive协同工作，允许用户在批处理和交互式查询之间切换，并且与Hadoop生态系统中的其他组件兼容。
7. **元数据管理**：Impala使用Hive Metastore来存储表和分区的信息，同时也直接与HDFS和HBase交互。
8. **硬件和软件需求**：Impala支持特定的操作系统，需要配置Hive Metastore、Java依赖项、网络设置以及满足特定的硬件和用户账户要求。
9. **集群规划**：包括对Impala部署的硬件规模、Schema设计等方面的考虑，以确保最佳性能。
安装和管理Impala涉及到安装过程后的配置步骤，包括设置Impala的配置参数，调整以适应特定环境的需求。这些配置可能涉及内存分配、并发查询限制、日志记录等，以优化Impala在生产环境中的性能和稳定性。
Apache Impala是一个针对大数据的高性能查询引擎，旨在提升Hadoop生态系统的实时分析能力，同时保持与现有工具的兼容性，使得数据分析更为便捷和高效。通过理解和利用其特性和集成机制，用户可以构建更强大的大数据处理和分析解决方案。
| Impala Concepts and Architecture | 16
can stop the Impala service, delete the Impala StateStore and Impala Catalog Server roles, add the roles on a
different host, and restart the Impala service.
Note:
In Impala 1.2.4 and higher, you can specify a table name with INVALIDATE METADATA after the table is created
in Hive, allowing you to make individual tables visible to Impala without doing a full reload of the catalog metadata.
Impala 1.2.4 also includes other changes to make the metadata broadcast mechanism faster and more responsive,
especially during Impala startup. See New Features in Impala 1.2.4 on page 739 for details.
Related information: Modifying Impala Startup Options on page 32, Starting Impala on page 31, Ports
Used by Impala on page 715
Developing Impala Applications
The core development language with Impala is SQL. You can also use Java or other languages to interact with Impala
through the standard JDBC and ODBC interfaces used by many business intelligence tools. For specialized kinds of
analysis, you can supplement the SQL built-in functions by writing user-defined functions (UDFs) in C++ or Java.
Overview of the Impala SQL Dialect
The Impala SQL dialect is highly compatible with the SQL syntax used in the Apache Hive component (HiveQL).
As such, it is familiar to users who are already familiar with running SQL queries on the Hadoop infrastructure.
Currently, Impala SQL supports a subset of HiveQL statements, data types, and built-in functions. Impala also
includes additional built-in functions for common industry features, to simplify porting SQL from non-Hadoop
systems.
For users coming to Impala from traditional database or data warehousing backgrounds, the following aspects of the
SQL dialect might seem familiar:
•
The SELECT statement includes familiar clauses such as WHERE, GROUP BY, ORDER BY, and WITH. You
will find familiar notions such as joins, built-in functions for processing strings, numbers, and dates, aggregate
functions, subqueries, and comparison operators such as IN() and BETWEEN. The SELECT statement is the
place where SQL standards compliance is most important.
•
From the data warehousing world, you will recognize the notion of partitioned tables. One or more columns
serve as partition keys, and the data is physically arranged so that queries that refer to the partition key columns
in the WHERE clause can skip partitions that do not match the filter conditions. For example, if you have 10 years
worth of data and use a clause such as WHERE year = 2015, WHERE year > 2010, or WHERE year IN
(2014, 2015), Impala skips all the data for non-matching years, greatly reducing the amount of I/O for the
query.
•
In Impala 1.2 and higher, UDFs let you perform custom comparisons and transformation logic during SELECT
and INSERT...SELECT statements.
For users coming to Impala from traditional database or data warehousing backgrounds, the following aspects of the
SQL dialect might require some learning and practice for you to become proficient in the Hadoop environment:
• Impala SQL is focused on queries and includes relatively little DML. There is no UPDATE or DELETE statement.
Stale data is typically discarded (by DROP TABLE or ALTER TABLE ... DROP PARTITION statements) or
replaced (by INSERT OVERWRITE statements).
• All data creation is done by INSERT statements, which typically insert data in bulk by querying from other tables.
There are two variations, INSERT INTO which appends to the existing data, and INSERT OVERWRITE which
replaces the entire contents of a table or partition (similar to TRUNCATE TABLE followed by a new INSERT).
Although there is an INSERT ... VALUES syntax to create a small number of values in a single statement, it is
far more efficient to use the INSERT ... SELECT to copy and transform large amounts of data from one table
to another in a single operation.
• You often construct Impala table definitions and data files in some other environment, and then attach Impala so
that it can run real-time queries. The same data files and table metadata are shared with other components of the
Hadoop ecosystem. In particular, Impala can access tables created by Hive or data inserted by Hive, and Hive can
剩余807页未读，继续阅读
2021-05-30 上传
2020-03-28 上传
2023-07-24 上传
2021-06-14 上传
2019-01-21 上传
weixin_43579062
- 粉丝: 0
- 资源: 1
我的内容管理
展开
