大数据上的SQL技术、架构与创新

需积分: 10 34 浏览量更新于2024-07-20 收藏 6.26MB PDF 举报

"SQL on Big Data: Technology, Architecture, and Innovation" 是一本由Sumit Pal撰写的专业书籍，出版于Apress（2016年），主要探讨了如何在大数据平台上执行SQL以及各种商业和开源产品。书中深入分析了不同SQL引擎的架构、执行方式、数据移动、延迟、可扩展性、性能和系统需求。在本书中，作者首先回顾了SQL在大数据中的历史和发展，解释了为何需要在大数据上使用SQL。接着，它详细介绍了以下四个关键领域的架构和创新： 1. 批处理架构：读者将了解Hive等现有SQL引擎的内部工作原理，以及它们如何不断进化以支持新功能并降低查询延迟。Hive是大数据处理中的一个常用工具，它允许用户使用SQL语句来操作分布式数据。 2. 交互式架构：这部分讲解了如何设计SQL引擎以支持对大型数据集的低延迟查询。这些引擎通常采用优化的数据存储和查询策略，以实现快速响应。 3. 流式处理架构：书中讨论了如何构建SQL引擎以处理数据流，利用内存和无锁数据结构来支持对动态数据的实时查询。这对于实时分析和决策至关重要。 4. 操作性架构：这部分介绍SQL引擎如何应用于事务性和操作性系统，以支持大数据平台上的事务处理。这包括对ACID（原子性、一致性、隔离性、持久性）属性的支持，确保数据的一致性和可靠性。 5. 创新架构：作者还探索了正在快速发展的新型SQL引擎，它们引入了创新的思想和概念，旨在解决性能、可扩展性和处理不同类型数据的问题。这些新技术可能涉及到混合事务/分析处理（HTAP），即在同一系统中同时支持在线事务处理(OLTP)和在线分析处理(OLAP)。这本书对于希望理解和应用SQL在大数据环境中的专业人士来说是一份宝贵的资源。通过学习，读者能够更好地理解如何选择和优化适合特定业务需求的大数据SQL解决方案，以及如何应对大数据场景下的性能挑战和集成问题。

CHAPTER 1 ■ WHY SQL ON BIG DATA?

Why SQL on Big Data?

Enterprise data hubs are being created with Hadoop and HDFS as a central data

repository for data from various sources, including operational systems, social media, the

Web, sensors, smart devices, as well as applications. Big data tools and frameworks are

then used to manage and run analytics to build data-driven products and gain actionable

insights from this data.

Despite its power, Hadoop has remained a tool for data scientists and developers and

is characterized by the following:

• Hadoop is not designed to answer analytics questions at business speed.

• Hadoop is not built to handle high-volume user concurrency.

In short, Hadoop is not consumable for business users.

With increasing adoption of big data tools by organizations, enterprises must figure

out how to leverage their existing BI tools and applications to overcome challenges

associated with massive data volumes, growing data diversity, and increasing information

demands. Existing enterprise tools for transactional, operational, and analytics workloads

struggle to deliver, suffering from slow response times, lack of agility, and an inability

to handle modern data types and unpredictable workload patterns. As enterprises start

to move their data to big data platforms, a plethora of SQL–on–big data technologies

has emerged to solve the challenges mentioned. The “SQL on big data” movement has

matured rapidly, though it is still evolving, as shown in Figure

1-2 .

Hadoop is designed to work with any data type—structured, unstructured, semi-

structured—which makes it very flexible, but, at the same time, working with it becomes

an exercise to use the lowest level APIs. This comprised a steep learning curve and makes

writing simple operations very time-consuming, with voluminous amounts of code. Hadoop’s

architecture leads to an impedance mismatch between data storage and data access.

While unstructured and streaming data types get a lot of attention for big data

workloads, a majority of enterprise applications still involve working with data that keeps

their businesses and systems working for their organizational purposes, also referred

to as operational data. Until a couple of years ago, Hive was the only available tool to

perform SQL on Hadoop. Today, there are more than a dozen competing commercial and

open source products for performing SQL on big data. Each of these tools competes on

latency, performance, scalability, compatibility, deployment options, and feature sets.

Figure 1-2. SQL tools on big data—a time line

Avrilia Floratou, Umar Farooq Minhas, and Fatma Özcan, “SQL-on-Hadoop: Full Circle Back to

Shared-Nothing Database Architectures,”

http://www.vldb.org/pvldb/vol7/p1295-floratou.pdf , 2014.

CHAPTER 1 ■ WHY SQL ON BIG DATA?

Traditionally, big data tools and technologies have mostly focused on building

solutions in the analytic space, from simple BI to advanced analytics. Use of big data

platforms in transactional and operational systems has been very minimal. With changes

to SQL engines on Hadoop, such as Hive 0.13 and later versions supporting transactional

semantics and the advent of open source products like Trafodion, and vendors such as

Splice Machines, building operational systems based on big data technologies seems to

be a possibility now.

SQL on big data queries fall broadly into five different categories:

• Reporting queries

• Ad hoc queries

• Iterative OLAP (OnLine Analytical Processing) queries

• Data mining queries

• Transactional queries

Why RDBMS Cannot Scale

Traditional database systems operate by reading data from disk, bringing it across an

I/O (input/output) interconnect, and loading data into memory and into a CPU cache

for data processing. Transaction processing applications, typically called OnLine

Transactional Processing (OLTP) systems, have a data flow that involves random I/O.

When data volumes are larger, with complex joins requiring multiphase processing,

data movement across backplanes and I/O channels works poorly. RDBMS (Relational

Database Management Systems) were initially designed for OLTP-based applications.

Data warehousing and analytics are all about data shuffling—moving data through

the processing engine as efficiently as possible. Data throughput is a critical metric in

such data warehouse systems. Using RDBMS designed for OLTP applications to build and

architect data warehouses results in reduced performance.

Most shared memory databases, such as MySQL, PostgreSQL, and SQL Server

databases, start to encounter scaling issues at terabyte size data without manual sharding.

However, manual sharding is not a viable option for most organizations, as it requires

a partial rewrite of every application. It also becomes a maintenance nightmare to

periodically rebalance shards as they grow too large.

Shared disk database systems, such as Oracle and IBM DB2, can scale up beyond

terabytes, using expensive, specialized hardware. With costs that can exceed millions per

server, scaling quickly becomes cost-prohibitive for most organizations.

SQL-on-Big-Data Goals

A SQL on-big-data solution has many goals, including to do exactly the same kind of

operations as in a traditional RDBMS, from an OLTP perspective or an OLAP/analytic

queries perspective. This book focuses on the analytic side of SQL-on-big-data solutions,

with architectural explanations for low-latency analytic queries. It also includes sections

to help understand how traditional OLTP-based solutions are implemented with SQL-on-

big-data solutions.

CHAPTER 1 ■ WHY SQL ON BIG DATA?

Some of the typical goals of an SQL-on-big-data solution include the following:

• Distributed, scale-out architecture : The idea is to support SQL on

distributed architectures to scale out data storage and compute

across clusters of machines. Before the advent of SQL on big data,

distributed architectures for storage and to compute were far and

few and extremely expensive.

Databases such as SQLServer, MySQL, and Postgres can’t

scale out without the heavy coding required to manually

shard and use the sharding logic at the application tier. Shared

disk databases such as Oracle or IBM DB2 are too expensive

to scale out, without spending millions on licensing.

• Avoid data movement from HDFS (Hadoop Distributed File

System) to external data stores : One of the other goals of

developing an SQL-on-big-data solution is to prevent data

movement from the data hub (HDFS) to an external store for

performing analytics. An SQL engine that could operate with

the data stored in the data node to perform the computation

would result in a vastly lower cost of data storage and also avoid

unnecessary data movement and delays to another data store for

performing analytics.

• An alternative to expensive analytic databases and appliances :

Support low-latency scalable analytic operations on large data

sets at a lower cost. Existing RDBMS engines are vertically scaled

machines that reach a ceiling in performance and scalability after

a certain threshold in data size. The solution was to invest in either

appliances that were extremely costly MPP (Massively Parallel

Processing) boxes with innovative architectures and solutions or

using scalable distributed analytic databases that were efficient,

based on columnar design and compression.

• Immediate availability of ingested data : The SQL on big data

has a design goal of accessing data as it is written, directly on

the storage cluster, instead of taking it out of the HDFS layer

and persisting it in a different system for consumption. This

can be called a “query-in-place” approach, and its benefits are

significant.

• Agility is enhanced, as consumption no longer requires

schema, ETL, and metadata changes.

• Lower operational cost and complexity result, as there is no

need to maintain a separate analytic database and reduce

data movement from one system to another. There is cost

savings of storage, licenses, hardware, process, and people

involved in the process.

CHAPTER 1 ■ WHY SQL ON BIG DATA?

• Unstructured data processing : With the schema-on-demand

approach in Hadoop, data is written to HDFS in its “native”

format. Providing access to semi-structured data sets based on

JSON/XML through an SQL query engine serves two purposes: it

becomes a differentiator for an SQL-on-big-data product, and it

also allows existing BI tools to communicate with these semi-

structured data sets, using SQL.

• Integrate—with existing BI tools : The ability to seamlessly

integrate with existing BI tools and software solutions. Use

existing SQL apps and BI tools and be productive immediately, by

connecting them to HDFS.

SQL-on-Big-Data Landscape

There is huge excitement and frantic activity in the field of developing SQL solutions

for big data/Hadoop. A plethora of tools has been developed, either by the open source

community or by commercial vendors, for making SQL available on the big data platform.

This is a fiercely competitive landscape wherein each tool/vendor tries to compete on any

of the given dimensions: low latency, SQL feature set, semi-structured or unstructured

data handling capabilities, deployment/ease of use, reliability, fault tolerance, in-

memory architecture, and so on. Each of these products and tools in the market has been

innovated either with a totally new approach to solving SQL-on-big-data problems or has

retrofitted some of the older ideas from the RDBMS world in the world of big data storage

and computation.

However, there is one common thread that ties these tools together: they work on

large data sets and are horizontally scalable.

SQL-on-big-data systems can be classified into two categories: native Hadoop-based

systems and database-Hadoop hybrids, in which the idea is to integrate existing tools

with the Hadoop ecosystem to perform SQL queries. Tools such as Hive belong to the first

category, while tools such as Hadapt, Microsoft PolyBase, and Pivotal’s HAWQ belong

to the second category. These tools heavily use the in-built database query optimization

techniques—a thoroughly researched area since the 1970s—and planning to schedule

query fragments and directly read HDFS data into database workers for processing.

Analytic appliance-based products have developed connectors to big data storage

systems, whether it is HDFS or NoSQL databases, and they work by siphoning off the data

from these storage systems and perform the queries within the appliance’s proprietary

SQL engine.

In this section, let’s look at the available products for SQL on big data—both open

source and commercial.

Figure

1-4 shows some of the SQL engines and products that work on a big data

platform. Tools on the right show open source products, while those on the left indicate

commercial products.

剩余164页未读，继续阅读

vanridin

粉丝: 108
资源: 1187

大数据上的SQL技术、架构与创新

Big Data SMACKBig Data SMACK(Apress,2016)

Big Data Analytics with Spark(Apress,2016)

Scalable Big Data Architecture(Apress,2015)

Beginning Big Data with Power BI and Excel 2013(Apress,2015)

Practical Hive(Apress,2016)

Pro Apache Phoenix(Apress,2016)

Scalable Big Data Architecture pdf 无水印 0分

Big Data Made Easy （英文高清原版）

Big Data Analytics with Spark 无水印pdf 0分

2025年软考高级 - 信息系统项目管理师考试备考全攻略

最新资源