构建云端大数据分析：Azure 数据湖、HDInsight 和 Spark 深入解析

需积分: 9 149 浏览量更新于2024-07-18 收藏 20.76MB PDF 举报

"Mastering Azure Analytics" 由 Zoiner Tejada 撰写，旨在帮助读者理解微软Azure平台上的众多服务，特别是针对大数据分析解决方案的20多种平台即服务（PaaS）产品。这本书提供了一个参考框架，帮助读者在构建自己的大数据分析解决方案时做出明智选择。在内容部分，提到了Architecting in the Cloud with Azure Data Lake, HDInsight, and Spark，这表明本书将深入探讨如何利用这些特定的Azure服务来构建云中的数据分析架构。Azure Data Lake是一个企业级的大数据存储解决方案，支持PB级别的数据存储，适用于结构化和非结构化的数据。它提供了高性能的数据访问，使得大规模数据分析变得可能。 HDInsight是Azure提供的一个托管Hadoop、Spark、Hive、Pig和Storm服务，让用户可以在云中轻松创建和管理大数据集群。通过HDInsight，用户可以处理和分析海量数据，而无需预先投资硬件或深入了解分布式计算的复杂性。 Apache Spark是用于大数据处理的开源框架，以其高速处理能力而著称，尤其适合实时流处理和机器学习任务。在Azure中集成Spark，用户可以快速开发和部署Spark应用程序，实现高效的数据洞察。本书会详细讲解如何结合使用这些服务，以实现高效的数据湖架构，包括数据摄入、存储、处理、分析和可视化等步骤。作者可能会涵盖如何设置和管理Data Lake存储，如何配置和扩展HDInsight集群，以及如何利用Spark进行复杂的数据分析和机器学习模型训练。此外，读者还将了解到如何利用Azure的其他相关服务，如Azure Stream Analytics（用于实时流处理）和Power BI（用于商业智能和数据可视化），以构建端到端的大数据分析解决方案。书中可能会包含实际案例研究和最佳实践，帮助读者将理论知识应用到实际项目中。 "Mastering Azure Analytics" 是一本全面介绍Azure大数据分析服务的指南，适合数据科学家、数据工程师和IT专业人士，他们希望充分利用Azure云平台来解决复杂的数据挑战，并构建高效率、可扩展的大数据分析系统。

In this definition, a data lake is a repository that intentionally leaves the data in its

raw or least-processed form to allow questions to be asked of it that would not be

answerable if the data were packaged into a particular structure or otherwise aggrega‐

ted.

That simple definition of a data lake should serve as the core, but as you will see in

reading this book, it belies the true extent of a data lake. In reality, a data lake includes

not just a single processing engine, but multiple processing engines, and because it

represents the enterprise-wide, centralized repository of source and processed data

(after all, it champions a “store all” approach to data management), it has other

requirements such as metadata management, discovery, and governance.

One final important note: the data lake concept as it is used today is intended for

batch processing, where high latency (time until results ready) is appropriate. That

said, support for lower-latency processing is a natural area of evolution for data lakes,

so this definition may evolve with the technology landscape.

With this broad definition of data lake, let us look at two different architectures that

can be used to act on the data managed by a data lake: lambda architecture and kappa

architecture.

Lambda Architecture

Lambda architecture was originally proposed by the creator of Apache Storm, Nathan

Marz. In his book, Big Data: Principles and Best Practices of Scalable Realtime Data

Systems (Manning), he proposed a pipeline architecture that aims to reduce the com‐

plexity seen in real-time analytics pipelines by constraining any incremental compu‐

tation to only a small portion of this architecture.

In lambda architecture, there are two paths for data to flow in the pipeline (see

Figure 1-2):

• A “hot” path where latency-sensitive data (e.g., the results need to be ready in

seconds or less) flows for rapid consumption by analytics clients

• A “cold” path where all data goes and is processed in batches that can tolerate

greater latencies (e.g., the results can take minutes or even hours) until results are

ready

When data flows into the “cold” path, this data is immutable. Any changes to the

value of particular datum are reflected by a new, timestamped datum being stored in

the system alongside any previous values. This approach enables the system to re-

compute the then-current value of a particular datum for any point in time across the

history of the data collected. Because the “cold” path can tolerate a greater latency

until the results are ready, the computation can afford to run across large data sets,

Lambda Architecture | 3

and the types of calculation performed can be time-intensive. The objective of the

“cold” path can be summarized as: take the time you need, but make the results

extremely accurate.

When data flows into the “hot” path, this data is mutable and can be updated in place.

In addition, the hot path places a latency constraint on the data (as the results are typ‐

ically desired in near–real time). The impact of this latency constraint is that the types

of calculations that can be performed are limited to those that can happen quickly

enough. This might mean switching from an algorithm that provides perfect accuracy

to one that provides an approximation. An example of this involves counting the

number of distinct items in a data set (e.g., the number of visitors to your website):

you can either count each individual datum (which can be very high latency if the

volume is high) or you can approximate the count using algorithms like

HyperLogLog. The objective of the hot path can be summarized as: trade off some

amount of accuracy in the results in order to ensure that the data is ready as quickly

as possible.

Figure 1-2. e lambda architecture captures all data entering the pipeline into immuta‐

ble storage, labeled “Master Data” in the diagram. is data is processed by the batch

layer and output to a serving layer in the form of batch views. Latency-sensitive calcula‐

tions are applied on the input data by the speed layer and exposed as real-time views.

Analytics clients can consume the data from either the speed layer views or the serving

layer views depending on the time frame of the data required. In some implementations,

the serving layer can host both the real-time views and the batch views.

The hot and cold paths ultimately converge at the analytics client application. The cli‐

ent must choose the path from which it acquires the result. It can choose to use the

less accurate but most up-to-date result from the hot path, or it can use the less timely

but more accurate result from the cold path. An important component of this deci‐

sion relates to the window of time for which only the hot path has a result, as the cold

path has not yet computed the result. Looking at this another way, the hot path has

results for only a small window of time, and its results will ultimately be updated by

4 | Chapter 1: Enterprise Analytics Fundamentals

the more accurate cold path in time. This has the effect of minimizing the volume of

data that components of the hot path have to deal with.

The motivation for the creation of the lambda architecture may be surprising. Yes,

enabling a simpler architecture for real-time data processing was important, but the

reason it came into existence was to provide human fault tolerance. In effect, it recog‐

nizes that we are moving to a time when we actually can keep all the raw data. Simul‐

taneously, it recognizes that bugs happen, even in production. Lambda architectures

offer a solution that is not just resilient to system failure, but tolerant of human mis‐

takes because it has all the input data and the capability to recompute (through batch

computation) any errant calculation.

Kappa Architecture

Kappa architecture surfaced in response to a desire to simplify the lambda architec‐

ture dramatically by making a single change: eliminate the cold path and make all

processing happen in a near–real-time streaming mode (Figure 1-3). Recomputation

on the data can still occur when needed; it is in effect streamed through the kappa

pipeline again. The kappa architecture was proposed by Jay Kreps based on his expe‐

riences at LinkedIn, and particularly his frustrations in dealing with the problem of

“code sharing” in lambda architectures—that is, keeping in sync the logic that does

the computation in the hot path with the logic that is doing the same calculation in

the cold path.

Figure 1-3. In the kappa architecture, analytics clients get their data only from the speed

layer, as all computation happens upon streaming data. Input events can be mirrored to

long-term storage to enable recomputation on historical data should the need arise.

Kappa architecture centers on a unified log (think of it as a highly scalable queue),

which ingests all data (which are considered events in this architecture). There is a

single deployment of this log in the architecture, whereby each event datum collected

is immutable, the events are ordered, and the current state of an event is changed only

by a new event being appended.

Kappa Architecture | 5

The unified log itself is designed to be distributed and fault tolerant, suitable to its

place at that heart of the analytics topology. All processing of events is performed on

the input streams and persisted as a real-time view (just as in the hot path of the

lambda architecture). To support the human-fault-tolerant aspects, the data ingested

from the unified log is typically persisted to a scalable, fault-tolerant persistent stor‐

age so that it can be recomputed even if the data has “aged out” of the unified log.

Kreps on Kappa

If you’re interested in reading more about kappa architecture, take

a look at I Heart Logs by Jay Kreps (O’Reilly), as Kreps applied it

toward event log processing and analytics.

If this architecture sounds vaguely familiar to you, it is probably because it is. The

patterns employed by the kappa architecture are the same as those you may have

come across if you have used the Event Sourcing pattern or CQRS (command query

responsibility segregation).

Choosing Between Lambda and Kappa

Arguing the merits of lambda architecture over kappa architecture and vice versa is

akin to arguing over programming languages—it quickly becomes a heated, quasi-

religious debate. Instead, for the purposes of this book we aim to use both architec‐

tures as motivations to illustrate how you can design and implement such pipelines in

Microsoft Azure. We leave it to you, the reader, to decide which architecture most

closely matches the needs of your analytics data pipeline.

The Azure Analytics Pipeline

In this book we will expand on the analytics data pipeline to understand the ways we

can build the one required by a particular scenario. We attempt to do this in two

directions: first, by broadly showing the lay of the land for all the Azure services in

the context of where they apply to the pipeline; and second, by taking on specific sce‐

narios that enable us to apply a subset of the services in implementing a solution for

that scenario. We will explore the concepts of data lakes, lambda architectures, and

kappa architectures in our solutions, and show how we can achieve them using the

latest services from Microsoft Azure.

Throughout this book, we will tease out the analytics data pipeline into more and

more granular components, so that we can categorically identify the Azure services

that act in support of a particular component. We will expand our analytics data pipe‐

line (source, ingest, storage, processing, delivery) with the following sub-

components, as illustrated by Figure 1-4.

6 | Chapter 1: Enterprise Analytics Fundamentals

剩余410页未读，继续阅读

无忧无悔无憾

粉丝: 2
资源: 58

构建云端大数据分析：Azure 数据湖、HDInsight 和 Spark 深入解析

Mastering Predictive Analytics with R(2nd) azw3

Mastering Predictive Analytics with R(2nd)

Mastering Predictive Analytics with Python

mastering go 2 pdf

mastering docker 4 pdf

发那科mastering恢复

mastering apache pulsar pdf

mastering docker – second edition

mastering sfml game development pdf

mastering your phd 的pdf版

最新资源