Apache Flink入门指南：实时流处理详解

需积分: 10 200 浏览量更新于2024-07-17 收藏 1.85MB PDF 举报

《Apache Flink入门指南》是一本由Data Artisans公司出品的专业小册子，由Ellen Friedman和Kostas Tzoumas撰写，旨在为读者提供对Apache Flink流处理技术的深入理解和实用指导。这本书适合快速学习者，特别是那些希望在实时和未来应用场景中运用Flink进行数据处理的人。 Flink是专为现代数据处理设计的开源框架，强调了低延迟、高吞吐量和容错性。书中首先介绍了Flink的基本概念，包括流处理与批处理的对比，以及它在实时计算中的优势，如事件时间（Event Time）和处理时间（Processing Time）的概念。此外，作者还深入解析了Flink的DataStream API和Table API，这两个API是Flink处理数据的主要工具，允许开发者编写高效且可扩展的流处理应用程序。书中详细阐述了Flink的流式窗口（Windowing）操作，这是实现复杂实时分析的关键特性，如滑动窗口（Sliding Windows）和会话窗口（Session Windows）。Flink的事件驱动模型和数据并行处理能力也是亮点，这些特性使得系统能够处理大规模的数据流并在分布式环境中保持高性能。章节中还涵盖了Flink的状态管理、故障恢复机制以及与外部系统的集成，比如与Kafka、Hadoop等大数据生态系统中的组件无缝对接。此外，书中也提到了Flink的生态，包括社区贡献、最佳实践和生态系统中的其他工具，帮助读者更好地融入Flink的开发环境。《Apache Flink入门指南》不仅提供了理论知识，还包含了许多实战示例，让读者通过实践掌握Flink的核心概念和技术。最后，版权信息表明本书享有所有权利，并提醒读者合法购买和使用，同时注明了编辑、生产编辑、校对人员等参与本书制作的人员名单。《Introduction to Apache Flink》是一本实用的教程，无论是初学者还是进阶开发者，都能从中获益，掌握如何在实时和分布式场景下利用Flink进行高效、灵活的数据处理。

exactly-once guarantees for maintaining accurate state, and even the

guarantees that Storm could provide came at a high overhead.

Overview of Lambda Architecture: Advantages and

Limitations

The need for affordable scale drove people to distributed file sys‐

tems such as HDFS and batch-based computing (MapReduce jobs).

But that approach made it difficult to deal with low-latency

insights. Development of real-time stream processing technology

with Apache Storm helped address the latency issue, but not as a

complete solution. For one thing, Storm did not guarantee state

consistency with exactly-once processing and did not handle event-

time processing. People who had these needs were forced to imple‐

ment these features in their application code.

A hybrid view of data analytics that mixed these approaches offered

one way to deal with these challenges. This hybrid, called Lambda

architecture, provided delayed but accurate results via batch Map‐

Reduce jobs and an in-the-moment preliminary view of new results

via Storm’s processing.

The Lambda architecture is a helpful framework for building big

data applications, but it is not sufficient. For example, with a

Lambda system based on MapReduce and HDFS, there is a time

window, in hours, when inaccuracies due to failures are visible.

Lambda architectures need the same business logic to be coded

twice, in two different programming APIs: once for the batch sys‐

tem and once for the streaming system. This leads to two codebases

that represent the same business problem, but have different kinds

of bugs. In practice, this is very difficult to maintain.

To compute values that depend on multiple

streaming events, it is necessary to retain data

from one event to another. This retained data is

known as the state of the computation. Accurate

handling of state is essential for consistency in

computation. The ability to accurately update

state after a failure or interruption is a key to

fault tolerance.

8 | Chapter 1: Why Apache Flink?

It’s hard to maintain fault-tolerant stream processing that has high

throughput with very low latency, but the need for guarantees of

accurate state motivated a clever compromise: what if the stream of

data from continuous events were broken into a series of small,

atomic batch jobs? If the batches were cut small enough—so-called

“micro-batches”—your computation could approximate true

streaming. The latency could not quite reach real time, but latencies

of several seconds or even subseconds for very simple applications

would be possible. This is the approach taken by Apache Spark

Streaming, which runs on the Spark batch engine.

More important, with micro-batching, you can achieve exactly-once

guarantees of state consistency. If a micro-batch job fails, it can be

rerun. This is much easier than would be true for a continuous

stream-processing approach. An extension of Storm, called Storm

Trident, applies micro-batch computation on the underlying stream

processor to provide exactly-once guarantees, but at a substantial

cost to latency.

However, simulating streaming with periodic batch jobs leads to

very fragile pipelines that mix DevOps with application develop‐

ment concerns. The time that a periodic batch job takes to finish is

tightly coupled with the timing of data arrival, and any delays can

cause inconsistent (a.k.a. wrong) results. The underlying problem

with this approach is that time is only managed implicitly by the

part of the system that creates the small jobs. Frameworks like Spark

Streaming mitigate some of the fragility, but not entirely, and the

sensitivity to timing relative to batches still leads to poor latency and

a user experience where one needs to think a lot about performance

in the application code.

These tradeoffs between desired capabilities have motivated contin‐

ued attempts to improve existing processors (for example, the devel‐

opment of Storm Trident to try to overcome some of the limitations

of Storm). When existing processors fall short, the burden is placed

on the application developer to deal with any issues that result. An

example is the case of micro-batching, which does not provide an

excellent fit between the natural occurrence of sessions in event data

and the processor’s need to window data only as multiples of the

batch time (recovery interval). With less flexibility and expressivity,

development time is slower and operations take more effort to

maintain properly.

Evolution of Stream Processing Technologies | 9

to a low-level programming API, it does not support event time, and

it does not have support for batch computations). And none of these

projects have been able to attract an open source community com‐

parable to the Flink community.

Now, let’s take a look at what Flink is and how the project came

about.

First Look at Apache Flink

The Apache Flink project home page starts with the tagline, “Apache

Flink is an open source platform for distributed stream and batch

data processing.” For many people, it’s a surprise to realize that Flink

not only provides real-time streaming with high throughput and

exactly-once guarantees, but it’s also an engine for batch data pro‐

cessing. You used to have to choose between these approaches, but

Flink lets you do both with one technology.

How did this top-level Apache project get started? Flink has its ori‐

gins in the Stratosphere project, a research project conducted by

three Berlin-based Universities as well as other European Universi‐

ties between 2010 and 2014. The project had already attracted a

broader community base, in part through presentations at several

public developer conferences including Berlin Buzzwords, NoSQL

Matters in Cologne, and others. This strong community base is one

reason the project was appropriate for incubation under the Apache

Software Foundation.

A fork of the Stratosphere code was donated in April 2014 to the

Apache Software Foundation as an incubating project, with an ini‐

tial set of committers consisting of the core developers of the sys‐

tem. Shortly thereafter, many of the founding committers left

university to start a company to commercialize Flink: data Artisans.

During incubation, the project name had to be changed from Strato‐

sphere because of potential confusion with an unrelated project. The

name Flink was selected to honor the style of this stream and batch

processor: in German, the word “flink” means fast or agile. A logo

showing a colorful squirrel was chosen because squirrels are fast,

agile and—in the case of squirrels in Berlin—an amazing shade of

reddish-brown, as you can see in Figure 1-3.

First Look at Apache Flink | 11

剩余108页未读，继续阅读

jackycai1983

粉丝: 0
资源: 7

Apache Flink入门指南：实时流处理详解

Introduction_to_Apache_Flink_MapR_final

Learning Apache Flink电子版

Stream_Processing_with_Apache_Flink.zip

最新资源