Apache Flink 1.7版实时流处理指南

4星 · 超过85%的资源需积分: 41 10 浏览量更新于2024-07-17 1 收藏 9.93MB PDF 举报

《Stream Processing with Apache Flink》是一本由Fabian Hueske和Vasiliki Kalavri合著的英文专业书籍，专注于Apache Flink在大数据流处理领域的基础知识、实现方法和操作应用。该书于2019年首次发布，针对的是Flink版本1.7，这是当时最新的技术资料，适合对实时数据处理感兴趣的开发者和数据工程师。本书的核心内容包括以下几个方面： 1. **Flink基础**：书中详细介绍了Flink的架构和核心概念，如DataStream API、时间窗口、事件时间处理等，帮助读者理解这个流处理框架的工作原理。 2. **编程模型**：讲解了如何使用Java或Scala编写高效的流处理应用程序，包括事件驱动的数据处理逻辑、并行性和容错机制等。 3. **实时计算**：讨论了如何设计实时分析管道，如实时聚合、实时机器学习、实时警报系统等实际应用场景。 4. **分布式部署**：涵盖了Flink在集群环境中的部署和管理，包括资源调度、故障恢复和性能优化策略。 5. **实践案例**：书中还提供了丰富的实战案例，通过实际项目的剖析，让读者掌握如何将理论知识应用到实际工作场景中。 6. **最新版本特性**：作为1.7版本的指南，书中紧跟技术发展，介绍了该版本的新功能和技术改进，确保读者了解最前沿的Flink技术。 7. **版权限制与购买信息**：O'Reilly Media出版，适用于教育、商业或销售推广用途，提供在线版供读者选择。书中还提供了联系方式以便获取更多信息。《Stream Processing with Apache Flink》不仅是一本技术参考书，也是一本实用的教程，适合那些希望深入学习和掌握Flink流处理技术的专业人士，无论是在开发实时数据分析系统，还是在优化现有系统的性能时，都能从中获益匪浅。随着大数据和实时分析需求的增长，这本书的重要性在不断凸显，是提升个人技能和团队能力的宝贵资源。

Figure 1-4. A stateful streaming application

Stateful stream processing applications often ingest their incoming events

from an event log. An event log stores and distributes event streams.

Events are written to a durable, append-only log, which means that the

order of written events cannot be changed. A stream that is written to an

event log can be read many times by the same or different consumers. Due

to the append-only property of the log, events are always published to all

consumers in exactly the same order. There are several event log systems

available as open source software, Apache Kafka being the most popular,

or as integrated services offered by cloud computing providers.

Connecting a stateful streaming application running on Flink and an event

log is interesting for multiple reasons. In this architecture the event log

persists the input events and can replay them in deterministic order. In case

of a failure, Flink recovers a stateful streaming application by restoring its

state from a previous checkpoint and resetting the read position on the

event log. The application will replay (and fast forward) the input events

from the event log until it reaches the tail of the stream. This technique is

used to recover from failures but can also be leveraged to update an

Event-driven applications have quite high requirements on the stream

processor that runs them. Not all stream processors are equally well-suited

to run event-driven applications. The expressiveness of the API and the

quality of state handling and event-time support determine the business

logic that can be implemented and executed. This aspect depends on the

APIs of the stream processor, what kinds of state primitives it provides,

and the quality of its support for event-time processing. Moreover, exactly-

once state consistency and the ability to scale an application are

fundamental requirements for event-driven applications. Apache Flink

checks all these boxes and is a very good choice to run this class of

applications.

Data Pipelines

Today’s IT architectures include many different datastores, such as

relational and special-purpose database systems, event logs, distributed

filesystems, in-memory caches, and search indexes. All of these systems

store data in different formats and data structures that provide the best

performance for their specific access pattern. It is common that companies

store the same data in multiple different systems to improve the

performance of data accesses. For example, information for a product that

is offered in a webshop can be stored in a transactional database, a web

cache, and a search index. Due to this replication of data, the data stores

must be kept in sync.

A traditional approach to synchronize data in different storage systems is

periodic ETL jobs. However, they do not meet the latency requirements

for many of today’s use cases. An alternative is to use an event log to

distribute updates. The updates are written to and distributed by the event

log. Consumers of the log incorporate the updates into the affected data

stores. Depending on the use case, the transferred data may need to be

normalized, enriched with external data, or aggregated before it is ingested

by the target data store.

Ingesting, transforming, and inserting data with low latency is another

common use case for stateful stream processing applications. This type of

application is called a data pipeline. Data pipelines must be able to process

large amounts of data in a short time. A stream processor that operates a

剩余401页未读，继续阅读

酷乐喵

粉丝: 1
资源: 1

Apache Flink 1.7版实时流处理指南

Stream Processing with Apache Flink完整书签高清pdf和epub版，以及评价超高的Streaming Systems

Stream Processing with Apache Flink

Stream Processing with Apache Flink, Fabian Hueske.pdf

flink count+time trigger

写一个基于Flink1.14.3 Table SQL API的mysql列加工数据处理JAVA代码

Apache Doris Flink Connector

对学习flink有帮助的手机游戏

Flink推荐学习书籍

flink gpload

stream 根据某个字段分类

最新资源