利用Confluent构建事件驱动的实时流处理系统指南

需积分: 9 192 浏览量更新于2024-07-18 收藏 5.05MB PDF 举报

《设计事件驱动系统：Apache Kafka流服务的概念与模式》是Ben Stopford撰写的一本专业书籍，由O'Reilly Media出版。本书聚焦于利用Confluent平台构建实时数据处理系统，特别是那些基于事件驱动的架构。在当今企业环境中，随着大数据和云计算的发展，事件驱动系统已经成为关键的技术趋势，尤其在数据库和Kafka集成的场景下，Confluent的解决方案因其高效性和可扩展性受到重视。书中涵盖了设计事件驱动系统的核心概念和实践模式，旨在帮助读者理解和应用Apache Kafka在构建实时服务中的作用。作者详细阐述了如何通过Kafka的streaming功能来处理和分析海量事件流，以及如何设计出高可用、可扩展且响应快速的系统。这包括理解Kafka的消息模型、分区、复制、事务等核心组件的工作原理，以及如何设置和管理Kafka集群以支持高性能的事件流转。读者将学习到如何利用Kafka的事件驱动特性，实现实时数据分析、监控、预警和决策支持等功能，从而优化业务流程、提高响应速度和灵活性。书中还可能探讨了与微服务架构、流处理计算框架（如Apache Flink或Spark Streaming）和其他技术栈的集成，以构建一个完整的事件驱动生态系统。此外，作者还分享了自己的设计理念和最佳实践，旨在帮助读者避免常见的陷阱，确保设计出既符合业务需求又能有效应对复杂性挑战的事件驱动系统。书中提供的实例和案例研究有助于读者在实际项目中应用所学知识，提升自己的技能。本书适合对事件驱动系统感兴趣的开发人员、架构师、数据工程师以及任何希望通过Apache Kafka驱动实时数据处理的人士阅读。无论是初次接触Kafka还是希望深入理解其在现代IT架构中的角色，这本书都是不可或缺的参考资料。对于已经在使用Confluent的企业，本书能够提供实用的指导和优化策略，帮助他们进一步提升数据处理效率和系统的稳定性。

PART I

Setting the Stage

The truth is the log.

—Pat Helland, “Immutability Changes Everything,” 2015

CHAPTER 1

Introduction

While the main focus of this book is the building of event-driven systems of dif‐

ferent sizes, there is a deeper focus on software that spans many teams. This is

the realm of service-oriented architectures: an idea that arose around the start of

the century, where a company reconfigures itself around shared services that do

commonly useful things.

This idea became quite popular. Amazon famously banned all intersystem com‐

munications by anything that wasn’t a service interface. Later, upstart Netflix

went all in on microservices, and many other web-based startups followed suit.

Enterprise companies did similar things, but often using messaging systems,

which have a subtly different dynamic. Much was learned during this time, and

there was significant progress made, but it wasn’t straightforward.

One lesson learned, which was pretty ubiquitous at the time, was that service-

based approaches significantly increased the probability of you getting paged at 3

a.m., when one or more services go down. In hindsight, this shouldn’t have been

surprising. If you take a set of largely independent applications and turn them

into a web of highly connected ones, it doesn’t take too much effort to imagine

that one important but flaky service can have far-reaching implications, and in

the worst case bring the whole system to a halt. As Steve Yegge put it in his

famous Amazon/Google post, “Organizing into services taught teams not to trust

each other in most of the same ways they’re not supposed to trust external devel‐

opers.”

What did work well for Amazon, though, was the element of organizational

change that came from being wholeheartedly service based. Service teams think

of their software as being a cog in a far larger machine. As Ian Robinson put it,

“Be of the web, not behind the web.” This was a huge shift from the way people

built applications previously, where intersystem communication was something

teams reluctantly bolted on as an afterthought. But the services model made

interaction a first-class entity. Suddenly your users weren’t just customers or

businesspeople; they were other applications, and they really cared that your ser‐

vice was reliable. So applications became platforms, and building platforms is

hard.

LinkedIn felt this pain as it evolved away from its original, monolithic Java appli‐

cation into 800–1,100 services. Complex dependencies led to instability, version‐

ing issues caused painful lockstep releases, and early on, it wasn’t clear that the

new architecture was actually an improvement.

One difference in the way LinkedIn evolved its approach was its use of a messag‐

ing system built in-house: Kafka. Kafka added an asynchronous publish-

subscribe model to the architecture that enabled trillions of messages a day to be

transported around the organization. This was important for a company in

hypergrowth, as it allowed new applications to be plugged in without disturbing

the fragile web of synchronous interactions that drove the frontend.

But this idea of rearchitecting a system around events isn’t new—event-driven

architectures have been around for decades, and technologies like enterprise

messaging are big business, particularly with (unsurprisingly) enterprise compa‐

nies. Most enterprises have been around for a long time, and their systems have

grown organically, over many iterations or through acquisition. Messaging sys‐

tems naturally fit these complex and disconnected worlds for the same reasons

observed at LinkedIn: events decouple, and this means different parts of the

company can operate independently of one another. It also means it’s easier to

plug new systems into the real time stream of events.

A good example is the regulation that hit the finance industry in January 2018,

which states that trading activity has to be reported to a regulator within one

minute of it happening. A minute may seem like a long time in computing terms,

but it takes only one batch-driven system, on the critical path in one business

silo, for that to be unattainable. So the banks that had gone to the effort of instal‐

ling real-time trade eventing, and plumbed it across all their product-aligned

silos, made short work of these regulations. For the majority that hadn’t it was a

significant effort, typically resulting in half-hearted, hacky solutions.

So enterprise companies start out complex and disconnected: many separate,

asynchronous islands—often with users of their own—operating independently

of one another for the most part. Internet companies are different, starting life as

simple, front-facing web applications where users click buttons and expect things

to happen. Most start as monoliths and stay that way for some time (arguably for

longer than they should). But as internet companies grow and their business gets

more complex, they see a similar shift to asynchronicity. New teams and depart‐

ments are introduced and they need to operate independently, freed from the

synchronous bonds that tie the frontend. So ubiquitous desires for online utilit‐

ies, like making a payment or updating a shopping basket, are slowly replaced by

4 | Chapter 1: Introduction

a growing need for datasets that can be used, and evolved, without any specific

application lock-in.

But messaging is no panacea. Enterprise service buses (ESBs), for example, have

vocal detractors and traditional messaging systems have a number of issues of

their own. They are often used to move data around an organization, but the

absence of any notion of history limits their value. So, even though recent events

typically have more value than old ones, business operations still need historical

data—whether it’s users wanting to query their account history, some service

needing a list of customers, or analytics that need to be run for a management

report.

On the other hand, data services with HTTP-fronted interfaces make lookups

simple. Anyone can reach in and run a query. But they don’t make it so easy to

move data around. To extract a dataset you end up running a query, then period‐

ically polling the service for changes. This is a bit of a hack, and typically the

operators in charge of the service you’re polling won’t thank you for it.

But replayable logs, like Kafka, can play the role of an event store: a middle

ground between a messaging system and a database. (If you don’t know Kafka,

don’t worry—we dive into it in Chapter 4.) Replayable logs decouple services

from one another, much like a messaging system does, but they also provide a

central point of storage that is fault-tolerant and scalable—a shared source of

truth that any application can fall back to.

A shared source of truth turns out to be a surprisingly useful thing. Microservi‐

ces, for example, don’t share their databases with one another (referred to as the

IntegrationDatabase antipattern). There is a good reason for this: databases have

very rich APIs that are wonderfully useful on their own, but when widely shared

they make it hard to work out if and how one application is going to affect oth‐

ers, be it data couplings, contention, or load. But the business facts that services

do choose to share are the most important facts of all. They are the truth that the

rest of the business is built on. Pat Helland called out this distinction back in

2006, denoting it “data on the outside.”

But a replayable log provides a far more suitable place to hold this kind of data

because (somewhat counterintuitively) you can’t query it! It is purely about stor‐

ing data and pushing it to somewhere new. This idea of pure data movement is

important, because data on the outside—the data services share—is the most

tightly coupled of all, and the more services an ecosystem has, the more tightly

coupled this data gets. The solution is to move data somewhere that is more

loosely coupled, so that means moving it into your application where you can

manipulate it to your heart’s content. So data movement gives applications a

level of operability and control that is unachievable with a direct, runtime

dependency. This idea of retaining control turns out to be important—it’s the

same reason the shared database pattern doesn’t work out well in practice.

Introduction | 5

剩余165页未读，继续阅读

才满西湖

粉丝: 2
资源: 2

利用Confluent构建事件驱动的实时流处理系统指南

An Event-Driven Approach for E-Services System Design

Event-Driven Architecture Overview

confluent-kafka-go-producer-example：基于Confluent的Kafka Go库的Kafka生产者示例

请详细说明如何使用confluent-kafka库在Python中实现高效的Kafka生产者和消费者，包括API调用和性能优化策略。

ERROR: Could not build wheels for confluent-kafka, which is required to install pyproject.toml-based projects

Could not build wheels for confluent-kafka, which is required to install pyproject.toml-based projects

在使用confluent-kafka库进行Kafka生产者和消费者的实现过程中，如何保证数据交互的性能和可靠性？

如何使用confluent-kafka库在Python中创建Kafka生产者和消费者，以实现高效的数据交互？请提供相关的API调用示例。

请使用confluent-kafka库

ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (confluent-kafka)

最新资源