Apache Kafka实时数据流处理指南

需积分: 10 146 浏览量更新于2024-07-18 1 收藏 7.06MB PDF 举报

Kafka入门指南 Kafka是Apache开源项目，用于实时数据处理和流处理。Kafka的出现解决了大规模数据处理的难题，提供了高吞吐量、低延迟、fault-tolerant的数据处理方式。 **Kafka的特点** * **高吞吐量**：Kafka能够处理高达数十万的消息记录，每秒钟处理数十万条消息。 * **低延迟**：Kafka的延迟非常低，可以实时处理数据。 * **fault-tolerant**：Kafka具有高可靠性，可以在节点故障时继续工作。 * **可扩展性**：Kafka支持水平扩展，可以根据需要增加或减少节点。 **Kafka架构** Kafka架构主要由三个部分组成： * **Producer**：生产者负责将数据发送到Kafka集群。 * **Broker**：Broker是Kafka集群的核心组件，负责处理消息的存储和传输。 * **Consumer**：消费者负责从Kafka集群中读取数据。 **Kafka的应用场景** Kafka广泛应用于各个领域，包括： * **日志处理**：Kafka可以实时处理日志数据，提供实时日志分析和监控。 * **流处理**：Kafka可以处理实时流数据，提供实时数据处理和分析。 * **消息队列**：Kafka可以作为消息队列，提供高可靠性和高吞吐量的消息处理。 **Confluent** Confluent是Kafka的商业版本，提供了更多的功能和支持，包括： * **Confluent Open Source**：Confluent Open Source是Kafka的商业版本，提供了更多的功能和支持。 * **Confluent Enterprise**：Confluent Enterprise是Kafka的企业版本，提供了更多的功能和支持，包括客户支持和服务等。 **Kafka的优点** Kafka有很多优点，包括： * **高性能**：Kafka能够处理高吞吐量的数据，提供了高性能的数据处理。 * **高可扩展性**：Kafka支持水平扩展，可以根据需要增加或减少节点。 * **高可靠性**：Kafka具有高可靠性，可以在节点故障时继续工作。 **Kafka的缺点** Kafka也存在一些缺点，包括： * **复杂性**：Kafka的架构和配置相对复杂，需要专业的技术人员进行维护。 * **资源占用**：Kafka需要占用大量的资源，包括CPU、内存和存储空间。 Kafka是大规模数据处理和流处理领域的领军者，提供了高性能、低延迟和高可靠性的数据处理方式。但是，Kafka也存在一些缺点，需要专业的技术人员进行维护和配置。

a platform oriented around real-time streams of events. This means that not only can

it connect off-the-shelf applications and data systems, it can power custom applica‐

tions built to trigger off of these same data streams. We think this architecture cen‐

tered around streams of events is a really important thing. In some ways these flows

of data are the most central aspect of a modern digital company, as important as the

cash flows you’d see in a financial statement.

The ability to combine these three areas—to bring all the streams of data together

across all the use cases—is what makes the idea of a streaming platform so appealing

to people.

Still, all of this is a bit different, and learning how to think and build applications ori‐

ented around continuous streams of data is quite a mindshift if you are coming from

the world of request/response style applications and relational databases. This book is

absolutely the best way to learn about Kafka; from internals to APIs, written by some

of the people who know it best. I hope you enjoy reading it as much as I have!

— Jay Kreps

Cofounder and CEO at

Conuent

Foreword | xv

Preface

The greatest compliment you can give an author of a technical book is “This is the

book I wish I had when I got started with this subject.” This is the goal we set for our‐

selves when we started writing this book. We looked back at our experience writing

Kafka, running Kafka in production, and helping many companies use Kafka to build

software architectures and manage their data pipelines and we asked ourselves,

“What are the most useful things we can share with new users to take them from

beginner to experts?” This book is a reflection of the work we do every day: run

Apache Kafka and help others use it in the best ways.

We included what we believe you need to know in order to successfully run Apache

Kafka in production and build robust and performant applications on top of it. We

highlighted the popular use cases: message bus for event-driven microservices,

stream-processing applications, and large-scale data pipelines. We also focused on

making the book general and comprehensive enough so it will be useful to anyone

using Kafka, no matter the use case or architecture. We cover practical matters such

as how to install and configure Kafka and how to use the Kafka APIs, and we also

dedicated space to Kafka’s design principles and reliability guarantees, and explore

several of Kafka’s delightful architecture details: the replication protocol, controller,

and storage layer. We believe that knowledge of Kafka’s design and internals is not

only a fun read for those interested in distributed systems, but it is also incredibly

useful for those who are seeking to make informed decisions when they deploy Kafka

in production and design applications that use Kafka. The better you understand how

Kafka works, the more you can make informed decisions regarding the many trade-

offs that are involved in engineering.

One of the problems in software engineering is that there is always more than one

way to do anything. Platforms such as Apache Kafka provide plenty of flexibility,

which is great for experts but makes for a steep learning curve for beginners. Very

often, Apache Kafka tells you how to use a feature but not why you should or

shouldn’t use it. Whenever possible, we try to clarify the existing choices, the trade‐

xvii

offs involved, and when you should and shouldn’t use the different options presented

by Apache Kafka.

Who Should Read This Book

Kaa: e Denitive Guide was written for software engineers who develop applica‐

tions that use Kafka’s APIs and for production engineers (also called SREs, devops, or

sysadmins) who install, configure, tune, and monitor Kafka in production. We also

wrote the book with data architects and data engineers in mind—those responsible

for designing and building an organization’s entire data infrastructure. Some of the

chapters, especially chapters 3, 4, and 11 are geared toward Java developers. Those

chapters assume that the reader is familiar with the basics of the Java programming

language, including topics such as exception handling and concurrency. Other chap‐

ters, especially chapters 2, 8, 9, and 10, assume the reader has some experience run‐

ning Linux and some familiarity with storage and network configuration in Linux.

The rest of the book discusses Kafka and software architectures in more general

terms and does not assume special knowledge.

Another category of people who may find this book interesting are the managers and

architects who don’t work directly with Kafka but work with the people who do. It is

just as important that they understand the guarantees that Kafka provides and the

trade-offs that their employees and coworkers will need to make while building

Kafka-based systems. The book can provide ammunition to managers who would

like to get their staff trained in Apache Kafka or ensure that their teams know what

they need to know.

Conventions Used in This Book

The following typographical conventions are used in this book:

Italic

Indicates new terms, URLs, email addresses, filenames, and file extensions.

Constant width

Used for program listings, as well as within paragraphs to refer to program ele‐

ments such as variable or function names, databases, data types, environment

variables, statements, and keywords.

Constant width bold

Shows commands or other text that should be typed literally by the user.

Constant width italic

Shows text that should be replaced with user-supplied values or by values deter‐

mined by context.

xviii | Preface

剩余321页未读，继续阅读

林大嘟嘟

粉丝: 0

Apache Kafka实时数据流处理指南

深入理解Kafka：权威指南

Kafka权威指南：预览版阅读

Kafka权威指南：预览版

Spark: The Definitive Guide: Big Data Processing Made Simple 英文.pdf版

kafka_the definitive guide(201707)

Kafka_The Definitive Guide_Real-Time Data and Stream Processing at Scale

Kafka_The Definitive Guide_Real-Time Data and Stream Processing at Scale-2017

高清彩版 Kafka_The Definitive Guide_Real-Time Data and Stream Processing at Scale

kafka the definitive guide

Kafka the Definitive Guide 2nd Edition

最新资源