Kafka权威指南：预览版

3星 · 超过75%的资源需积分: 9 82 浏览量更新于2024-07-20 收藏 1.98MB PDF 举报

"Kafka: The Definitive Guide 预览版" 本书《Kafka: The Definitive Guide》由Neha Narkhede、Gwen Shapira和Todd Palino共同撰写，旨在全面深入地介绍Apache Kafka这一分布式流处理平台。Kafka是一个强大的开源系统，它被设计用于构建实时数据管道和流应用，能够处理大量的实时数据。Kafka的核心特性包括高吞吐量、持久化、分区和复制，使其在大数据领域广泛应用。在Kafka中，消息以发布/订阅模式进行传输，允许生产者发布数据到主题（topics），消费者则订阅这些主题来消费数据。Kafka通过将数据存储在磁盘上，并使用高效的数据结构来实现高性能。此外，Kafka还支持多个消费者组，使得数据可以被多个消费者并行处理，增强了系统的处理能力。 Kafka集群由多个服务器（broker）组成，每个broker负责一部分topic的分区（partition）。分区是Kafka的可扩展性和容错性的关键，因为它允许数据在不同的服务器之间分布，同时提供了数据顺序保证和幂等性。如果一个broker故障，其上的分区可以自动转移到其他健康的broker，确保服务的连续性。本书涵盖了设置和管理Kafka集群的基础知识，包括安装、配置和监控。读者将学习如何创建和管理topics，配置生产者和消费者，以及如何使用Kafka Connect与其他系统集成。书中还详细介绍了Kafka的高级特性，如Kafka Streams，这是一个轻量级的库，用于在Kafka topic之间进行流处理，以及Kafka Connect，用于简化与外部系统的数据导入和导出。此外，书中还会讨论Kafka的扩展性策略，如增加更多的broker、调整分区数量以及使用Zookeeper（Kafka的协调组件）进行集群管理。在安全性方面，会涉及如何实现Kafka的认证和授权，保护数据的隐私和完整性。《Kafka: The Definitive Guide》不仅适合初学者了解Kafka的基本概念和用法，也对有经验的开发者提供了深入的技术细节和最佳实践。通过阅读本书，读者可以掌握如何在实际项目中有效地利用Kafka处理实时数据流，构建可靠的、高性能的数据处理系统。

Individual Queue Systems

At the same time that you have been waging this war with metrics, one of your cow‐

orkers has been doing similar work with log messages. Another has been working on

tracking user behavior on the front-end website and providing that information to

developers who are working on machine learning, as well as creating some reports for

management. You have all followed a similar path of building out systems that decou‐

ple the publishers of the information from the subscribers to that information. Figure

1-4 shows such an infrastructure, with three separate pub/sub systems.

Figure 1-4. Multiple publish/subscribe systems

This is certainly a lot better than utilizing point to point connections (as in Figure

1-2), but there is a lot of duplication. Your company is maintaining multiple systems

for queuing data, all of which have their own individual bugs and limitations. You

also know that there will be more use cases for messaging coming soon. What you

would like to have is a single centralized system that allows for publishing of generic

types of data, and that will grow as your business grows.

Enter Kafka

Apache Kafka is a publish/subscribe messaging system designed to solve this prob‐

lem. It is often described as a “distributed commit log”. A filesystem or database com‐

mit log is designed to provide a durable record of all transactions so that they can be

replayed to consistently build the state of a system. Similarly, data within Kafka is

stored durably, in order, and can be read deterministically. In addition, the data can

14 | Chapter 1: Meet Kafka

www.it-ebooks.info

be distributed within the system to provide additional protections against failures, as

well as significant opportunities for scaling performance.

Messages and Batches

The unit of data within Kafka is called a message. If you are approaching Kafka from a

database background, you can think of this as similar to a row or a record. A message

is simply an array of bytes, as far as Kafka is concerned, so the data contained within

it does not have a specific format or meaning to Kafka. Messages can have an optional

bit of metadata which is referred to as a key. The key is also a byte array, and as with

the message, has no specific meaning to Kafka. Keys are used when messages are to

be written to partitions in a more controlled manner. The simplest such scheme is to

treat partitions as a hash ring, and assure that messages with the same key are always

written to the same partition. Usage of keys is discussed more thoroughly in Chap‐

ter 3.

For efficiency, messages are written into Kafka in batches. A batch is just a collection

of messages, all of which are being produced to the same topic and partition. An indi‐

vidual round trip across the network for each message would result in excessive over‐

head, and collecting messages together into a batch reduces this. This, of course,

presents a tradeoff between latency and throughput: the larger the batches, the more

messages that can be handled per unit of time, but the longer it takes an individual

message to propagate. Batches are also typically compressed, which provides for more

efficient data transfer and storage at the cost of some processing power.

Schemas

While messages are opaque byte arrays to Kafka itself, it is recommended that addi‐

tional structure be imposed on the message content so that it can be easily under‐

stood. There are many options available for message schema, depending on your

application’s individual needs. Simplistic systems, such as Javascript Object Notation

(JSON) and Extensible Markup Language (XML), are easy to use and human reada‐

ble. However they lack features such as robust type handling and compatibility

between schema versions. Many Kafka developers favor the use of Apache Avro,

which is a serialization framework originally developed for Hadoop. Avro provides a

compact serialization format, schemas that are separate from the message payloads

and that do not require generated code when they change, as well as strong data typ‐

ing and schema evolution, with both backwards and forwards compatibility.

A consistent data format is important in Kafka, as it allows writing and reading mes‐

sages to be decoupled. When these tasks are tightly coupled, applications which sub‐

scribe to messages must be updated to handle the new data format, in parallel with

the old format. Only then can the applications that publish the messages be updated

to utilize the new format. New applications that wish to use data must be coupled

Enter Kafka | 15

www.it-ebooks.info

Producers and Consumers

Kafka clients are users of the system, and there are two basic types: producers and

consumers.

Producers create new messages. In other publish/subscribe systems, these may be

called publishers or writers. In general, a message will be produced to a specific topic.

By default, the producer does not care what partition a specific message is written to

and will balance messages over all partitions of a topic evenly. In some cases, the pro‐

ducer will direct messages to specific partitions. This is typically done using the mes‐

sage key and a partitioner that will generate a hash of the key and map it to a specific

partition. This assures that all messages produced with a given key will get written to

the same partition. The producer could also use a custom partitioner that follows

other business rules for mapping messages to partitions. Producers are covered in

more detail in Chapter 3.

Consumers read messages. In other publish/subscribe systems, these clients may be

called subscribers or readers. The consumer subscribes to one or more topics and

reads the messages in the order they were produced. The consumer keeps track of

which messages it has already consumed by keeping track of the

oset of messages.

The offset is another bit of metadata, an integer value that continually increases, that

Kafka adds to each message as it is produced. Each message within a given partition

has a unique offset. By storing the offset of the last consumed message for each parti‐

tion, either in Zookeeper or in Kafka itself, a consumer can stop and restart without

losing its place.

Consumers work as part of a consumer group. This is one or more consumers that

work together to consume a topic. The group assures that each partition is only con‐

sumed by one member. In Figure 1-6, there are three consumers in a single group

consuming a topic. Two of the consumers are working from one partition each, while

the third consumer is working from two partitions. The mapping of a consumer to a

partition is often called ownership of the partition by the consumer.

In this way, consumers can horizontally scale to consume topics with a large number

of messages. Additionally, if a single consumer fails, the remaining members of the

group will rebalance the partitions being consumed to take over for the missing

member. Consumers and consumer groups are discussed in more detail in Chapter 4.

Enter Kafka | 17

www.it-ebooks.info

剩余117页未读，继续阅读

ZYQ2006

粉丝: 1
资源: 4

Kafka权威指南：预览版

Kafka The Definitive Guide Real-Time Data and Stream Processing at Scale epub

Kafka_The Definitive Guide_Real-Time Data and Stream Processing at Scale

Kafka The Definitive Guide Real-Time Data and Stream Processing at Scale azw3

docker run -it --name kafka01 -p 19092:9092 -d -e KAFKA_BROKER_ID=0 -e KAFKA_ZOOKEEPER_CONNECT=192.168.233.129:12181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.233.129:19092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 wurstmeister/kafka:latest 根据这条命令帮我写个docker-compose

kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type topics --entity-name kafka_eagle

/opt/module/kafka/bin/kafka-topics.sh --zookeeper hadoop102:2181 --topic calllog --create --replication-factor 1 --partitions 3

最新资源