Apache Kafka指南：Cloudera版

需积分: 1 117 浏览量更新于2024-07-09 收藏 2.97MB PDF 举报

"Apache Kafka 指南" Apache Kafka 是一个高度可扩展且分布式的消息队列系统，由 Apache Software Foundation 开发并维护。它最初由 LinkedIn 设计，现在被广泛应用于大数据处理、实时流数据平台以及微服务之间的通信等多个领域。Cloudera 提供的这个文档可能是针对使用 Cloudera 管理的 Kafka 集群的操作和管理指南。文档中的重要通知强调了 Cloudera 对其知识产权的保护，包括商标、服务标识等，未经许可不得复制或使用。同时，文档中包含的代码遵循 Apache License Version 2.0 的开源协议，允许用户自由使用和修改，但需遵守相应的条款和条件。 Kafka 的核心特性包括以下几个方面： 1. **消息中间件**：Kafka 作为消息队列，允许生产者发布消息到主题（topics），消费者则可以订阅并消费这些消息。这种模式支持离线和实时的数据处理。 2. **高吞吐量**：Kafka 被设计为处理大规模的实时数据流，能够以极高的速度处理大量消息。 3. **持久化与复制**：Kafka 将消息持久化到磁盘，并通过副本机制确保数据的容错性。即使部分节点失败，系统仍能继续运行。 4. **分区与并行处理**：消息被分到不同的分区中，每个分区可以被多个消费者并行消费，提高了处理效率。 5. **消费者组**：Kafka 支持消费者组的概念，同一组内的消费者会均衡地分配分区，实现负载均衡和容错。 6. **灵活性**：Kafka 可以与其他大数据组件如 Hadoop、Spark 等集成，提供灵活的数据处理方案。 7. **API 支持**：Kafka 提供了多种编程语言的客户端库，方便开发者在各种环境中使用。 8. **集群管理**：Cloudera 提供的管理工具使得 Kafka 集群的管理和监控变得更加方便，包括配置、监控、故障恢复等。 9. **安全性**：Kafka 可以通过 SSL 和 SASL 进行安全认证和加密，确保数据传输的安全。 10. **扩展性**：随着数据量的增长，Kafka 集群可以通过添加更多节点进行水平扩展。在实际应用中，Apache Kafka 常用于日志收集、用户行为追踪、流式处理、事件驱动架构等多种场景。Cloudera 的文档可能详细介绍了如何在 Cloudera 环境下部署、配置、监控和管理 Kafka，包括设置主题、创建消费者、管理集群性能优化等方面的内容。此外，文档可能还会涵盖故障排查、最佳实践以及与其他 Cloudera 产品（如 Hadoop）的集成方法。对于使用 Cloudera 管理的 Kafka 系统的管理员和开发人员来说，这份指南是非常宝贵的参考资料。

Kafka in Cloudera Manager

Cloudera Manager is the recommended way to administer your cluster, including administering Kafka. When you open

the Kafka service from Cloudera Manager, you see the following dashboard:

Figure 8: Kafka Service in Cloudera Manager

16 | Apache Kafka Guide

Kafka in Cloudera Manager

If you set chroot and then use only the host and port in the connect string, you'll see the following exception:

InvalidReplicationFactorException: replication factor: 3 larger than available brokers:

Kafka Producers

Kafka producers are the publishers responsible for writing records to topics. Typically, this means writing a program

using the KafkaProducer API. To instantiate a producer:

KafkaProducer<String, String> producer = new

KafkaProducer<>(producerConfig);

Most of the important producer settings, and mentioned below, are in the configuration passed by this constructor.

Serialization of Keys and Values

For each producer, there are two serialization properties that must be set, key.serializer (for the key) and

value.serializer (for the value). You can write custom code for serialization or use one of the ones already provided

by Kafka. Some of the more commonly used ones are:

• ByteArraySerializer: Binary data

• StringSerializer: String representations

Managing Record Throughput

There are several settings to control how many records a producer accumulates before actually sending the data to

the cluster. This tuning is highly dependent on the data source. Some possibilities include:

• batch.size: Combine this fixed number of records before sending data to the cluster.

• linger.ms: Always wait at least this amount of time before sending data to the cluster; then send however many

records has accumulated in that time.

• max.request.size: Put an absolute limit on data size sent. This technique prevents network congestion caused

by a single transfer request containing a large amount of data relative to the network speed.

• compression.type: Enable compression of data being sent.

• retries: Enable the client for retries based on transient network errors. Used for reliability.

Acknowledgments

The full write path for records from a producer is to the leader partition and then to all of the follower replicas. The

producer can control which point in the path triggers an acknowledgment. Depending on the acks setting, the producer

may wait for the write to propagate all the way through the system or only wait for the earliest success point.

Valid acks values are:

• 0: Do not wait for any acknowledgment from the partition (fastest throughput).

• 1: Wait only for the leader partition response.

• all: Wait for follower partitions responses to meet minimum (slowest throughput).

Partitioning

In Kafka, the partitioner determines how records map to partitions. Use the mapping to ensure the order of records

within a partition and manage the balance of messages across partitions. The default partitioner uses the entire key

to determine which partition a message corresponds to. Records with the same key are always mapped to the same

partition (assuming the number of partitions does not change for a topic). Consider writing a custom partitioner if you

have information about how your records are distributed that can produce more efficient load balancing across

partitions. A custom partitioner lets you take advantage of the other data in the record to control partitioning.

18 | Apache Kafka Guide

Kafka Clients

If a partitioner is not provided to the KafkaProducer, Kafka uses a default partitioner.

The ProducerRecord class is the actual object processed by the KafkaProducer. It takes the following parameters:

• Kafka Record: The key and value to be stored.

• Intended Destination: The destination topic and the specific partition (optional).

Kafka Consumers

Kafka consumers are the subscribers responsible for reading records from one or more topics and one or more partitions

of a topic. Consumers subscribing to a topic can happen manually or automatically; typically, this means writing a

program using the KafkaConsumer API.

To instantiate a consumer:

KafkaConsumer<String, String> kafkaConsumer = new

KafkaConsumer<>(consumerConfig);

The KafkaConsumer class has two generic type parameters. Just as producers can send data (the values) with keys,

the consumer can read data by keys. In this example both the keys and values are strings. If you define different types,

you need to define a deserializer to accommodate the alternate types. For deserializers you need to implement the

org.apache.kafka.common.serialization.Deserializer interface.

The most important configuration parameters that we need to specify are:

• bootstrap.servers: A list of brokers to initially connect to. List 2 to 3 brokers; you don't needed to list the full

cluster.

• group.id: Every consumer belongs to a group. That way they’ll share the partitions of a topic.

• key.deserializer/value.deserializer: Specify how to convert the Java representation to a sequence of

bytes to send data through the Kafka protocol.

Subscribing to a topic

Subscribing to a topic using the subscribe() method call:

kafkaConsumer.subscribe(Collections.singletonList(topic), rebalanceListener);

Here we specify a list of topics that we want to consume from and a 'rebalance listener.' Rebalancing is an important

part of the consumer's life. Whenever the cluster or the consumers’ state changes, a rebalance will be issued. This will

ensure that all the partitions are assigned to a consumer.

After subscribing to a topic, the consumer polls to see if there are new records:

while (true) {

data = kafkaConsumer.poll();

// do something with 'data'

}

The poll returns multiple records that can be processed by the client. After processing the records the client commits

offsets synchronously, thus waiting until processing completes before continuing to poll.

The last important point is to save the progress. This can be done by the commitSync() and commitAsync() methods

respectively.

PLACEHOLDER FOR CODE SNIPPET

Auto commit is not recommended; manual commit is appropriate in the majority of use cases.

Apache Kafka Guide | 19

Kafka Clients

Groups and Fetching

Kafka consumers are usually assigned to a group. This happens statically by setting the group.id configuration property

in the consumer configuration. Consuming with groups will result in the consumers balancing the load in the group.

That means each consumer will have their fair share of partitions. Also it can never be more consumers than partitions

as that way there would be idling consumers.

As shown in the figure below, both consumer groups share the partitions and each partition multicasts messages to

both consumer groups. The consumers pull messages from the broker instead of the broker periodically pushing what

is available. This helps the consumer as it won’t be overloaded and it can query the broker at its own speed. Furthermore,

to avoid tight looping, it uses a so called “long-poll”. The consumer sends a fetch request to poll for data and receives

a reply only when enough data accumulates on the broker.

Figure 9: Consumer Groups and Fetching from Partitions

Protocol between Consumer and Broker

This section details how the protocol works, what messages are going on the wire and how that contributes to the

overall behavior of the consumer. When discussing the internals of the consumers, there are a couple of basic terms

to know:

Heartbeat

When the consumer is alive and is part of the consumer group, it sends heartbeats. These are short periodic messages

that tell the brokers that the consumer is alive and everything is fine.

Session

Often one missing heartbeat is not a big deal, but how do you know if a consumer is not sending heartbeats for

long enough to indicate a problem? A session is such a time interval. If the consumer didn’t send any heartbeats

for longer than the session, the broker can consider the consumer dead and remove it from the group.

Coordinator

The special broker which manages the group on the broker side is called the coordinator. The coordinator handles

heartbeats and assigns the leader. Every group has a coordinator that organizes the startup of a consumer group

and assist whenever a consumer leaves the group.

Leader

The leader consumer is elected by the coordinator. Its job is to assign partitions to every consumer in the group at

startup or whenever a consumer leaves or joins the group. The leader holds the assignment strategy, it is decoupled

from the broker. That means consumers can reconfigure the partition assignment strategy without restarting the

brokers.

20 | Apache Kafka Guide

Kafka Clients

剩余174页未读，继续阅读

connectforgot

粉丝: 0
资源: 4

Apache Kafka指南：Cloudera版

cloudera 对kafka 安全、配置、优化和管理的建议 pdf

Introduction to Apache Kafka.pdf

Kyligence_Enterprise_3_4-zh.pdf

藏经阁-领军行业大数据及AI实战-93.pdf

ClouderaManager-中文手册-实战总结.pdf

Cloudera-数据云-一站式DataOps自助开发.pdf

大数据题库-选择题.pdf

基于Spark的电商用户分析系统-开题报告.pdf

开源日志系统比较：scribe、chukwa、kafka、flume.pdf

藏经阁-Building realtime BI Systems with Kafka, Spark and Kudu.pdf

最新资源