在k8s上部署 zookeeper + kafka 集群 配置教程
时间: 2023-12-31 07:06:29 浏览: 251
bitnami版本的zookeeper和kafka的docker-compose配置
好的,以下是一个基本的在k8s上部署zookeeper + kafka集群的配置教程:
1. 创建一个namespace
首先,我们需要创建一个namespace,用于部署zookeeper和kafka集群。可以使用以下命令创建一个名为“zookeeper-kafka”的namespace:
```
kubectl create namespace zookeeper-kafka
```
2. 部署Zookeeper
接下来,我们需要部署Zookeeper。可以使用以下YAML文件创建一个Zookeeper服务:
```
apiVersion: v1
kind: Service
metadata:
name: zookeeper
spec:
selector:
app: zookeeper
ports:
- name: client
port: 2181
protocol: TCP
- name: follower
port: 2888
protocol: TCP
- name: leader
port: 3888
protocol: TCP
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: zookeeper
spec:
serviceName: zookeeper
replicas: 3
selector:
matchLabels:
app: zookeeper
template:
metadata:
labels:
app: zookeeper
spec:
containers:
- name: zookeeper
image: zookeeper:3.5.8
ports:
- name: client
containerPort: 2181
- name: follower
containerPort: 2888
- name: leader
containerPort: 3888
volumeMounts:
- name: datadir
mountPath: /data
env:
- name: ZOO_MY_ID
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: ZOO_SERVERS
value: zookeeper-0.zookeeper:2888:3888,zookeeper-1.zookeeper:2888:3888,zookeeper-2.zookeeper:2888:3888
volumeClaimTemplates:
- metadata:
name: datadir
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
```
这将创建一个3个Pod的Zookeeper StatefulSet,并创建一个名为“zookeeper”的Service,暴露Zookeeper的客户端端口2181,follower端口2888和leader端口3888。
3. 部署Kafka
现在,我们可以部署Kafka。以下是一个Kafka部署的YAML文件示例:
```
apiVersion: v1
kind: Service
metadata:
name: kafka
spec:
type: NodePort
selector:
app: kafka
ports:
- name: kafka
port: 9092
nodePort: 30092
protocol: TCP
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: kafka
spec:
serviceName: kafka
replicas: 3
selector:
matchLabels:
app: kafka
template:
metadata:
labels:
app: kafka
spec:
containers:
- name: kafka
image: wurstmeister/kafka:2.13-2.7.0
ports:
- name: kafka
containerPort: 9092
env:
- name: KAFKA_BROKER_ID
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: KAFKA_ZOOKEEPER_CONNECT
value: zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181
- name: KAFKA_ADVERTISED_LISTENERS
value: PLAINTEXT://$(hostname -f):9092
- name: KAFKA_LISTENERS
value: PLAINTEXT://0.0.0.0:9092
- name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
value: "3"
volumeMounts:
- name: datadir
mountPath: /data
volumeClaimTemplates:
- metadata:
name: datadir
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
```
这将创建一个3个Pod的Kafka StatefulSet和一个名为“kafka”的Service,它将Kafka的9092端口暴露为NodePort 30092。
4. 验证部署
现在,您可以使用以下命令检查Zookeeper和Kafka是否正在运行:
```
kubectl get pods -n zookeeper-kafka
```
您应该看到3个Zookeeper和3个Kafka Pod处于“Running”状态。
接下来,您可以使用以下命令检查Kafka是否正在监听端口30092(或您自己选择的端口):
```
kubectl get services -n zookeeper-kafka
```
您应该看到一个名为“kafka”的service,它将Kafka的9092端口暴露为30092端口。可以使用此端口测试Kafka是否正常运行。
至此,您已经成功地在k8s上部署了zookeeper + kafka集群。
阅读全文