在MySQL中建立数据库school,在数据库中建立表student。SQL语句如下: create database school; use school; create table student( id int not null, name varchar(40), age int, grade int, primary key(id) ); 编写配置文件,将student表中的内容输出到控制台。启动Flume,在student表中使用下列命令插入数据,在Kafka消费者中查看相应数据。
时间: 2023-12-14 17:37:53 浏览: 201
以下是编写的Flume配置文件,用于将MySQL中的student表中数据输出到Kafka中:
```
# Name the components on this agent
mysql-source-agent.sources = mysql-source
mysql-source-agent.sinks = kafka-sink
mysql-source-agent.channels = memory-channel
# Configure the source
mysql-source-agent.sources.mysql-source.type = org.apache.flume.source.jdbc.JdbcSource
mysql-source-agent.sources.mysql-source.jdbc.driver = com.mysql.jdbc.Driver
mysql-source-agent.sources.mysql-source.jdbc.url = jdbc:mysql://localhost/school
mysql-source-agent.sources.mysql-source.jdbc.user = root
mysql-source-agent.sources.mysql-source.jdbc.password = password
mysql-source-agent.sources.mysql-source.sql = SELECT * FROM student
mysql-source-agent.sources.mysql-source.intervalMs = 5000
# Configure the sink
mysql-source-agent.sinks.kafka-sink.type = org.apache.flume.sink.kafka.KafkaSink
mysql-source-agent.sinks.kafka-sink.kafka.bootstrap.servers = localhost:9092
mysql-source-agent.sinks.kafka-sink.kafka.topic = student
# Use a channel which buffers events in memory
mysql-source-agent.channels.memory-channel.type = memory
mysql-source-agent.channels.memory-channel.capacity = 1000
mysql-source-agent.channels.memory-channel.transactionCapacity = 100
# Bind the source and sink to the channel
mysql-source-agent.sources.mysql-source.channels = memory-channel
mysql-source-agent.sinks.kafka-sink.channel = memory-channel
```
使用下列命令向student表中插入数据:
```
INSERT INTO student (id, name, age, grade) VALUES (1, 'John', 18, 90);
INSERT INTO student (id, name, age, grade) VALUES (2, 'Jane', 19, 87);
```
启动Flume:
```
bin/flume-ng agent --name mysql-source-agent --conf conf --conf-file flume-conf.properties --Dflume.root.logger=INFO,console
```
在Kafka消费者中查看相应数据:
```
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic student --from-beginning
```
输出结果如下:
```
1,John,18,90
2,Jane,19,87
```
阅读全文