在MySQL中建立数据库school,在数据库中建立表student。SQL语句如下: create database school; use school; create table student( id int not null, name varchar(40), age int, grade int, primary key(id) ); 编写配置文件,将student表中的内容输出到控制台。启动Flume,在student表中使用下列命令插入数据,在Kafka消费者中查看相应数据。
时间: 2023-12-14 18:38:45 浏览: 126
插入数据的SQL语句为:
```
INSERT INTO student (id, name, age, grade) VALUES (1, 'Tom', 18, 90);
INSERT INTO student (id, name, age, grade) VALUES (2, 'Jerry', 19, 80);
INSERT INTO student (id, name, age, grade) VALUES (3, 'Alice', 20, 85);
```
Flume配置文件如下:
```
# flume.conf
agent.sources = mysql-source
agent.channels = kafka-channel
agent.sinks = kafka-sink
# Define the source
agent.sources.mysql-source.type = org.apache.flume.source.jdbc.JdbcSource
agent.sources.mysql-source.jdbc.driver = com.mysql.jdbc.Driver
agent.sources.mysql-source.jdbc.url = jdbc:mysql://localhost:3306/school
agent.sources.mysql-source.jdbc.user = root
agent.sources.mysql-source.jdbc.password = password
agent.sources.mysql-source.jdbc.fetchSize = 10
agent.sources.mysql-source.sql = SELECT * FROM student
agent.sources.mysql-source.incremental.column.name = id
agent.sources.mysql-source.incremental.value = 0
# Define the channel
agent.channels.kafka-channel.type = memory
agent.channels.kafka-channel.capacity = 1000
agent.channels.kafka-channel.transactionCapacity = 100
# Define the sink
agent.sinks.kafka-sink.type = org.apache.flume.sink.kafka.KafkaSink
agent.sinks.kafka-sink.kafka.bootstrap.servers = localhost:9092
agent.sinks.kafka-sink.kafka.topic = student
agent.sinks.kafka-sink.batchSize = 100
agent.sinks.kafka-sink.requiredAcks = 1
# Bind the source and sink to the channel
agent.sources.mysql-source.channels = kafka-channel
agent.sinks.kafka-sink.channel = kafka-channel
```
启动Flume命令如下:
```
bin/flume-ng agent --conf conf --conf-file flume.conf --name agent -Dflume.root.logger=INFO,console
```
在Kafka消费者中查看数据:
```
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic student --from-beginning
```
输出结果如下:
```
{"id":1,"name":"Tom","age":18,"grade":90}
{"id":2,"name":"Jerry","age":19,"grade":80}
{"id":3,"name":"Alice","age":20,"grade":85}
```
阅读全文