在flume/conf目录下创建flume-hdfs.conf文件,并对其进行基础测试配置
时间: 2023-12-17 22:02:29 浏览: 199
flume-flume-hdfs.conf
好的,您可以按照以下步骤进行基础测试配置:
1. 打开终端并进入flume/conf目录
2. 创建flume-hdfs.conf文件:`touch flume-hdfs.conf`
3. 编辑flume-hdfs.conf文件,添加以下内容:
```
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444
# Describe the sink
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = /user/flume/%{topic}/%y-%m-%d
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.writeFormat = Text
a1.sinks.k1.hdfs.useLocalTimeStamp = true
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
```
4. 启动netcat服务,监听端口44444:`nc -lk 44444`
5. 启动flume agent,指定配置文件:`bin/flume-ng agent --conf conf --conf-file conf/flume-hdfs.conf --name a1 -Dflume.root.logger=INFO,console`
现在,您可以在netcat终端中输入一些文本,并查看它是否被写入HDFS中的文件。文件路径应该类似于/user/flume/netcat/年-月-日/。
阅读全文