您现在的位置是: 首页 > Kafka整合Flume实时数据采集


Kafka整合Flume实时数据采集

纯真年代 2018-10-20 21:37:13 0 314人围观

1)架构图


2)avro-memory-kafka.conf  

$FLUME_HOME/conf/avro-memory-kafka.conf

kafka sink

avro-memory-kafka.sources  =  avro-source 
avro-memory-kafka.sinks = kafka-sink
avro-memory-kafka.channels = memory-channel

avro-memory-kafka.sources.avro-source.type = avro
avro-memory-kafka.sources.avro-source.bind = hadoop000
avro-memory-kafka.sources.avro-source.port = 44444

avro-memory-kafka.sinks.kafka-sink.type = org.apache.flume.sink.kafka.KafkaSink
avro-memory-kafka.sinks.kafka-sink.kafka.bootstrap.servers = hadoop000:9092
avro-memory-kafka.sinks.kafka-sink.kafka.topic = hello_topic
avro-memory-kafka.sinks.kafka-sink.flumeBatchSize = 5

avro-memory-kafka.channels.memory-channel.type = memory

avro-memory-kafka.sources.avro-source.channels = memory-channel
avro-memory-kafka.sinks.kafka-sink.channel = memory-channel

3)启动zookeeper、单节点单broker的kafka直达车

4)依次启动avro-memory-kafka.conf、exec-memory-avro.conf  直达车

flume-ng agent \
--name avro-memory-kafka \
--conf $FLUME_HOME/conf \
--conf-file $FLUME_HOME/conf/avro-memory-kafka.conf \
-Dflume.root.logger=INFO,console


flume-ng agent \
--name exec-memory-avro \
--conf $FLUME_HOME/conf \
--conf-file $FLUME_HOME/conf/exec-memory-avro.conf \
-Dflume.root.logger=INFO,console

5)启动一个控制台消费者

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic hello_topic


6)我们监控的文件是data/flume-test.log

echo hello >> flume-test.log如图



说明:整理于网络 
本文章作为个人笔记,也供大家监督 

 

全部评论