Flume监听文件目录sink至hdfs按照每天切割
采用的channels为file,sink为hdfs,此处往hdfs写的策略是当时间达到3600s或者文件大小达到128M。可以自己调整
agent1.sources = source1 agent1.sinks = sink1 agent1.channels = channel1 # Describe/configure spooldir source1 #agent1.sources.source1.type = spooldir #agent1.sources.source1.spoolDir = /var/log/apache/flumeSpool1 #agent1.sources.source1.fileHeader = true # Describe/configure tail -F source1 agent1.sources.source1.type = exec agent1.sources.source1.command = tail -n 0 -F /tmp/log.log agent1.sources.source1.channels = channel1 # Describe/configure nc source1 #agent1.sources.source1.type = netcat #agent1.sources.source1.bind = localhost #agent1.sources.source1.port = 44444 #configure host for source agent1.sources.source1.interceptors = i1 i2 agent1.sources.source1.interceptors.i1.type = host agent1.sources.source1.interceptors.i1.hostHeader = hostname agent1.sources.source1.interceptors.i2.type = timestamp # Describe sink1 #agent1.sinks.sink1.type = logger agent1.sinks.sink1.type = hdfs #a1.sinks.k1.channel = c1 #agent1.sinks.sink1.hdfs.path =hdfs://xxx:9000/tmp/tail/%Y-%m-%d/%H%M%S agent1.sinks.sink1.hdfs.path =hdfs://xxx:9000/tmp/tail/%Y-%m-%d/%H agent1.sinks.sink1.hdfs.filePrefix = %{hostname}/events- agent1.sinks.sink1.hdfs.inUsePrefix = . agent1.sinks.sink1.hdfs.maxOpenFiles = 5000 agent1.sinks.sink1.hdfs.batchSize= 1000 agent1.sinks.sink1.hdfs.fileType = DataStream agent1.sinks.sink1.hdfs.writeFormat =Text agent1.sinks.sink1.hdfs.rollSize = 128000000 agent1.sinks.sink1.hdfs.rollCount = 0 agent1.sinks.sink1.hdfs.rollInterval = 3600 #agent1.sinks.sink1.hdfs.round = true #agent1.sinks.sink1.hdfs.roundValue = 10 #agent1.sinks.sink1.hdfs.roundUnit = minute agent1.sinks.sink1.hdfs.useLocalTimeStamp = true # Use a channel which buffers events in memory agent1.channels.channel1.type = memory agent1.channels.channel1.keep-alive = 120 agent1.channels.channel1.capacity = 500000 agent1.channels.channel1.transactionCapacity = 600 # Bind the source and sink to the channel agent1.sources.source1.channels = channel1 agent1.sinks.sink1.channel = channel1
生产实例:
# The configuration file needs to define the sources, # the channels and the sinks. # Sources, channels and sinks are defined per agent, # in this case called 'a #agent section producer.sources = s producer.channels = c producer.sinks = r #producer.sources.s.type = seq producer.sources.s.channels = c producer.sources.s.type = exec producer.sources.s.command=tail -n 0 -F /usr/local/nginx/nginxlog/access.log producer.sources.s.deletePolicy=never #producer.sources.s.type = avro #producer.sources.s.bind = localhost #producer.sources.s.port = 10000 # Each sink's type must be defined(给谁了) #producer.sinks.r.type = avro #producer.sinks.r.hostname = 10.1.1.100 #producer.sinks.r.port = 20000 producer.sources.source1.interceptors = i1 producer.sources.source1.interceptors.i1.type = timestamp producer.sinks.r.type = hdfs producer.sinks.r.hdfs.path = hdfs://localhost:8010/user/hive/warehouse/tail/%Y-%m-%d producer.sinks.r.hdfs.inUsePrefix = . producer.sinks.r.hdfs.maxOpenFiles = 5000 producer.sinks.r.hdfs.batchSize= 1000 producer.sinks.r.hdfs.fileType = DataStream producer.sinks.r.hdfs.writeFormat =Text producer.sinks.r.hdfs.rollSize = 128000000 producer.sinks.r.hdfs.rollCount = 0 producer.sinks.r.hdfs.rollInterval = 3600 producer.sinks.r.hdfs.useLocalTimeStamp = true producer.sinks.r.request.required.acks=1 producer.sinks.r.max.message.size=1000000 producer.sinks.r.producer.type=sync producer.sinks.r.custom.encoding=UTF-8 #Specify the channel the sink should use producer.sinks.r.channel = c # Each channel's type is defined. producer.channels.c.type = memory producer.channels.c.capacity = 1000000 producer.channels.c.transactionCapacity = 1000 #producer.channels.c.type=file #producer.channels.c.checkpointDir=/usr/local/flumeng/checkpointdir/tcpdir/example_agent #producer.channels.c.dataDirs=/usr/local/flumeng/datadirs/tddirs/example_agen
实例启动脚本:
#发送到hdfs #./flume-ng agent -c /usr/local/flumeng/apache-flume-1.5.2-bin/conf/ -f /usr/local/flumeng/apache-flume-1.5.2-bin/conf/flume-hdfs.properties -n producer -Dflume.root.logger=INFO,console
相关推荐
zzjmay 2020-06-07
chenguangchun 2020-04-18
WeiHHH 2020-09-23
憧憬 2020-08-21
ViMan0 2020-08-14
tomli 2020-07-26
eternityzzy 2020-07-19
飞鸿踏雪0 2020-07-09
zzjmay 2020-07-04
swazerz 2020-06-22
ViMan0 2020-06-21
sujins 2020-06-14
sujins 2020-06-14
Cloudeep 2020-06-14
ViMan0 2020-06-14
sujins 2020-06-13