sparkStreaming整合flume
sparkStreaming整合flume
- 需要把spark-streaming-flume-sink_2.11-2.1.3.jar放到flume的lib目录下
- 使用scala-library-2.11.8.jar替换之前的scala-library-2.10依赖的jar包
9.1 poll拉模式整合
1、flume配置文件
a1.sources = r1 a1.sinks = k1 a1.channels = c1 #source a1.sources.r1.channels = c1 a1.sources.r1.type = spooldir a1.sources.r1.spoolDir = /root/data a1.sources.r1.fileHeader = true #channel a1.channels.c1.type =memory a1.channels.c1.capacity = 20000 a1.channels.c1.transactionCapacity=5000 #sinks a1.sinks.k1.channel = c1 a1.sinks.k1.type = org.apache.spark.streaming.flume.sink.SparkSink a1.sinks.k1.hostname=yh a1.sinks.k1.port = 8888
下载地址 - http://spark.apache.org/docs/latest/streaming-flume-integration.html - 下载位置 ![图片描述][1]
相关推荐
chenguangchun 2020-07-26
myt0 2020-07-18
IT影风 2020-07-18
chenguangchun 2020-06-28
jiaomrswang 2020-06-26
myt0 2020-06-16
xiaoxiaojavacsdn 2020-06-08
zzjmay 2020-06-07
strongyoung 2020-06-04
ErixHao 2020-05-20
啦啦啦啦啦 2020-05-15
wanfuchun 2020-05-14
onwaygoahead 2020-05-05
xiaoxiaojavacsdn 2020-05-01
chenguangchun 2020-04-18
QAnyang 2020-03-14
wsong 2020-03-13