sparkStreaming整合flume

sparkStreaming整合flume

  • 需要把spark-streaming-flume-sink_2.11-2.1.3.jar放到flume的lib目录下
  • 使用scala-library-2.11.8.jar替换之前的scala-library-2.10依赖的jar包

9.1 poll拉模式整合

  • 1、flume配置文件

    a1.sources = r1
          a1.sinks = k1
          a1.channels = c1
          
          #source
          a1.sources.r1.channels = c1
          a1.sources.r1.type = spooldir
          a1.sources.r1.spoolDir = /root/data
          a1.sources.r1.fileHeader = true
          
          #channel
          a1.channels.c1.type =memory
          a1.channels.c1.capacity = 20000
          a1.channels.c1.transactionCapacity=5000
          
          #sinks
          a1.sinks.k1.channel = c1
          a1.sinks.k1.type = org.apache.spark.streaming.flume.sink.SparkSink
          a1.sinks.k1.hostname=yh
          a1.sinks.k1.port = 8888
下载地址
 - http://spark.apache.org/docs/latest/streaming-flume-integration.html
 - 下载位置
![图片描述][1]

相关推荐