Flume + zookeeper + kafka收集Nginx日志

相关环境

用户统一为hadoop

前置操作

各主机间启动ssh连接

这一步至关重要,如果没有配置成功，会影响到hadoop,kafka集群之间的连接

[hadoop@c1 ~]$ ssh-keygen
[hadoop@c1 ~]$ sudo vim /etc/ssh/sshd_config
    ...
    PubkeyAuthentication yes
    ...
[hadoop@c1 ~]$ systemctl restart sshd
[hadoop@c1 ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys  # ssh 本机
[hadoop@c1 ~]$ sudo vim /etc/hosts  # 添加三台主机对应ip host
    ...
    192.168.1.200 c1
    192.168.1.201 c1_1
    192.168.1.202 c1_2
    ...
[hadoop@c1 ~]$ ssh-copy-id -i  ~/.ssh/id_rsa.pub hadoop@c1_1
[hadoop@c1 ~]$ ssh-copy-id -i  ~/.ssh/id_rsa.pub hadoop@c1_2

其他两台机器重复上面的操作即可.完成后，可以ssh一下各台机子(包括本机)是否还需要密码

安装软件

# 下载jdk1.8+
[hadoop@c1 ~] mkdir -p ~/app/jdk1.8 && tar -zxvf jdk-8u171-linux-x64.tar.gz -C ~/app/jdk1.8
# 下载flume1.6
[hadoop@c1 ~] tar -zxvf apache-flume-1.6.0-bin.tar.gz -C ~/app/
# 下载zookeeper3.4.10
[hadoop@c1 ~] tar -zxvf zookeeper-3.4.10.tar.gz -C ~/app/
# 下载kafka_2.11-1.1.0
[hadoop@c1 ~] tar -xzf kafka_2.11-1.1.0.tgz -C ~/app/
# 环境变量
[hadoop@c1 ~] vim .bash_profile
    ...
    export JAVA_HOME=/home/hadoop/app/jdk1.8.0
    export PATH=$JAVA_HOME/bin:$PATH
    export FLUME_HOME=/home/hadoop/app/flume-1.6.0
    export PATH=$FLUME_HOME/bin:$PATH
    export ZK_HOME=/home/hadoop/app/zookeeper-3.4.10
    export PATH=$ZK_HOME/bin:$PATH
    export KAFKA_HOME=/home/hadoop/app/kafka_2.11-1.1.0
    export PATH=$KAFKA_HOME/bin:$PATH
    ...
[hadoop@c1 ~] source .bash_profile
# 复制软件和环境变量到其他主机
[hadoop@c1 ~]scp -r ~/app hadoop@c1_1:~
[hadoop@c1 ~]scp -r ~/app hadoop@c1_2:~
[hadoop@c1 ~]scp .bash_profile hadoop@c1_1:~
[hadoop@c1 ~]scp .bash_profile hadoop@c1_2:~
# 到其他主机执行source .bash_profile

配置文件

• flume配置文件

  # vim ${FLUME_HOME}/conf/nginx_kafka.conf
  nginx-kafka.sources = r1                                                                                     
  nginx-kafka.sinks = k1
  nginx-kafka.channels = c1
  
  nginx-kafka.sources.r1.type = exec
  nginx-kafka.sources.r1.command = tail -f /home/hadoop/data/access.log
  nginx-kafka.sources.r1.shell = /bin/sh -c
  # flume1.6 kafka sink 写法
  nginx-kafka.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
  nginx-kafka.sinks.k1.brokerList = c1:9092
  nginx-kafka.sinks.k1.topic= nginxtopic
  nginx-kafka.sinks.k1.batchSize=10
  
  nginx-kafka.channels.c1.type = memory
  nginx-kafka.sources.r1.channels = c1
  nginx-kafka.sinks.k1.channel = c1

• zookeeper配置文件

  # cp ${ZK_HOME}/conf/zoo_simple.cfg ${ZK_HOME}/conf/zoo.cfg && vim ${ZK_HOME}/conf/zoo.cfg
  tickTime=2000
  initLimit=10
  syncLimit=5
  dataDir=/home/hadoop/data/zookeeper
  clientPort=2181
  # 注意，当前主机的zookeeper server不能设置hostname,必须是0.0.0.0 否则无法连接
  server.1=0.0.0.0:2888:3888                                                   
  server.2=c1_1:2888:3888
  server.3=c1_2:2888:3888

  创建zookeeper集群id

  echo "1">/home/hadoop/data/zookeeper/myid

  其他主机重复相同操作,server.x需要和myid值一致,

• kafka配置文件

  kafka配置文件需要改动的只有几个

  # ${KAFKA_HOME}/config/server.properties
  broker.id=0
  host.name=c1
  listeners=PLAINTEXT://192.168.1.200:9092
  advertised.listeners=PLAINTEXT://c1:9092
  zookeeper.connect=c1:2181,c1_1:2181,c1_2:2181

  broker.id从0开始且在集群中唯一

  listeners需要填上IP
  advertised.listeners需要填上hostname

  这里我这么设置是没问题的，但是不清楚为啥这么设置

  其他主机kafka配置文件同样的操作

编写集群启动脚本

• zookeeper集群脚本

  # vim start_zookeeper.sh
  #!/bin/bash
  echo "start zkServer..."
  for i in c1 c1_1 c1_2
      do
          ssh hadoop@$i "source ~/.bash_profile;zkServer.sh start"
      done

  # vim stop_zookeeper.sh
  #!/bin/bash
  echo "stop zkServer..."
  for i in c1 c1_1 c1_2
      do
          ssh hadoop@$i "source ~/.bash_profile;zkServer.sh stop"             
      done

  chmod a+x start_zookeeper.sh stop_zookeeper.sh

• kafka集群脚本

  # vim start_kafka.sh
  #!/bin/sh
  echo "start kafka..."
  for i in c1 c1_1 c1_2
  do
      ssh hadoop@$i "source ~/.bash_profile;kafka-server-start.sh -daemon ${KAFKA_HOME}/config/server.properties &"
      echo "done"                                                             
  done

  # vim stop_kafka.sh
  #!/bin/sh
  echo "stop kafka..."
  for i in c1 c1_1 c1_2
  do
      ssh hadoop@$i "source ~/.bash_profile;kafka-server-stop.sh"            
  done

  chmod a+x start_kafka.sh stop_kafka.sh

实战

1. 启动程序

   # 启动zookeeper
   [hadoop@c1 ~]$ ./start_zookeeper.sh
   [hadoop@c1 ~]$ zkServer.sh status
   ZooKeeper JMX enabled by default
   Using config: /home/hadoop/zookeeper-3.4.10/bin/../conf/zoo.cfg
   Mode: follower
   # 启动kafka
   [hadoop@c1 ~]$ ./start_kafka.sh
   [hadoop@c1 ~]$ jps
   2953 QuorumPeerMain  # zookeeper 进程
   3291 Kafka  #kafka进程
   3359 Jps

2. 创建Topic

   [hadoop@c1 ~]$ kafka-topics.sh --create --zookeeper c1:2181,c1_1:2181,c1_2:2181 --replication-factor 3 --partitions 1 --topic nginxtopic

3. 检查Topic

   [hadoop@c1 ~]$ kafka-topics.sh --zookeeper c1:2181,c1_1:2181,c1_2:2181 --list
   nginx

4. 启动消费者

   [hadoop@c1 ~]$ kafka-console-consumer.sh --bootstrap-server c1:9092,c1_1:9092,c1_2:9092 -topic nginxtopic --from-beginning

5. 模拟日志

   #vim create_log.sh
   ---
   #!/bin/sh
   # access.log-xxx 等多个文件是生产环境拖下来的真是日志
   cat access.log-*| while read -r line
   do
   echo $line >> /home/hadoop/logs/access.log
   sleep 0.$(($RANDOM%5+1))  # 防止日志写入过快
   done

6. 启动flume

   新开一个窗口

   [hadoop@c1 ~]$ flume-ng agent --conf-file  conf/nginx_kafka.conf -c conf/ --name nginx-kafka -Dflume.root.logger=DEBUG,console

稍等片刻后

flume输出日志

kafka-console-consume 输出日志

至此项目已经完全跑起来了~

错误排查及解决

• not in the sudoers file. This incident will be reported
  没有sudo的操作权限，需要在root权限下编辑/etc/sudoer

  ...
  ## Allow root to run any commands anywhere 
  root    ALL=(ALL)   ALL
  hadoop  ALL=(ALL)   ALL
  ...

• 已经添加过ssh仍需要输入密码

  chmod 700 ~/.ssh
  chmod 644 ~/.ssh/authorized_keys

• zookeeper: It is probably not running

  1. 有可能是ssh无法免密访问其他主机
  2. 有可能是没有正确的写myid

可以在zookeeper.out 查看详细的错误信息