ELK日志收集demo

liulin0

2019-12-27

架构目标

ELK日志收集demo

说明

系统： CentOS Linux release 7.5.1804

ELK版本： filebeat-6.8.5-x86_64.rpm, logstash-6.8.5.rpm, elasticsearch-6.8.5.rpm, kibana-6.8.5-x86_64.rpm kafka_2.11-2.0.0 zookeeper-3.4.12

地址	名称	功能, 按图左至右
192.168.9.133	test1.xiong.com	nginx + 虚拟主机 + filebeat
192.168.9.134	test2.xiong.com	nginx + 虚拟主机 + filebeat
192.168.9.135	test3.xiong.com	elasticsearch + kibana + logstash
192.168.9.136	test4.xiong.com	elasticsearch + kibana + logstash
192.168.9.137	test5.xiong.com	redis + logstash (这里使用kafka)
192.168.9.138	test6.xiong.com	redis + logstash (这里使用kafka)

实践并不需要这么多准备4台即可

1、配置

1.1、主机名

~]# cat /etc/hosts
192.168.9.133 test1.xiong.com
192.168.9.134 test2.xiong.com
192.168.9.135 test3.xiong.com
192.168.9.136 test4.xiong.com
192.168.9.137 test5.xiong.com
192.168.9.138 test6.xiong.com

# 关闭防火墙 以及selinux
systemctl stop firewalld  
sed -i ‘/SELINUX/s/enforcing/disabled/‘ /etc/selinux/config

~]# crontab -l   # 时间同步
    */1 * * * * /usr/sbin/ntpdate pool.ntp.org &>/dev/null 

# 安装jdk 135, 136, 137, 138需要安装
~]# tar xf jdk-8u181-linux-x64.tar.gz -C /usr/java/
    cd /usr/java/
    ln -sv jdk1.8.0_181/ default
    ln -sv default/ jdk

# 设置打开文件的个数
echo "* hard nofile 65536" >> /etc/security/limits.conf
echo "* soft nofile 65536" >> /etc/security/limits.conf

java]# cat /etc/profile.d/java.sh 
    export JAVA_HOME=/usr/java/jdk
    export PATH=$JAVA_HOME/bin:$PATH

java]# source /etc/profile.d/java.sh
java]# java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

1.2、服务端安装elk

这里配置主机 9.135、9.136

# 安装服务端的ELK
~]# rpm -vih elasticsearch-6.8.5.rpm kibana-6.8.5-x86_64.rpm logstash-6.8.5.rpm 

# 修改配置
~]# cd /etc/elasticsearch  
# 修改完之后同步，只需要修改Network.host\node.name
elasticsearch]# grep -v "^#" elasticsearch.yml
cluster.name: myElks           # 集群名称
node.name: test3.xiong.com     # 根据主机修改主机名
path.data: /opt/elasticsearch/data   # 数据目录
path.logs: /opt/elasticsearch/logs   # 日志目录
network.host: 0.0.0.0
network.publish_host: 192.168.9.136  # 监听地址
# 发现地址ping
discovery.zen.ping.unicast.hosts: ["192.168.9.135", "192.168.9.136"]
# 最小需要多少个节点  节点数计算 (N/2)+1
discovery.zen.minimum_master_nodes: 2
# 开启跨域访问支持
http.cors.enabled: true 
http.cors.allow-origin: "*"

# 修改数据目录以及日志 注意权限问题
elasticsearch]# mkdir /opt/elasticsearch/{data,logs} -pv
elasticsearch]# chown elasticsearch.elasticsearch /opt/elasticsearch/ -R

# 修改启动文件 
elasticsearch]# vim /usr/lib/systemd/system/elasticsearch.service
# 在[Service]下添加环境变量
    Environment=JAVA_HOME=/usr/java/jdk  # 指定java家目录
    LimitMEMELOCK=infinity  # 最大化使用内存

elasticsearch]# vim jvm.options  # 修改启动的jvm内存，这里应该为内存的一半或不大于30G
    -Xms2g
    -Xmx2g

# 启动服务   需要注意的是两台主机都需要同样配置， 可以使用ansible之类的工具
systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl restart elasticsearch

# 检查服务端口是否监听成功，  或查看  systemctl status elasticsearch
elasticsearch]# ss -tnl | grep 92
LISTEN     0      128       ::ffff:192.168.9.136:9200                    :::*
LISTEN     0      128       ::ffff:192.168.9.136:9300                    :::*

# 查看主机是否加入集群
elasticsearch]# curl 192.168.9.135:9200/_cat/nodes
192.168.9.136 7 95  1 0.00 0.06 0.11 mdi * test4.xiong.com
192.168.9.135 7 97 20 0.45 0.14 0.09 mdi - test3.xiong.com

# 查看master
elasticsearch]# curl 192.168.9.135:9200/_cat/master
fVkp7Ld3RDGmWlGpm6t7kg 192.168.9.136 192.168.9.136 test4.xiong.com

1.2.1、安装插件head

# 两台主机 9.135 9.136 安装
1、安装nmp
    ]# yum -y install epel-release  # 需要先安装epel源
    ]# yum -y install npm
2、安装elasticsearch-head插件
    ]# cd /usr/local/src/
    ]# git clone git://github.com/mobz/elasticsearch-head.git
    ]# cd /usr/local/src/elasticsearch-head/
    elasticsearch-head ]# npm install grunt -save  # 生成执行文件
    elasticsearch-head]# ll node_modules/grunt  # 确定文件是否产生
    elasticsearch-head ]# npm install

3、启动head
node_modules]# nohup npm run start &
    ss -tnl | grep 9100  # 查看端口是否存在，存在之后直接访问web

9.135:9100 与9.136:9100 可以只配一台

ELK日志收集demo

1.2.2、配置kibana

kibana]# grep -v "^#" kibana.yml  | grep -v "^$"
server.port: 5601
server.host: "0.0.0.0"
server.name: "test3.xiong.com"  # 另一台只需要修改主机名
elasticsearch.hosts: ["http://192.168.9.135:9200", "http://192.168.9.135:9200"]

kibana]# systemctl restart kibana
kibana]# ss -tnl | grep 5601   # 检查端口是否监听
LISTEN     0      128          *:5601                     *:*

ELK日志收集demo

1.2.3、配置logstash

logstash]# vim /etc/default/logstash 
    JAVA_HOME="/usr/java/jdk"   # 增加java环境变量

1.3、nginx+filebeat

主机： 192.168.9.133， 9.134

1.3.1、安装

~]# cat /etc/yum.repos.d/nginx.repo   # 配置nginx yum源
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true

[nginx-mainline]
name=nginx mainline repo
baseurl=http://nginx.org/packages/mainline/centos/$releasever/$basearch/
gpgcheck=1
enabled=0
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true

~]# yum -y install nginx
~]# rpm -vih filebeat-6.8.5-x86_64.rpm

1.3.2、将日志修改为json格式

]# vim /etc/nginx/nginx.conf
http {
    # 添加日志格式， log_format
    log_format access_json ‘{"@timestamp":"$time_iso8601",‘
        ‘"host":"$server_addr",‘
        ‘"clientip":"$remote_addr",‘
        ‘"size":$body_bytes_sent,‘
        ‘"responsetime":$request_time,‘
        ‘"upstreamtime":"$upstream_response_time",‘
        ‘"upstreamhost":"$upstream_addr",‘
        ‘"http_host":"$host",‘
        ‘"url":"$uri",‘
        ‘"domain":"$host",‘
        ‘"xff":"$http_x_forwarded_for",‘
        ‘"referer":"$http_referer",‘
        ‘"status":"$status"}‘;
}

server {   # 在server段中使用
    access_log  /var/log/nginx/default_access.log access_json;
}

~]# vim /etc/nginx/nginx.conf   两台nginx都需要添加
# http段中添加， 另外一台当备份， 
    upstream kibana {
       server 192.168.9.135:5601 max_fails=3 fail_timeout=30s;
       server 192.168.9.136:5601 backup;
    }

~]# vim /etc/nginx/conf.d/two.conf 
server {
    listen 5601;
    server_name 192.168.9.133;  # 注意修改主机地址
    access_log  /var/log/nginx/kinaba_access.log access_json;

    location / {
       proxy_pass http://kibana;
    }
}

1.4、logstash+kafka

主机： 192.168.9.137， 9.138

1.4.1、安装kafka

1、 安装jdk 版本1.8

2、安装kafka与zookeeper    注意： 安装两台机器除了监听地址，其它保持一致
    mv kafka_2.11-2.0.0/ zookeeper-3.4.12/ /opt/hadoop/
    cd /opt/hadoop/
    ln -sv kafka_2.11-2.0.0/ kafka
    ln -sv zookeeper-3.4.12/ zookeeper
    cd /opt/hadoop/kafka/config
    vim server.properties   # 修改监听地址
        listeners=PLAINTEXT://192.168.9.138:9092
        log.dirs=/opt/logs/kafka_logs

    vim zookeeper.properties
        dataDir=/opt/logs/zookeeper

    将/opt/hadoop/zookeeper/conf/zoo_sample.cfg 复制为zoo.cfg
    vim /opt/hadoop/zookeeper/conf/zoo.cfg
        dataDir=/opt/logs/zookeeperDataDir

    mkdir /opt/logs/{zookeeper,kafka_logs,zookeeperDataDir} -pv

    chmod +x /opt/hadoop/zookeeper/bin/*.sh
    chmod +x /opt/hadoop/kafka/bin/*.sh 

3、自启
cat kafka.service 
    [Unit]
    Description=kafka 9092
    # 定义kafka.server 应该在zookeeper之后启动
    After=zookeeper.service
    # 强依赖, 必须zookeeper先启动
    Requires=zookeeper.service

    [Service]
    Type=simple
    Environment=JAVA_HOME=/usr/java/default
    Environment=KAFKA_PATH=/opt/hadoop/kafka:/opt/hadoop/kafka/bin
    ExecStart=/opt/hadoop/kafka/bin/kafka-server-start.sh /opt/hadoop/kafka/config/server.properties
    ExecStop=/opt/hadoop/kafka/bin/kafka-server-stop.sh
    Restart=always

    [Install]
    WantedBy=multi-user.target

cat zookeeper.service 
    [Unit]
    Description=Zookeeper Service
    After=network.target
    ConditionPathExists=/opt/hadoop/zookeeper/conf/zoo.cfg

    [Service]
    Type=forking
    Environment=JAVA_HOME=/usr/java/default
    ExecStart=/opt/hadoop/zookeeper/bin/zkServer.sh  start
    ExecStop=/opt/hadoop/zookeeper/bin/zkServer.sh  stop
    Restart=always

    [Install]
    WantedBy=multi-user.target

4、启动
mv kafka.service zookeeper.service /usr/lib/systemd/system
systemctl restart zookeeper kafka
systemctl status zookeeper
systemctl status kafka

ss -tnl
LISTEN      0      50      ::ffff:192.168.9.138:9092        :::*
LISTEN      0      50                        :::2181        :::*

LISTEN      0      50      ::ffff:192.168.9.137:9092        :::*

1.4.2、安装logstash

1、安装logstash
    rpm -ivh logstash-6.8.5.rpm 

    # 或直接yum安装\ 创建repo仓库
    ]# vim /etc/yum.repos.d/logstash.repo
        [logstash-6.x]
        name=Elastic repository for 6.x packages
        baseurl=https://artifacts.elastic.co/packages/6.x/yum
        gpgcheck=1
        gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
        enabled=1
        autorefresh=1
        type=rpm-md

2、配置logstash启动文件 
  sed -i "1a\JAVA_HOME="/usr/java/jdk"" /etc/default/logstash

2、日志收集

2.1、配置nginx-filebeat

# 查看nginx上  filebeat配置    地址： 192.168.9.133
~]# grep -v "#" /etc/filebeat/filebeat.yml | grep -v "^$"
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/kinaba_access.log  # 注意这个文件需要给755权限
  exclude_lines: [‘^DBG‘]
  exclude_files: [‘.gz$‘]
  fields:
    type: kinaba-access-9133
    ip: 192.168.9.133

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3

output.logstash:
  hosts: ["192.168.9.137:5044"]
  worker: 2     # 开启两个工作线程

2.2、配置logstash

~]# cat /etc/logstash/conf.d/nginx-filebeats.conf 
    input {
      beats {
        port => 5044
        codec => "json"
      }
    }

    output {
    #  stdout {   # 养成习惯 先打印 rubydebug输出至屏幕，然后在添加kafka
    #    codec => "rubydebug"
    #  }
      kafka {
         bootstrap_servers => "192.168.9.137:9092"
         codec => "json"
         topic_id => "logstash-kinaba-nginx-access"
      }
    }

# 屏幕输出：  /usr/share/logstash/bin/logstash -f nginx-filebeats.conf
# 检查：     /usr/share/logstash/bin/logstash -f nginx-filebeats.conf -t
# 重启logstash 
# 查看日志：tailf /var/log/logstash/logstash-plain.log

# 查看主题
~]# /opt/hadoop/kafka/bin/kafka-topics.sh --list --zookeeper 192.168.9.137:2181
    logstash-kinaba-nginx-access

# 查看主题内容
~]# /opt/hadoop/kafka/bin/kafka-console-consumer.sh --bootstrap-server 192.168.9.137:9092 --topic logstash-kinaba-nginx-access --from-beginning
    {"host":{"architecture":"x86_64","containerized":false,"os":{"version":"7 (Core)","codename":"Core","platform":"centos","family":"redhat","name":"CentOS Linux"},"name":"test1.xiong.com","id":"e70c4e18a6f243c69211533f14283599"},"@timestamp":"2019-12-27T02:06:17.326Z","log":{"file":{"path":"/var/log/nginx/kinaba_access.log"}},"fields":{"type":"kinaba-access-9133","ip":"192.168.9.133"},"message":"{\"@timestamp\":\:\"-\",\"referer\":\"http://192.168.9.133:5601/app/timelion\",\"status\":\"304\"}","source":"/var/log/nginx/kinaba_access.log","@version":"1","offset":83382,"beat":{"version":"6.8.5","hostname":"test1.xiong.com","name":"test1.xiong.com"},"prospector":{"type":"log"},"input":{"type":"log"},"tags":["beats_input_codec_plain_applied"]}

2.3、ELK上的logstash

# 主机：   192.168.9.135
]# cat /etc/logstash/conf.d/logstash-kinaba-nginx.conf 

input {
  kafka {
    bootstrap_servers => "192.168.9.137:9092"
    decorate_events => true
    consumer_threads => 2
    topics => "logstash-kinaba-nginx-access"
    auto_offset_reset => "latest"
  }
}

output {
#  stdout {   # 养成好习惯，每次都必打印
#    codec => "rubydebug"
#  }

  if [fields][type] == "kinaba-access-9133" {
    elasticsearch {
      hosts => ["192.168.9.135:9200"]
      codec => "json"
      index => "logstash-kinaba-access-%{+YYYY.MM.dd}"
    }
  }
}

# 屏幕输出：  /usr/share/logstash/bin/logstash -f logstash-kinaba-nginx.conf 
# 检查：     /usr/share/logstash/bin/logstash -f logstash-kinaba-nginx.conf  -t
# 查看日志：tailf /var/log/logstash/logstash-plain.log
# 重启logstash

# 静待一会， 多访问几次web, 然后在查看索引
 ~]# curl  http://192.168.9.135:9200/_cat/indices
green open logstash-kinaba-access-2019.12.27 AcCjLtCPTryt6DZkl5KbPw 5 1 100 0 327.7kb 131.8kb

ELK日志收集demo