Monit : 开源监控工具介绍

· Monit 简介

Monit是一个轻量级(500KB)跨平台的用来监控Unix/linux系统的开源工具。部署简单，并且不依赖任何第三方程序、插件或者库。

Monit可以监控服务器进程、文件、文件系统、网络状态（HTTP/SMTP等协议）、远程主机、服务器资源变化等等。 并且可以设定资源变化后需要做的动作，比如服务失败后自动重启，邮件告警等等。

Monit内置了WEB UI，可以一目了然地了解监控项的情况。Monit是监控本机服务的工具，M/Monit是其配套产品用以对Monit统一管理，但是M/Monit并不是开源的，需要购买。

本文我们介绍在CentOS7.3上部署Monit用来监控进程状态，并实现服务失败后自动重启以及邮件告警通知的功能。监控其他资源的方法在默认配置文件中都有举例这里不一一介绍。

Monit官网：https://mmonit.com/monit

Monit手册：https://mmonit.com/monit/documentation/monit.html

· Monit 安装

# rpm -ivh http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
# yum install -y monit

·Monit 常用命令

monit -t # 配置文件检测
monit # 启动monit daemon
monit -c /var/monit/monitrc # 启动monit daemon时指定配置文件
monit reload # 重新加载配置文件
monit status # 查看所有监控项务状态
monit status nginx # 查看nginx服务状态
monit stop all # 停止所有服务，这里需要注意的是，如果开启了自动重启功能，停止某个被监控的服务必须用monit stop xxx，若用系统命令停止服务，Monit会自动再把服务起来。
monit stop nginx # 停止nginx服务
monit start all # 启动所有服务
monit start nginx # 启动nginx服务
monit -V # 查看版本

·Monit 配置文件

/etc/monit.conf 主配置文件

/etc/monit.d/ 各项服务单独配置文件路径，在主配置文件中将其include进来。

/etc/monit.conf 举例说明：

配置文件关键字：'if', 'and', 'with(in)', 'has', 'us(ing|e)', 'on(ly)', 'then', 'for', 'of'

###############################################################################
## Monit control file
###############################################################################
##
## Comments begin with a '#' and extend through the end of the line. Keywords
## are case insensitive. All path's MUST BE FULLY QUALIFIED, starting with '/'.
##
## Below you will find examples of some frequently used statements. For 
## information about the control file and a complete list of statements and 
## options, please have a look in the Monit manual.
##
##
###############################################################################
## Global section
###############################################################################
##
## Start Monit in the background (run as a daemon):
## 设置检测周期30s
set daemon  30              # check services at 30 seconds intervals
#   with start delay 240    # optional: delay the first check by 4-minutes (by 
#                           # default Monit check immediately after Monit start)
#
#
## Set syslog logging. If you want to log to a standalone log file instead,
## specify the full path to the log file
##设置log路径，这里默认记录到syslog
set logfile syslog
#
#
## Set the location of the Monit lock file which stores the process id of the
## running Monit instance. By default this file is stored in $HOME/.monit.pid
#
# set pidfile /var/run/monit.pid
#
## Set the location of the Monit id file which stores the unique id for the
## Monit instance. The id is generated and stored on first Monit start. By 
## default the file is placed in $HOME/.monit.id.
#
# set idfile /var/.monit.id
#
## Set the location of the Monit state file which saves monitoring states
## on each cycle. By default the file is placed in $HOME/.monit.state. If
## the state file is stored on a persistent filesystem, Monit will recover
## the monitoring state across reboots. If it is on temporary filesystem, the
## state will be lost on reboot which may be convenient in some situations.
#
# set statefile /var/.monit.state
#
## Set the list of mail servers for alert delivery. Multiple servers may be 
## specified using a comma separator. If the first mail server fails, Monit 
# will use the second mail server in the list and so on. By default Monit uses 
# port 25 - it is possible to override this with the PORT option.
#
# set mailserver mail.bar.baz,               # primary mailserver
#                backup.bar.baz port 10025,  # backup mailserver on port 10025
#                localhost                   # fallback relay
## 设置邮件服务器用来发送邮件告警通知
  set mailserver mail.abcd.so
#
## By default Monit will drop alert events if no mail servers are available. 
## If you want to keep the alerts for later delivery retry, you can use the 
## EVENTQUEUE statement. The base directory where undelivered alerts will be 
## stored is specified by the BASEDIR option. You can limit the queue size 
## by using the SLOTS option (if omitted, the queue is limited by space
## available in the back end filesystem).
#
# set eventqueue
#     basedir /var/monit  # set the base directory where events will be stored
#     slots 100           # optionally limit the queue size
#
#
## Send status and events to M/Monit (for more informations about M/Monit 
## see http://mmonit.com/). By default Monit registers credentials with 
## M/Monit so M/Monit can smoothly communicate back to Monit and you don't
## have to register Monit credentials manually in M/Monit. It is possible to
## disable credential registration using the commented out option below. 
## Though, if safety is a concern we recommend instead using https when
## communicating with M/Monit and send credentials encrypted.
#
# set mmonit http://monit:[email protected]:8080/collector
#     # and register without credentials     # Don't register credentials
#
#
## Monit by default uses the following format for alerts if the the mail-format
## statement is missing::
## --8<--# 设置邮件告警通知格式
 set mail-format {
      from: monit@$HOST
   subject: monit alert --  $EVENT $SERVICE
   message: $EVENT Service $SERVICE
                 Date:        $DATE
                 Action:      $ACTION
                 Host:        $HOST
                 Description: $DESCRIPTION

            Your faithful employee,
            Monit
 }
## --8<--
##
## You can override this message format or parts of it, such as subject
## or sender using the MAIL-FORMAT statement. Macros such as $DATE, etc.
## are expanded at runtime. For example, to override the sender, use:
#
# set mail-format { from: [email protected] }
#
#
## You can set alert recipients whom will receive alerts if/when a 
## service defined in this file has errors. Alerts may be restricted on 
## events by using a filter as in the second example below.
## 设置邮件告警通知人，Monit默认会通知monit进程本身的变化情况，如果不想收到monit进程自身的通知，加上but not on {instance}配置
 set alert  [email protected] but not on { instance }                    # receive all alerts
#
## Do not alert when Monit starts, stops or performs a user initiated action.
## This filter is recommended to avoid getting alerts for trivial cases.
#
# set alert [email protected] 
#
#
## Monit has an embedded HTTP interface which can be used to view status of 
## services monitored and manage services from a web interface. The HTTP 
## interface is also required if you want to issue Monit commands from the
## command line, such as 'monit status' or 'monit restart service' The reason
## for this is that the Monit client uses the HTTP interface to send these
## commands to a running Monit daemon. See the Monit Wiki if you want to 
## enable SSL for the HTTP interface. 
## 设置UI界面访问信息
set httpd port 2812 and
    use address 10.2.2.28  # only accept connection from localhost
#   allow localhost        # allow localhost to connect to the server and
    allow admin:monit      # require user 'admin' with password 'monit'

###############################################################################
## Services
###############################################################################
##
## Check general system resources such as load average, cpu and memory
## usage. Each test specifies a resource, conditions and the action to be
## performed should a test fail.
#
#  check system $HOST
#    if loadavg (1min) > 4 then alert
#    if loadavg (5min) > 2 then alert
#    if cpu usage > 95% for 10 cycles then alert
#    if memory usage > 75% then alert
#    if swap usage > 25% then alert
#
#    
## Check if a file exists, checksum, permissions, uid and gid. In addition
## to alert recipients in the global section, customized alert can be sent to 
## additional recipients by specifying a local alert handler. The service may 
## be grouped using the GROUP option. More than one group can be specified by
## repeating the 'group name' statement.
#    
#  check file apache_bin with path /usr/local/apache/bin/httpd
#    if failed checksum and 
#       expect the sum 8f7f419955cefa0b33a2ba316cba3659 then unmonitor
#    if failed permission 755 then unmonitor
#    if failed uid root then unmonitor
#    if failed gid root then unmonitor
#    alert [email protected] on {
#           checksum, permission, uid, gid, unmonitor
#        } with the mail-format { subject: Alarm! }
#    group server
#
#    
## Check that a process is running, in this case Apache, and that it respond
## to HTTP and HTTPS requests. Check its resource usage such as cpu and memory,
## and number of children. If the process is not running, Monit will restart 
## it by default. In case the service is restarted very often and the 
## problem remains, it is possible to disable monitoring using the TIMEOUT
## statement. This service depends on another service (apache_bin) which
## is defined above.
#    
#  check process apache with pidfile /usr/local/apache/logs/httpd.pid
#    start program = "/etc/init.d/httpd start" with timeout 60 seconds
#    stop program  = "/etc/init.d/httpd stop"
#    if cpu > 60% for 2 cycles then alert
#    if cpu > 80% for 5 cycles then restart
#    if totalmem > 200.0 MB for 5 cycles then restart
#    if children > 250 then restart
#    if loadavg(5min) greater than 10 for 8 cycles then stop
#    if failed host www.tildeslash.com port 80 protocol http 
#       and request "/somefile.html"
#    then restart
#    if failed port 443 type tcpssl protocol http
#       with timeout 15 seconds
#    then restart
#    if 3 restarts within 5 cycles then unmonitor
#    depends on apache_bin
#    group server
#  # 监控进程可以通过上面监控pid文件的方式，当没有pid文件时，可以通过MATCHING正则表达式来匹配进程。# 测试一个进程是否匹配来自命令行使用的模式monit procmatch "regex-pattern"，这将列出匹配或不匹配的所有进程，regex模式。# 我们这里监控了包含shop-pad-server字段的进程，并指明了启动以及停止的命令，这样在进程因故断掉后，Monit会自动重启进程。# 同时若进程ID变动，会发送邮件通知到之前指定的收件人。

check process shop-pad-server with MATCHING shop-pad-server
      start program = "/usr/bin/nohup /home/azureuser/pad-server/run.sh > /home/azureuser/pad-server/nohup.out 2>&1 &" 
      stop program  = "/usr/bin/ps -ef | /usr/bin/grep shop-pad| /usr/bin/grep -v grep | /usr/bin/awk '{print $2}' | xargs kill"
      if changed pid then alert

#    
## Check filesystem permissions, uid, gid, space and inode usage. Other services,
## such as databases, may depend on this resource and an automatically graceful
## stop may be cascaded to them before the filesystem will become full and data
## lost.
#
#  check filesystem datafs with path /dev/sdb1
#    start program  = "/bin/mount /data"
#    stop program  = "/bin/umount /data"
#    if failed permission 660 then unmonitor
#    if failed uid root then unmonitor
#    if failed gid disk then unmonitor
#    if space usage > 80% for 5 times within 15 cycles then alert
#    if space usage > 99% then stop
#    if inode usage > 30000 then alert
#    if inode usage > 99% then stop
#    group server
#
#
## Check a file's timestamp. In this example, we test if a file is older 
## than 15 minutes and assume something is wrong if its not updated. Also,
## if the file size exceed a given limit, execute a script
#
#  check file database with path /data/mydatabase.db
#    if failed permission 700 then alert
#    if failed uid data then alert
#    if failed gid data then alert
#    if timestamp > 15 minutes then alert
#    if size > 100 MB then exec "/my/cleanup/script" as uid dba and gid dba
#
#
## Check directory permission, uid and gid.  An event is triggered if the 
## directory does not belong to the user with uid 0 and gid 0.  In addition, 
## the permissions have to match the octal description of 755 (see chmod(1)).
#
#  check directory bin with path /bin
#    if failed permission 755 then unmonitor
#    if failed uid 0 then unmonitor
#    if failed gid 0 then unmonitor
#
#
## Check a remote host availability by issuing a ping test and check the 
## content of a response from a web server. Up to three pings are sent and 
## connection to a port and an application level network check is performed.
#
#  check host myserver with address 192.168.1.1
#    if failed ping then alert
#    if failed port 3306 protocol mysql with timeout 15 seconds then alert
#    if failed port 80 protocol http
#       and request /some/path with content = "a string"
#    then alert
#
#
## Check a network link status (up/down), link capacity changes, saturation
## and bandwidth usage.
#
#  check network public with interface eth0
#    if failed link then alert
#    if changed link then alert
#    if saturation > 90% then alert
#    if download > 10 MB/s then alert
#    if total upload > 1 GB in last hour then alert
#
#
## Check custom program status output.
#
#  check program myscript with path /usr/local/bin/myscript.sh
#    if status != 0 then alert
#
#
###############################################################################
## Includes
###############################################################################
##
## It is possible to include additional configuration parts from other files or
## directories.
#
#  include /etc/monit.d/*
#
# 导入其他单项服务的监控配置
# Include all files from /etc/monit.d/
include /etc/monit.d/*

· 通知周期设置

　Monit默认情况下如果一个服务失败只发送一个通知：
　　alert foo@bar　如果您希望在服务保持处于失败状态时每十个周期通知一次，您可以使用：
　　alert foo@bar with reminder on 10 cycles　同样，如果您想在每个失败的周期获得通知，您可以使用：
　　alert foo@bar with reminder on 1 cycle要禁止某些用户和服务的警报，可以在服务检查的局部配置里添加语句：
　　noalert mail-address

·  服务检测周期设置

可以使用every语句修改服务检查计划。

1.轮询周期倍数
EVERY [number] CYCLES

2.Cron-style
EVERY [cron]

# [cron]
# * * * * *
# 分 时 日 月 周

3.与Cron-style相反（do-not-check）
NOT EVERY [cron]

示例：
示例1：每两个周期检查一次
check process nginx with pidfile /var/run/nginx.pid
every 2 cycles

示例2：在上午8点到下午7点之间检查每个工作日
check program checkOracleDatabase
with path /var/monit/programs/checkoracle.pl
every "* 8-19 * * 1-5"

示例3：在星期日0AM到3AM之间不要在备份窗口中运行检查，否则运行具有常规轮询周期频率的检查。
check process mysqld with pidfile /var/run/mysqld.pid
not every "* 0-3 * * 0"

注意不要使用特定的分钟，因为Monit可能不会在那分钟运行。

· WEB UI界面

参考：

http://www.cnblogs.com/52fhy/p/6412547.html

http://blog.csdn.net/senlin1202/article/details/54291176

https://mmonit.com/monit/documentation/monit.html