centos7 rabbitmq集群搭建+高可用

虚拟机环境(所有节点)

环境
[ ~]# cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)
[ ~]# uname -r 3.10.0-229.el7.x86_64
10.0.0.20 node1 ram节点 (rabbitmq )
10.0.0.21 node2 disc节点 (rabbitmq + haproxy + keepalived )
10.0.0.22 node3 ram节点 (rabbitmq + haproxy + keepalived )

搭建(在所有节点执行)

先搭建好基础集群

添加EPEL源
[ ~]# rpm -Uvh https://dl.Fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
添加Erlang
[ ~]# rpm -Uvh http://packages.erlang-solutions.com/erlang-solutions-1.0-1.noarch.rpm
安装RabbitMQ
[ ~]# wget http://www.rabbitmq.com/releases/rabbitmq-server/v3.6.6/rabbitmq-server-3.6.6-1.el7.noarch.rpm
[ ~]# rpm --import https://www.rabbitmq.com/rabbitmq-signing-key-public.asc
[ ~]# yum install -y rabbitmq-server-3.6.6-1.el7.noarch.rpm
启动服务
[ ~]# systemctl enable rabbitmq-server.service
[ ~]# systemctl start rabbitmq-server.service
查看服务状态
[ ~]# systemctl status rabbitmq-server
查看端口监听状态 5672为集群间通信端口,15672为web端管理端口
[ ~]# netstat -lnutp
查看配置路径
[ ~]# cd /var/log/rabbitmq/
[ rabbitmq]# vim rabbit\@node2.log =INFO REPORT==== 20-Aug-2017::08:03:11
=== Starting RabbitMQ 3.6.12 on Erlang R16B03-1
Copyright (C) 2007-2017 Pivotal Software, Inc.
Licensed under the MPL. See http://www.rabbitmq.com/
=INFO REPORT==== 20-Aug-2017::08:03:11 ===
node :
home dir : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config (not found)
cookie hash : vNIydXbvZSku+QwtRGImSQ==
log : /var/log/rabbitmq/ sasl
log : /var/log/rabbitmq/ database
dir : /var/lib/rabbitmq/mnesia/
 
根据上面的提示发现没有 rabbitmq.config 文件,可以自己创建
[ rabbitmq]# cd /etc/rabbitmq/
[ rabbitmq]# vim rabbitmq.config
 
编辑内容如下
[{rabbit, [{loopback_users, []}]}].
这里的意思是开放使用,rabbitmq默认创建的用户guest,密码也是guest,这个用户默认只能是本机访问,localhost或者127.0.0.1,从外部访问需要添加上面的配置 
保存配置后重启服务
[ rabbitmq]# systemctl restart rabbitmq-server
此时就可以从外部访问了,但此时再看log文件,发现内容还是原来的,还是显示没有找到配置文件
[ rabbitmq]# vim rabbit\@node2.log =INFO REPORT==== 20-Aug-2017::08:03:11 ===
Starting RabbitMQ 3.6.12 on Erlang R16B03-1 Copyright (C) 2007-2017
Pivotal Software, Inc. Licensed under the MPL. See http://www.rabbitmq.com/
=INFO REPORT==== 20-Aug-2017::08:03:11 ===
node : home dir : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config (not found)
cookie hash : vNIydXbvZSku+QwtRGImSQ== log : /var/log/rabbitmq/
sasl log : /var/log/rabbitmq/
database dir : /var/lib/rabbitmq/mnesia/
 
手动删除这个文件再重启服务,不过这不影响使用
[ rabbitmq]# rm rabbit\@node2.log
[ rabbitmq]# systemctl restart rabbitmq-server
在此查看log文件,OK了
[ rabbitmq]# vim rabbit\@node2.log
=INFO REPORT==== 20-Aug-2017::08:26:56 ===
Starting RabbitMQ 3.6.12 on Erlang R16B03-1
Copyright (C) 2007-2017 Pivotal Software, Inc.
Licensed under the MPL. See http://www.rabbitmq.com/
=INFO REPORT==== 20-Aug-2017::08:26:56 ===
node : home dir : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash : vNIydXbvZSku+QwtRGImSQ== log : /var/log/rabbitmq/
sasl log : /var/log/rabbitmq/
database dir : /var/lib/rabbitmq/mnesia/
 
启用RabbitMQ Web插件
[ ~]# rabbitmq-plugins enable rabbitmq_management
关闭Rabbitmq Web管理接口
[ rabbitmq]# rabbitmq-plugins disable rabbitmq_management
 
RabbitMQ用户管理
添加用户为:root ,密码为:admin)
[ ~]# rabbitmqctl add_user admin admin
设置用户角色(设置admin用户为管理员角色)
[ ~]# rabbitmqctl set_user_tags admin administrator
设置用户权限(设置admin用户配置、写、读的权限)
[ ~]# rabbitmqctl set_permissions -p / admin ".*" ".*" ".*"
删除用户(删除guest用户)
[ ~]# rabbitmqctl delete_user guest
如果配置防火墙,开启端口
[ rabbitmq]# firewall-cmd --zone=public --add-port=15672/tcp --permanent [ rabbitmq]# firewall-cmd --reload
在Web输入服务器IP:端口
http://IP:15672
 
RabbitMQ集群配置
设置hosts解析,所有节点配置相同
[ ~]# vim /etc/hosts 10.0.0.20 node1 10.0.0.21 node2 10.0.0.22 node3
设置Erlang Cookie
RabbitMQ节点之间和命令行工具 (e.g. rabbitmqctl)是使用Cookie互通的,Cookie是一组随机的数字+字母的字符串。当RabbitMQ服务器启动的时候,Erlang VM会自动创建一个随机内容的Cookie文件。
如果是通过源安装RabbitMQ的话,Erlang Cookie 文件在/var/lib/rabbitmq/.erlang.cookie。
如果是通过源码安装的RabbitMQ,Erlang Cookie文件$HOME/.erlang.cookie。
 
修改node1改文件的权限
[ ~]# chmod 777 /var/lib/rabbitmq/.erlang.cookie
然后将文件复制到node2、node3上面
node2
[ ~]# cd /var/lib/rabbitmq
[ rabbitmq ~]# chmod 777 .erlang.cookie
[ rabbitmq ~]# scp -r 10.0.0.20:/var/lib/rabbitmq/.erlang.cookie .
[ rabbitmq ~]# chmod 400 .erlang.cookie
node3
[ ~]# cd /var/lib/rabbitmq
[ rabbitmq ~]# chmod 777 .erlang.cookie
[ rabbitmq ~]# scp -r 10.0.0.20:/var/lib/rabbitmq/.erlang.cookie .
[ rabbitmq ~]# chmod 400 .erlang.cookie
 
将node1的文件权限恢复过来
[ ~]# chmod 400 /var/lib/rabbitmq/.erlang.cookie
 
查看三台机器上的 .erlang.cookie的值是一致的
[ ~]# cat /var/lib/rabbitmq/.erlang.cookie CAWVPZYTTTUGFZTYNUHG
[ ~]# cat /var/lib/rabbitmq/.erlang.cookie CAWVPZYTTTUGFZTYNUHG
[ ~]# cat /var/lib/rabbitmq/.erlang.cookie CAWVPZYTTTUGFZTYNUHG
 
使用detached参数,在后台启动Rabbit Node,要先停止现有的Rabbitmq-server,再重新在后台执行
node1
[ ~]# rabbitmqctl stop
[ ~]# rabbitmq-server -detached
[ ~]# rabbitmqctl cluster_status Cluster status of node ... [{nodes,[{disc,[]}]}, {running_nodes,[]}, {cluster_name,<<"">>}, {partitions,[]}, {alarms,[{,[]}]}]
node2
[ ~]# rabbitmqctl stop
[ ~]# rabbitmq-server -detached
[ ~]# rabbitmqctl cluster_status Cluster status of node ... [{nodes,[{disc,[]}]}, {running_nodes,[]}, {cluster_name,<<"">>}, {partitions,[]}, {alarms,[{,[]}]}]
node3
[ ~]# rabbitmqctl stop
[ ~]# rabbitmq-server -detached
[ ~]# rabbitmqctl cluster_status Cluster status of node ... [{nodes,[{disc,[]}]}, {running_nodes,[]}, {cluster_name,<<"">>}, {partitions,[]}, {alarms,[{,[]}]}]
 
将node1、node2、node3组成集群
因为rabbitmq-server启动时,会一起启动节点和应用,它预先设置RabbitMQ应用为standalone模式。
要将一个节点加入到现有的集群中,你需要停止这个应用并将节点设置为原始状态,然后就为加入集群准备好了。
如果使用./rabbitmqctl stop,应用和节点都将被关闭。所以使用rabbitmqctl stop_app仅仅关闭应用。
node1
[ ~]# rabbitmqctl stop_app Stopping node ...
[ ~]# rabbitmqctl join_cluster --rm Clustering node with ...
[ ~]# rabbitmqctl start_app Starting node ...
 
node3
[ ~]# rabbitmqctl stop_app Stopping node ...
[ ~]# rabbitmqctl join_cluster --rm Clustering node with ...
[ ~]# rabbitmqctl start_app Starting node ...
 
此时 node1 与 node3 也会自动建立连接
如果要使用内存节点,则可以使用以下命令
[ ~]# rabbitmqctl join_cluster --ram
集群配置好后,可以在 RabbitMQ 任意节点上执行 rabbitmqctl cluster_status 来查看是否集群配置成功
[ ~]# rabbitmqctl cluster_status Cluster status of node ... [{nodes,[{disc,[]},{{ram,[,]}]}, {running_nodes,[,,]}, {cluster_name,<<"">>}, {partitions,[]}, {alarms,[{,[]},{,[]},{,[]}]}]
 
RabbitMQ镜像功能(所有节点执行)
 
设置policy,以所有队列被镜像到集群其他所有节点,一个节点挂掉然后重启后会自动同步队列消息(我们生产环境采用这个方式)
[ ~]# rabbitmqctl set_policy ha-all-queue "^" ‘{"ha-mode":"all","ha-sync-mode":"automatic"}‘
安装软件负载均衡器HAProxy(node3相同操作)
安装haproxy
[ ~]# yum install -y haproxy
安装完以后,配置文件的目录在/etc/haproxy/haproxy.cfg,以下是我修改后的配置文件
[ ~]# cd /etc/haproxy/
[ haproxy]# vim haproxy.cfg
###########全局配置#########
global
  log /dev/log local0
  log /dev/log local1 notice
# chroot /var/lib/haproxy
# 改变当前工作目录
# stats socket /run/haproxy/admin.sock mode 660 level admin                  # 创建监控所用的套接字目录
# stats socket /var/lib/haproxy/stats
  pidfile /var/run/haproxy.pid                          # haproxy的pid存放路径,启动进程的用户必须有权限访问此文件
  maxconn 4000                                           # 最大连接数,默认4000
  user haproxy                                            # 默认用户
  group haproxy                                        # 默认用户组
  daemon                                                 # 创建1个进程进入deamon模式运行。此参数要求将运行模式设置为"daemon
 
 
###########默认配置#########
defaults
   log global
   mode http
# option dontlognull
   option httplog
   timeout connect 5000                 # 连接超时时间
   timeout client 50000                  # 客户端连接超时时间
   timeout server 50000              # 服务器端连接超时时间
   option httpclose                     # 每次请求完毕后主动关闭http通道
   option httplog                        # 日志类别http日志格式
   option forwardfor                 # 如果后端服务器需要获得客户端真实ip需要配置的参数,可以从Http Header中获得客户端ip
   option redispatch               # serverId对应的服务器挂掉后,强制定向到其他健康的服务器
   timeout connect 10000      # default 10 second timeout if a backend is not found
   maxconn 60000                # 最大连接数
   retries 3                           # 3次连接失败就认为服务不可用,也可以通过后面设置
#     errorfile 400 /etc/haproxy/errors/400.http
#     errorfile 403 /etc/haproxy/errors/403.http
#     errorfile 408 /etc/haproxy/errors/408.http
#     errorfile 500 /etc/haproxy/errors/500.http
#     errorfile 502 /etc/haproxy/errors/502.http
#     errorfile 503 /etc/haproxy/errors/503.http
#     errorfile 504 /etc/haproxy/errors/504.http
 
####################################################################
 
listen http_front bind 0.0.0.0:1080                #监听端口
stats refresh 30s                            #统计页面自动刷新时间
stats uri /haproxy?stats                  #统计页面url
stats realm Haproxy Manager         #统计页面密码框上提示文本
stats auth admin:admin                  #统计页面用户名和密码设置
#stats hide-version                        #隐藏统计页面上HAProxy的版本信息
 
#####################我把RabbitMQ的管理界面也放在HAProxy后面了
 
###############################
listen rabbitmq_admin
    bind 0.0.0.0:15672
    server node1 10.0.0.20:15672 
    server node2 10.0.0.21:15672
    server node3 10.0.0.22:15672
####################################################################
 
listen rabbitmq_cluster
    bind 0.0.0.0:5672
    option tcplog
    mode tcp
    timeout client 3h
    timeout server 3h
    option clitcpka
    balance roundrobin                             #负载均衡算法(#banlance roundrobin 轮询,balance source 保存session值,支持static-rr,leastconn,first,uri等参数)
    #balance url_param userid #balance url_param session_id check_post 64 #balance hdr(User-Agent) #balance hdr(host) #balance hdr(Host) use_domain_only #balance rdp-cookie #balance leastconn #balance source //ip server node1 10.0.0.20:5672 check inter 5s rise 2 fall 3 #check inter 2000 是检测心跳频率,rise 2是2次正确认为服务器可用,fall 3是3次失败认为服务器不可用 server node2 10.0.0.21:5672 check inter 5s rise 2 fall 3 server node3 10.0.0.22:5672 check inter 5s rise 2 fall 3
启动
[ ~]# systemctl haproxy start
关闭
[ ~]# service haproxy stop
重启
[ ~]# service haproxy restart
加入开机启动
[ ~]# systemctl enable haproxy
查看开机启动的程序
[ ~]# ls /etc/systemd/system/multi-user.target.wants/
Web访问
http://10.0.0.21:1080/haproxy?stats
 
配置Haproxy增加日志记录功能(node3相同操作)
创建日志文件/var/log/haproxy/haproxy.log
[ ~]# cd /var/log/ [ log]# mkdir haproxy [ log]# cd haproxy [ haproxy]# touch haproxy.log [ haproxy]# chmod a+w haproxy.log
开启rsyslog的haproxy日志记录功能,编辑/etc/rsyslog.conf文件,将下列两句开头的 # 注释去掉
[ ~]# vim /etc/rsyslog.conf
#$ModLoad imudp #$UDPServerRun 514
$ModLoad imudp $UDPServerRun 514
# Save boot messages also to boot.log local7.* /var/log/boot.log
之后添加
# Save haproxy log local0.* /var/log/haproxy/haproxy.log
修改/etc/sysconfig/rsyslog 文件
[ ~]# vim /etc/sysconfig/rsyslog
SYSLOGD_OPTIONS=""
改为
SYSLOGD_OPTIONS="-r -m 0 -c 2"
修改/etc/haproxy/haproxy.cfg文件,在global区段添加
log 127.0.0.1 local0
重启rsyslog和haproxy服务
[ ~]# service haproxy restart [ ~]# service rsyslog restart
查看日志
[ ~]# tail -f /var/log/haproxy/haproxy.log
 
安装keepalived(node3相同操作 keepalived.conf 配置不同)
 
安装
[ ~]# yum -y install keepalived
查看版本
[ ~]# keepalived -v
配置 keepalived.conf (10.0.0.21)
[ ~]# vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { } notification_email_from smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id rabbitmq #集群名称,相同集群名称必须相同 } vrrp_instance VI_1 { state MASTER #主从设置,这里设置为主 interface eth0 virtual_router_id 51 priority 150 #定义访问优先级,数值越大优先级越高 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 10.0.0.30 #置虚拟IP,需在相同网段 } }
 
配置 10.0.0.22(备)
[ ~]# vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { } notification_email_from smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id rabbitmq #集群名称,必须和web01设置相同 } vrrp_instance VI_1 { state BACKUP #主从设置,这里设置为从 interface eth0 virtual_router_id 51 priority 100 #定义访问优先级,从要比主的数值小 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 10.0.0.30 #设置虚拟IP,必须相同 } }
 
重启服务查看node2(10.0.0.21)网卡上是否有 10.0.0.30 IP
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:e6:36:7b brd ff:ff:ff:ff:ff:ff inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0 valid_lft forever preferred_lft forever inet 10.0.0.30/32 scope global eth0
而此是node3(10.0.0.22)的网卡上是没有10.0.0.30 IP
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:15:ba:84 brd ff:ff:ff:ff:ff:ff inet 10.0.0.22/24 brd 10.0.0.255 scope global eth0 valid_lft forever preferred_lft forever
可以关闭node2(10.0.0.21)的keepalived 服务,查看IP是否转移到了node3(10.0.0.22)上
node2(10.0.0.21)上已经没有了10.0.0.30
[ ~]# systemctl stop keepalived 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:e6:36:7b brd ff:ff:ff:ff:ff:ff inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0 valid_lft forever preferred_lft forever
 
查看node3(10.0.0.22)
 
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:15:ba:84 brd ff:ff:ff:ff:ff:ff inet 10.0.0.22/24 brd 10.0.0.255 scope global eth0 valid_lft forever preferred_lft forever inet 10.0.0.30/32 scope global eth0 valid_lft forever preferred_lft forever
 
此时10.0.0.30 已经转移到了 10.0.0.22上
启动keepalived即可,10.0.0.30是对外提供的统一地址。
通过10.0.0.30:5672就可以访问rabbitmq服务
注意:keepalived可能运行多个实例,比如redis和haproxy共存,在这种情况下,必须注意几点:
一、VIP必须各实例不同
二、virtual_router_id必须各实例不同