LVS双机热备和负载均衡一体化安装方法
heartbeat 以心跳机制监控实体主机是否存活，实现ipvsadm和ldirectord服务的故障转移。
ldirectord 可用来監控实体主機的服務进程是否正常，以决定在ipvsadm配置中删除或加入。
在本案例中，将使用CentOS6系统来安装传统的heartbeat+ldirector以实现双机热备的功能,因为RHEL均已经移除相关套件，所以安装比较麻烦一点，但只要一步一步做下来，其实也并不是太困难，而且还能够学习到很多东西。
另外请注意，因为本案例中将DR主机和RS主机合并在一起以节省IT资源，因此本案例只适合部署于同一子网内的主机(因为虚拟的VIP不能改变)，也因此之故，ipvsadm也就可选择使用Director Routing 模式。
________
| |
| client |
|________|
|
CIP->VIP | | ^
v | | VIP->CIP
|
------------------------------------
| |
| |
RIP1=192.168.57.101 RIP2=192.168.57.102 (eth0)
VIP =192.168.57.200 VIP =192.168.57.200 (eth0:0, arps)
VIP =192.168.57.200 VIP =192.168.57.200 (all lo:0,non-arping)
_____________ _____________
| | | |
| Director1 + | | Director2 + |
| RealServer1 | | RealServer2 |
|_____________| |_____________|
nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn
调试应注意的情况:
uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu
另外，调试中要注意一个情况，ipvsadm所添加的规则应用于内核的路由，优先于其他网络
服务。举个例子来说，比如某IP的端口80被ipsvadm设置为转发，而同时本机又有HTTPD服务
使用此IP的80端口，那么ipvsadm规则将应用在先，而其他IP的端口80的服务则不受影响。
nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn
注意ARP地址的更新：
uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu
因为我使用虚拟主机进行测试，所以ARP地址需要手动更新，否则测试会停留在旧地址上：
# arping -I vboxnet1 192.168.57.200;
一般LINUX系统预设的ARP缓存timeout时间是60秒，如有需要，您可以调整此内核参数：
# cat /proc/sys/net/ipv4/neigh/vboxnet1/gc_stale_time
# echo 5 > /proc/sys/net/ipv4/neigh/vboxnet1/gc_stale_time
如果您使用硬件设备作网关，如有需要，则请相应调整的硬件参数，例如，CISCO路由器的
预设ARP缓存时间是14400秒（4小时），如果您不想修改此参数，那就肯定需要手动刷新了。
CISCO 路由器用于操作 arp cache 的相关命令可参考如下：
router# show arp
router# clear arp-cache
────────────────────────────────────────────────────────────────────────────────
LVS
────────────────────────────────────────────────────────────────────────────────
Official Site:
http://www.linuxvirtualserver.org/
http://zh.linuxvirtualserver.org/
Ref:
http://www.ultramonkey.org/3/lvs.html
http://www.ultramonkey.org/3/linux-ha.html
http://www.ultramonkey.org/3/ldirectord.html
# yum install ipvsadm;
注意： 在本案例中，因为双机热备需要两边node完全相同的配置，因此所有命令都同时
适用于两台node主机，除了 heartbeat配置配置文档/etc/ha.d/ha.cf有点稍微不同之外，
其他配置也完全相同。
────────────────────────────────────────────────────────────────────────────────
Linux-HA:注意最新版本(heartbeat 3)作了相当大的改动
────────────────────────────────────────────────────────────────────────────────
http://www.linux-ha.org/wiki/Main_Page
Heartbeat 3 的组成
‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
Heartbeat 3 按模块拆分为三个子项目，提供了一个名为 cluster-glue 的组件， 专用于
Local Resource Manager 的管理。即heartbeat + cluster-glue + resouce-agent三部分：
(1) hearbeat集群心跳, 负责维护集群各节点的信息以及它们之前通信;
(2) cluster-glue相当于一个中间层，可将heartbeat和crm(pacemaker)联系起来，包括LRM和STONITH;
(3) resource-agent各种资源的ocf脚本，这些脚本将被LRM调用从而实现各种资源启动、停止、监控等;
http://www.linux-ha.org/wiki/Download
In post-2.1.4 releases, Linux-HA code is packaged in three different sub-projects:
* Heartbeat:http://www.linux-ha.org/wiki/Heartbeat
* Cluster Glue:http://www.linux-ha.org/wiki/Cluster_Glue
* Resource Agents:http://www.linux-ha.org/wiki/Resource_Agents
http://www.linux-ha.org/wiki/Resource_Agents
Pacemaker supports three types of Resource Agents:
* LSB Resource Agents:http://www.linux-ha.org/wiki/LSB_Resource_Agents
* OCF Resource Agents:http://www.linux-ha.org/wiki/OCF_Resource_Agents
* legacy Heartbeat Resource Agents: http://www.linux-ha.org/wiki/Heartbeat_Resource_Agents
Mercurial Repositories:
http://hg.linux-ha.org/agents/
################################################################################
RHEL6/CentOS6 (Heartbeat3: heartbeat + cluster-glue + resouce-agent)
################################################################################
REF:
http://ftp.jaist.ac.jp/pub/Linux/Fedora/
http://ftp.jaist.ac.jp/pub/Linux/Fedora/epel/6/i386/
http://ftp.jaist.ac.jp/pub/Linux/Fedora/epel/6/x86_64/
# rpm -Uvh http://ftp.jaist.ac.jp/pub/Linux/Fedora/epel/6/i386/epel-release-6-7.noarch.rpm
# yum search heartbeat;
# yum install heartbeat;
────────────────────────────────────────────────────────────────────────────────
Install ldirectord:
────────────────────────────────────────────────────────────────────────────────
nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn
Note: RHEL6 only added ldirectord to RHN Satellite server in a separate channel.
uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu
http://horms.net/projects/ldirectord/
http://en.wikipedia.org/wiki/Ldirectord
ldirector (Linux Director Daemon) is a background process computer program used
to monitor and administer real servers in the Linux Virtual Server (LVS) cluster.
********************************************************************************
Download: http://www.linbit.com/support/hb/rhel6/ (Need non-free support login )
********************************************************************************
wget http://www.linbit.com/support/hb/rhel6/ldirectord-3.9.3-1.i686.rpm
wget http://www.linbit.com/support/hb/rhel6/ldirectord-3.9.3-1.x86_64.rpm
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# vi /etc/yum.repos.d/slinux.repo;#(same as centos6, not include ldirectord)
--------------------------------------------------------------------------------
[scientific-linux]
name=Scientific Linux
#baseurl=http://ftp.scientificlinux.org/linux/scientific/6/x86_64/os/
baseurl=http://ftp.scientificlinux.org/linux/scientific/6/i386/os/
enabled=0
--------------------------------------------------------------------------------
Note: this is not working, here is only for recoding the RedHat dirty things !
http://hg.linux-ha.org/agents/
https://github.com/ClusterLabs/resource-agents
# yum install gcc autoconf automake glib2-devel;
# yum install heartbeat resource-agents cluster-glue-libs-devel;
# yum install perl-Socket6 perl-libwww-perl perl-IO-Socket-INET6 perl-MailTools;
# yum install docbook-*;
Download the resource-agents:
# wget https://github.com/ClusterLabs/resource-agents/zipball/master \
-O ClusterLabs-resource-agents-v3.9.3-47-g0bf8af1.zip
# unzip ClusterLabs-resource-agents-v3.9.3-47-g0bf8af1.zip
# cd ClusterLabs-resource-agents-v3.9.3-47-g0bf8af1/;
OR you can download with git tools:
# yum install git;
# git clone git://github.com/ClusterLabs/resource-agents
# cd resource-agents/
# ./autogen.sh
# ./configure --hlep
# ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var
--------------------------------------------------------------------------------
resource-agents configuration:
Version = 3.9.3.47-0bf8a
Build Version = :%H$
Features =
Prefix = /usr
Executables = /usr/sbin
Man pages = /usr/share/man
Libraries = /usr/lib
Header files = /usr/include
Arch-independent files = /usr/share
Documentation = /usr/share/doc/resource-agents
State information = /var
System configuration = /etc
RA state files = /var/run/resource-agents
AIS Plugins =
Libraries = -lglib-2.0
Stack Libraries =
--------------------------------------------------------------------------------
# make
# cp ldirectord/ldirectord /usr/sbin/
# cp ldirectord/init.d/ldirectord /etc/init.d/
# chmod 755 /usr/sbin/ldirectord
# chmod 755 /etc/init.d/ldirectord
# cp ldirectord/ldirectord.cf /etc/ha.d/
# vi /etc/ha.d/ldirectord.cf;
--------------------------------------------------------------------------------
checktimeout=3
checkinterval=1
logfile = "/var/log/ldirectord.log"
autoreload=yes
#quiescent=yes
quiescent=no
virtual=192.168.57.200:80
real=192.168.57.100:80 gate
real=192.168.58.100:80 gate
#real=192.168.57.100:80 masq
#real=192.168.58.100:80 masq
#real=192.168.57.100:80 ipip
#real=192.168.58.100:80 ipip
fallback=127.0.0.1:80 gate
service=http
request="healthcheck.html"
receive="OK"
virtualhost=www.testing.net
scheduler=rr
#scheduler=wrr
#scheduler=wlc
persistent=600
#netmask=255.255.255.255
protocol=tcp
checktype=negotiate
checkport=80
--------------------------------------------------------------------------------
参数说明：
gatesame as ipvsadm-gLVS/DR
ipipsame as ipvsadm-iLVS/TUN
masqsame as ipvsadm-mLVS/NAT
# vi /etc/rc.local;
--------------------------------------------------------------------------------
/sbin/ifconfig lo:0 192.168.57.200 netmask 255.255.255.255 broadcast 192.168.57.255 up
/sbin/route add -host 192.168.57.200 dev lo:0
echo 0 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
--------------------------------------------------------------------------------
────────────────────────────────────────────────────────────────────────────────
Configurate heartbeat(for DR fail-over)
────────────────────────────────────────────────────────────────────────────────
# cp /usr/share/doc/heartbeat-3.0.4/ha.cf /etc/ha.d/
# cp /usr/share/doc/heartbeat-3.0.4/authkeys /etc/ha.d/
# dd if=/dev/urandom bs=512 count=1 | openssl md5
# vi /etc/ha.d/authkeys;#(Same on both DR servers);
--------------------------------------------------------------------------------
auth 1
1 sha1 46b61a7f417d395c712d9ecf30665aaf
--------------------------------------------------------------------------------
# chmod 600 /etc/ha.d//authkeys;
# vi /etc/init.d/startvip;#(Same on both DR servers);
--------------------------------------------------------------------------------
#!/bin/bash
VIP=192.168.57.200
case $1 in
start)
IP=`/sbin/ifconfig lo:0 | grep $VIP`
if [ ! -z "$IP" ] ; then
/sbin/ifconfig lo:0 down
fi
/sbin/ifconfig eth0:0 $VIP netmask 255.255.255.255 broadcast 192.168.57.255 up
/sbin/arping -q -c 1 -w 1 -I eth0 $VIP ; echo ""
;;
stop)
IP=`/sbin/ifconfig eth0:0 | grep $VIP`
if [ ! -z "$IP" ] ; then
/sbin/ifconfig eth0:0 down
fi
echo 0 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
/sbin/ifconfig lo:0 $VIP netmask 255.255.255.255 broadcast 192.168.57.255 up
/sbin/route add -host $VIP dev lo:0
;;
status)
/sbin/ifconfig eth0:0
;;
*)
echo "Syntax incorrect. You need one of {start|stop|status}"
;;
esac
--------------------------------------------------------------------------------
chmod 755 /etc/init.d/startvip;
关于 apring 命令的说明：
/sbin/arping -q -c 1 -w 1 -I eth0 $VIP ; echo ""
如前所述，这行命令用于通知本地网络更新arp地址，因为在shell环境这个命令总是返回
大于0的结果，导致heartbeat误认为服务失败，所以我在此命令后跟了个echo命令以消除其
返回数值。请注意，使用 arping 命令不保证一定能够更新您的网络环境中的所有arp缓存
记录，在实用中请对照您的网络设备（路由器）的手册调整相关设置。
********************************************************************************
注意：在此案例中请不要使用Heartbeat资源所自带的IPAddr脚本来启动虚拟IP地址，因为
本案例设计架构所使用的本地 lo:0 含有与 eth0:0 相同的VIP地址，因此 IPAddr 脚本可
能会报错并会拒绝启动相关资源。
********************************************************************************
# vi /etc/ha.d/haresources;#(Same on both DR servers);
--------------------------------------------------------------------------------
centos6-ocfs2-1 startvip ldirectord
--------------------------------------------------------------------------------
Note: 这里不需要加入ipvsadm服务，因为现在ipvsadm服务改由ldirectord服务负责启动。
DR1# vi /etc/ha.d/ha.cf;#(on DR centos6-ocfs2-1);
--------------------------------------------------------------------------------
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
bcast eth0
ucast eth0 192.168.57.102
keepalive 2
auto_failback on
node centos6-ocfs2-1
node centos6-ocfs2-2
--------------------------------------------------------------------------------
Note: 这里的IP地址是对端node主机的IP地址。
DR2# vi /etc/ha.d/ha.cf;#(on DR centos6-ocfs2-2);
--------------------------------------------------------------------------------
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
bcast eth0
ucast eth0 192.168.57.101
keepalive 2
auto_failback on
node centos6-ocfs2-1
node centos6-ocfs2-2
--------------------------------------------------------------------------------
Note: 这里的IP地址是对端node主机的IP地址。
设置并启动服务：
# chkconfig --list ipvsadm;
# chkconfig ipvsadm off;
# chkconfig ldirectord off;
# chkconfig heartbeat on;
注意：在这里，Centos 只需要启动heartbeat,然后由heartbeat负责启动ldirectord服务，
再由ldirectord服务启动并管理ipvsadm服务。
最后，分别在 RS1 和 RS2 主机启动 heartbeat 服务：
RS1# service heartbeat start;
RS2# service heartbeat start;
观察启动结果：
# service ldirectord status;
# service ipvsadm status;
# ip addr show;