Redis集群搭建的几种方式

前面使用了哨兵集群搭建单个分片高可用的结构,实现了主从复制和故障转移,但是没有实现分布式。接下来使用Redis集群搭建,实现哨兵集群没有实现的分布式高可用。

redis-cluster搭建需求

下面准备6个redis节点,搭建如图所示的三主三从分布式redis集群,实现两两互联,以master8001为例,它可以连接另外两个主节点,以及三个从节点。搭建过程可以使用原生redis命令,也可以使用ruby自带的脚本(需要先安装ruby)。

Redis集群搭建的几种方式

ruby脚本辅助搭建

使用ruby脚本辅助安装,即使用redis根目录/src/redis-trib.rb脚本来完成,需要先安装ruby。

-rwxrwxr-x. 1 root root   60852 Sep 21  2017 redis-trib.rb

(1)安装ruby,这里解压了压缩包,进行编译和安装。但是安装完只是最基本的一步,后面还需要对ruby进行配置,比较麻烦。

# 解压
[ /home/software]#  tar -zxvf  ruby-2.3.1.tar.gz
# 检查
[ /home/software/ruby-2.3.1]#  . /configure
# 编译,安装
[ /home/software/ruby-2.3.1]# make && make install
# 省略具体安装日志,安装完成后使用ruby -v如果能查看到版本号,就安装ok
[ /home/software/ruby-2.3.1]#  ruby -v
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_4-linux]

(2)先准备一个redis-cluster.conf启动加载文件,基本配置参考前面博文https://www.cnblogs.com/youngchaolin/p/11983705.html#_label0 ,主要是bind设置不绑定ip、protected mode设置为no、daemonize设置为yes、其他还有pid、logfile、rdb等,保持原设置不变。

(3)配置appendonly为yes,启动aof持久化方式,这是除了rdb外redis提供的第二种持久化方式。

# AOF and RDB persistence can be enabled at the same time without problems.
# If the AOF is enabled on startup Redis will load the AOF, that is the file
# with the better durability guarantees.
#
# Please check http://redis.io/topics/persistence for more information.
# 配置第二种持久化方式aof
appendonly yes
 597 # The name of the append only file (default: "appendonly.aof")
# 标识aof持久化文件名,以端口号区分
appendfilename "appendonly6379.aof"

其中rdb和aof两种持久化的主要区别为:

a.rdb保存的是具体的key-value数据,如name-messi,aof保存的是操作记录,如set name messi。

b.因为aof保存的数据更全,redis启动默认加载的是aof,rdb启动可以单独开启。

c.rdb持久化频率是按照持久化策略来的,容易造成数据丢失,而aof是每秒保存一次数据,数据不容易丢失。

d.如果对数据的可靠性要求高,使用aof持久化,如果需要一定的数据恢复能力,但是又不需要很高的可靠性,就选择rdb。

(4)开启集群模式,如果不开启无法计算槽道号。

# Normal Redis instances can‘t be part of a Redis Cluster; only nodes that are
# started as cluster nodes can. In order to start a Redis instance as a
# cluster node enable the cluster support uncommenting the following:
# 需设置yes,否则无法计算槽道号,无法创建集群
cluster-enabled yes

(5)需要开启集群节点状态记录文件,这个文件会自动更新,每个redis节点都需要一个这样的文件。

# Every cluster node has a cluster configuration file. This file is not
# intended to be edited by hand. It is created and updated by Redis nodes.
# Every Redis Cluster node requires a different cluster configuration file.
# Make sure that instances running in the same system do not have
# overlapping cluster configuration file names.
# 开启集群节点状态记录文件
cluster-config-file nodes-6379.conf

(6)将上面的配置文件保存,接下来复制到6个不同的文件夹下,其中文件夹的名字就是redis节点的端口号,根据上图准备8000、8001、8002、8003、8004和8005六个文件夹。复制完成后,修改对应端口号,如8000文件夹的配置文件,将里面全是6379的部分替换为8000,其他类似修改即可。

# 创建目录
[ /home/software/redis-3.2.11]# mkdir 8000 8001 8002 8003 8004 8005
# 复制文件
[ /home/software/redis-3.2.11]# cp redis.conf 8000/
[ /home/software/redis-3.2.11]# cp redis.conf 8001/
[ /home/software/redis-3.2.11]# cp redis.conf 8002/
[ /home/software/redis-3.2.11]# cp redis.conf 8003/
[ /home/software/redis-3.2.11]# cp redis.conf 8004/
[ /home/software/redis-3.2.11]# cp redis.conf 8005/
# 递归查看是否copy成功
[ /home/software/redis-3.2.11]# ls -R 800*
8000:
redis.conf

8001:
redis.conf

8002:
redis.conf

8003:
redis.conf

8004:
redis.conf

8005:
redis.conf# 修改端口略

(7)启动各个节点,发现均是cluster的方式启动,这是创建集群的基础,另外查看集群为fail,是因为槽道还没分配的原因,且每个节点cluster nodes查看集群节点信息都只能找到自己一个节点,两两互联还没有建立。

# 加载修改后的配置文件启动各个节点
[ /home/software/redis-3.2.11]# redis-server 8000/redis.conf
[ /home/software/redis-3.2.11]# redis-server 8001/redis.conf
[ /home/software/redis-3.2.11]# redis-server 8002/redis.conf
[ /home/software/redis-3.2.11]# redis-server 8003/redis.conf
[ /home/software/redis-3.2.11]# redis-server 8004/redis.conf
[ /home/software/redis-3.2.11]# redis-server 8005/redis.conf
# 查看是否以cluster方式启动,ok
[ /home/software/redis-3.2.11]# ps -ef|grep redis
root     16028     1  0 15:43 ?        00:00:00 redis-server *:8000 [cluster]
root     16033     1  0 15:43 ?        00:00:00 redis-server *:8001 [cluster]
root     16037     1  0 15:43 ?        00:00:00 redis-server *:8002 [cluster]
root     16041     1  0 15:43 ?        00:00:00 redis-server *:8003 [cluster]
root     16045     1  0 15:43 ?        00:00:00 redis-server *:8004 [cluster]
root     16049     1  0 15:43 ?        00:00:00 redis-server *:8005 [cluster]
root     16053  1409  0 15:43 pts/0    00:00:00 grep redis
# 登录一个节点,查看集群信息
[ /home/software/redis-3.2.11]# redis-cli -c -p 8000
127.0.0.1:8000> cluster info
cluster_state:fail # 只要有一个槽道没分配,就是fail
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1 # 只能发现自己一个节点,两两互联暂未建立
cluster_size:0
cluster_current_epoch:0 # 集群的纪元
cluster_my_epoch:0
cluster_stats_messages_sent:0 # 两两互联发送数据量
cluster_stats_messages_received:0 # 两两互联接受数据量
# 查看集群节点,发现只有自己一个
127.0.0.1:8000> cluster nodes
dc0e2e16888426089de31466ff398679eec81b86 :8000 myself,master - 0 0 0 connected

(8)启动src/redis-trib.rb命令后想查看帮忙信息发现报错。这是正常的,这意味着踏入了一个大坑,接下来的需要填坑了,参考文末博文,完成了报错解决。

# 报错
/usr/local/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require‘: cannot load such file -- redis (LoadError)
    from /usr/local/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require‘
    from src/redis-trib.rb:25:in `<main>‘

(9)上面报错是需要redis的gem包来安装ruby和redis的接口,先安装gem包,网址https://rubygems.org/gems/redis/,选择对应的版本下载,我这里是选择3.2.1版本,下载好后上传到了redis根目录/src目录下了,具体位置放哪不清楚,放src下是可以验证通过的。

-rw-r--r--. 1 root root   73728 Dec 12 16:51 redis-3.2.1.gem

(10)准备使用gem install redis命令安装接口,发现报错,需要安装zlib,网址http://www.zlib.net,本次选择了1.2.11版本。

ERROR:  Loading command: install (LoadError)
    cannot load such file -- zlib
ERROR:  While executing gem ... (NoMethodError)
    undefined method `invoke_with_build_args‘ for nil:NilClass

(11)解压zlib压缩包后,进入解压目录,先指定安装目录,后编译安装zlib。

# 指定安装目录/usr/local/zlib
[ /home/software/zlib-1.2.11]# ./configure --prefix=/usr/local/zlib
Checking for gcc...
Checking for shared library support...
Building shared library libz.so.1.2.11 with gcc.
Checking for size_t... Yes.
Checking for off64_t... Yes.
Checking for fseeko... Yes.
Checking for strerror... Yes.
Checking for unistd.h... Yes.
Checking for stdarg.h... Yes.
Checking whether to use vs[n]printf() or s[n]printf()... using vs[n]printf().
Checking for vsnprintf() in stdio.h... Yes.
Checking for return value of vsnprintf()... Yes.
Checking for attribute(visibility) support... Yes.
You have new mail in /var/spool/mail/root
# 编译后安装
[ /home/software/zlib-1.2.11]# make && make install
...省略
chmod 644 /usr/local/zlib/include/zlib.h /usr/local/zlib/include/zconf.h

(12)进入ruby根目录/ext/zlib,先使用ruby extconf.rb命令,执行完后会在当前目录下生成Makefile文件。

[ /home/software/ruby-2.3.1/ext/zlib]# ll
total 892
-rw-r--r--. 1 root root    146 Apr 16  2013 depend
-rw-r--r--. 1 root root   1447 Dec 16  2015 extconf.rb# 使用ruby extconf.rb命令后会生成一个Makefile文件
-rw-r--r--. 1 root root   7656 Dec 12 17:14 Makefile
-rw-r--r--. 1 root root   7090 Dec 12 17:14 mkmf.log
-rw-r--r--. 1 root root 124168 Dec 19  2015 zlib.c
-rw-r--r--. 1 root root 418576 Dec 12 17:18 zlib.o
-rwxr-xr-x. 1 root root 336032 Dec 12 17:18 zlib.so# 安装时需添加配置信息,指定zlib目录下的include和lib,zlib目录就是前面编译指定安装的目录
[ /home/software/ruby-2.3.1/ext/zlib]# ruby extconf.rb --with-zlib-include=/usr/local/zlib/include/ --with-zlib-lib=/usr/local/zlib/lib
# 省略 [ /home/software/ruby-2.3.1/ext/zlib]# make && make install # 省略

接下来编译,本次编译顺利通过了,如果没有通过出现如下报错,还需要修改Makefile内容。

make: *** No rule to make target `/include/ruby.h‘, needed by `zlib.o‘.  Stop

修改Makefile内容,将$(top_srcdir)整个换成绝对路径,我这里没有报错,因此没有修改。

zlib.o: $(top_srcdir)/include/ruby.h 改成:zlib.o: ../../include/ruby.h

最后编译通过会提示如下内容。

[ /home/software/ruby-2.3.1/ext/zlib]# make && make install
compiling zlib.c
linking shared-object zlib.so
/usr/bin/install -c -m 0755 zlib.so /usr/local/lib/ruby/site_ruby/2.4.0/x86_64-linux

(13)继续安装openssl,如果不安装切换到redis目录使用gem install redis会报‘‘unable to require openssl...‘‘的报错,因此需要继续像安装zlib一样安装openssl。

网址https://www.openssl.org/source/,解压后进入安装目录,执行以下命令,编译安装执行时间本机比较长。

[ /home/software/openssl-1.0.2t]# ./config -fPIC --prefix=/usr/local/openssl enable-shared
[ /home/software/openssl-1.0.2t]# ./config -t
# 这里会编译很久
[ /home/software/openssl-1.0.2t]# $ make && make install

(14)进入ruby根目录/ext/openssl下,执行ruby extconf.rb命令,也会在当前目录下生成Makefile文件,虽然编译和安装,光荣的报错了。

[ /home/software/ruby-2.3.1/ext/openssl]# ruby extconf.rb  --with-openssl-include=/usr/local/openssl/include/ --with-openssl-lib=/usr/local/openssl/lib
[ /home/software/ruby-2.3.1/ext/openssl]# make && make install

(15)类似上面安装zlib,也将Makefile文件的\$(top_srcdir)全部改成"../..",这里在使用了":%s/\${top_srcdir}/..\/../g"进行了全局替换。再次编译安装就通过了。

Redis集群搭建的几种方式

(16)到这里,就差不多可以了,切换目录到redis根目录,然后执行gem install redis,再次报错。没辙了,网上找了一种土办法,先让安装成功。

[ /home/software/redis-3.2.11]# gem install redis
ERROR:  Could not find a valid gem ‘redis‘ (>= 0), here is why:
          Unable to download data from https://rubygems.org/ - SSL_connect returned=1 errno=0 state=error: certificate verify failed (https://api.rubygems.org/specs.4.8.gz)

临时办法,大概意思让https连接改成http连接。

[ /home/software/redis-3.2.11]# gem sources -r https://rubygems.org
source https://rubygems.org not present in cache
[ /home/software/redis-3.2.11]# gem sources -a http://rubygems.org
https://rubygems.org is recommended for security over http://rubygems.org

Do you want to add this insecure source? [yn]  y
http://rubygems.org added to sources
You have new mail in /var/spool/mail/root

继续执行命令,终于ok了,可谓一波三折,这个方法只能作为参考,换一个redis或ruby又不知道会发生什么报错。

# 继续
[ /home/software/redis-3.2.11/src]# gem install redis
Successfully installed redis-3.2.1
Parsing documentation for redis-3.2.1
Installing ri documentation for redis-3.2.1
Done installing documentation for redis after 0 seconds #成功了
WARNING:  Unable to pull data from ‘https://rubygems.org/‘: SSL_connect returned=1 errno=0 state=error: certificate verify failed (https://api.rubygems.org/specs.4.8.gz)
1 gem installed
You have new mail in /var/spool/mail/root
# 试运行redis-trib.rb脚本,终于ok了
[ /home/software/redis-3.2.11]# src/redis-trib.rb
Usage: redis-trib <command> <options> <arguments ...>

  create          host1:port1 ... hostN:portN
                  --replicas <arg>
  check           host:port
  info            host:port
  fix             host:port
                  --timeout <arg>
  reshard         host:port
                  --from <arg>
                  --to <arg>
                  --slots <arg>
                  --yes
                  --timeout <arg>
                  --pipeline <arg>
  rebalance       host:port
                  --weight <arg>
                  --auto-weights
                  --use-empty-masters
                  --timeout <arg>
...省略

(17)前面准备好了后,就可以使用ruby脚本来开始构建集群了。上面启动后的redis节点都是各自为政互不联系,接下来先将8000、8001、8002作为主节点连起来,并分配槽道号,本次使用默认分配。

# 使用create
[ /home/software/redis-3.2.11]# src/redis-trib.rb create 192.168.200.140:8000 192.168.200.140:8001 192.168.200.140:8002
>>> Creating cluster
>>> Performing hash slots allocation on 3 nodes...
Using 3 masters:
192.168.200.140:8000
192.168.200.140:8001
192.168.200.140:8002
M: dc0e2e16888426089de31466ff398679eec81b86 192.168.200.140:8000
   slots:0-5460 (5461 slots) master
M: aa79ace502e5369236b62ed61c0eb43733ddcbde 192.168.200.140:8001
   slots:5461-10922 (5462 slots) master
M: 719d1dd412faf6b4e1eb348c65c329a262e393e1 192.168.200.140:8002
   slots:10923-16383 (5461 slots) master
Can I set the above configuration? (type ‘yes‘ to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
>>> Performing Cluster Check (using node 192.168.200.140:8000)
M: dc0e2e16888426089de31466ff398679eec81b86 192.168.200.140:8000
   slots:0-5460 (5461 slots) master
   0 additional replica(s)
M: aa79ace502e5369236b62ed61c0eb43733ddcbde 192.168.200.140:8001
   slots:5461-10922 (5462 slots) master
   0 additional replica(s)
M: 719d1dd412faf6b4e1eb348c65c329a262e393e1 192.168.200.140:8002
   slots:10923-16383 (5461 slots) master
   0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
You have new mail in /var/spool/mail/root
[ /home/software/redis-3.2.11]# redis-cli -c -p 8000
# 查看发现集群状态已变成ok,说明槽道没有一个没分配
127.0.0.1:8000> cluster into
(error) ERR Wrong CLUSTER subcommand or number of arguments
127.0.0.1:8000> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:3
cluster_size:3
cluster_current_epoch:3
cluster_my_epoch:1
cluster_stats_messages_sent:68
cluster_stats_messages_received:68
# 三个主节点都分配了槽道
127.0.0.1:8000> cluster nodes# 节点信息# 第一个为节点id,用40位的十六进制表示# 第二个为节点ip+端口# 第三个为角色# 第四个为主节点id,没有主节点就用‘-‘表示# 第五个是与操作相关的时间戳# 第六个代表序号# 第七个代表连接状态# 第八个代表槽道号区间
aa79ace502e5369236b62ed61c0eb43733ddcbde 192.168.200.140:8001 master - 0 1576145625541 2 connected 5461-10922
719d1dd412faf6b4e1eb348c65c329a262e393e1 192.168.200.140:8002 master - 0 1576145623522 3 connected 10923-16383
dc0e2e16888426089de31466ff398679eec81b86 192.168.200.140:8000 myself,master - 0 0 1 connected 0-5460

(18)将8003挂接到8000,8004挂接到8001,8005挂接到8002成为从节点,使用add node命令,这里需要用到两个选项,一个是--salve,代表新加的节点是从,一个是--master-id,指定需要添加的主是谁,最后分别添加slaveHost:slavePort existHost:existPort,后面的节点可以是集群中任意一个加入集群的节点。

# 以添加8003为例 ,注意主节点可以是任意一个主节点,不一定是当前从的主节点[ /home/software/redis-3.2.11]# src/redis-trib.rb add-node --slave --master-id dc0e2e16888426089de31466ff398679eec81b86 192.168.200.140:8003 192.168.200.140:8000
>>> Adding node 192.168.200.140:8003 to cluster 192.168.200.140:8000
>>> Performing Cluster Check (using node 192.168.200.140:8000)
M: dc0e2e16888426089de31466ff398679eec81b86 192.168.200.140:8000
   slots:0-5460 (5461 slots) master
   0 additional replica(s)
M: aa79ace502e5369236b62ed61c0eb43733ddcbde 192.168.200.140:8001
   slots:5461-10922 (5462 slots) master
   0 additional replica(s)
M: 719d1dd412faf6b4e1eb348c65c329a262e393e1 192.168.200.140:8002
   slots:10923-16383 (5461 slots) master
   0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.# 底层还是调用redis原生命令cluster meet
>>> Send CLUSTER MEET to node 192.168.200.140:8003 to make it join the cluster.
Waiting for the cluster to join.
>>> Configure node as replica of 192.168.200.140:8000.
[OK] New node added correctly.
You have new mail in /var/spool/mail/root

登录集群客户端查看集群信息,发现搭建成功,实现了三主三从redis集群的搭建。

127.0.0.1:8000> cluster nodes
aa79ace502e5369236b62ed61c0eb43733ddcbde 192.168.200.140:8001 master - 0 1576152242486 2 connected 5461-10922
dc0e2e16888426089de31466ff398679eec81b86 192.168.200.140:8000 myself,master - 0 0 1 connected 0-5460
824a999176c12eb6ebd944b75944fd97c65b579d 192.168.200.140:8005 slave 719d1dd412faf6b4e1eb348c65c329a262e393e1 0 1576152244002 3 connected
312accb9be27c48c107a19ab501a75a8c5d321c6 192.168.200.140:8004 slave aa79ace502e5369236b62ed61c0eb43733ddcbde 0 1576152239461 2 connected
ada1903bcb5744900916151dd385936ceb6ce049 192.168.200.140:8003 slave dc0e2e16888426089de31466ff398679eec81b86 0 1576152243496 1 connected
719d1dd412faf6b4e1eb348c65c329a262e393e1 192.168.200.140:8002 master - 0 1576152244507 3 connected 10923-16383

ruby脚本简化搭建

可以看出,这个搭建比较繁琐, 如果解决了ruby的报错问题,只需要一个命令就可以完成三主三从redis集群的搭建,下面记录一下。

(1)停止redis所有进程。

[ /home/software/redis-3.2.11]# redis-cli -c -p 8000 shutdown
[ /home/software/redis-3.2.11]# redis-cli -c -p 8001 shutdown
[ /home/software/redis-3.2.11]# redis-cli -c -p 8002 shutdown
[ /home/software/redis-3.2.11]# redis-cli -c -p 8003 shutdown
[ /home/software/redis-3.2.11]# redis-cli -c -p 8004 shutdown
[ /home/software/redis-3.2.11]# redis-cli -c -p 8005 shutdown
[ /home/software/redis-3.2.11]# ps -ef|grep redis
root      3657 18054  0 20:13 pts/2    00:00:00 grep redis

(2)删除rdb和aof持久化文件,删除集群节点状态记录文件。

[ /home/software/redis-3.2.11]# rm -rf appendonly800*
[ /home/software/redis-3.2.11]# rm -rf dump800*
[ /home/software/redis-3.2.11]# rm -rf nodes-800*

(3)再次启动6个redis节点。

# 启动
[ /home/software/redis-3.2.11]# redis-server 8000/redis.conf
[ /home/software/redis-3.2.11]# redis-server 8001/redis.conf
[ /home/software/redis-3.2.11]# redis-server 8002/redis.conf
[ /home/software/redis-3.2.11]# redis-server 8003/redis.conf
[ /home/software/redis-3.2.11]# redis-server 8004/redis.conf
[ /home/software/redis-3.2.11]# redis-server 8005/redis.conf
[ /home/software/redis-3.2.11]# redis-cli -c -p 8000
# 还没分配槽道,redis集群状态fail
127.0.0.1:8000> cluster info
cluster_state:fail
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1
cluster_size:0
cluster_current_epoch:0
cluster_my_epoch:0
cluster_stats_messages_sent:0
cluster_stats_messages_received:0

(4)使用ruby脚本src/redis-trib.rb create --replicas 1 六个的节点ip:端口,这一条命令就可以自动搭建了三主三从。

# 一个命令
[ /home/software/redis-3.2.11]# src/redis-trib.rb create --replicas 1 192.168.200.140:8000 192.168.200.140:8001 192.168.200.140:8002 192.168.200.140:8003 192.168.200.140:8004 192.168.200.140:8005
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.200.140:8000
192.168.200.140:8001
192.168.200.140:8002
Adding replica 192.168.200.140:8003 to 192.168.200.140:8000
Adding replica 192.168.200.140:8004 to 192.168.200.140:8001
Adding replica 192.168.200.140:8005 to 192.168.200.140:8002
M: 6537ac1cb42f209600ea20b1d6310e7b789257d4 192.168.200.140:8000
   slots:0-5460 (5461 slots) master
M: 5da79737cf493085724a689bdb5bd878c9121c07 192.168.200.140:8001
   slots:5461-10922 (5462 slots) master
M: 49f40898d2d18cc162b00e5f69593f0b91af5766 192.168.200.140:8002
   slots:10923-16383 (5461 slots) master
S: a92a3eb2cfff488e0af6ca07d9b612f2f3595c0a 192.168.200.140:8003
   replicates 6537ac1cb42f209600ea20b1d6310e7b789257d4
S: ea41d1e7a41e5c5b19baa8e6cbac15b7d4d2bdb8 192.168.200.140:8004
   replicates 5da79737cf493085724a689bdb5bd878c9121c07
S: bede8e5c11fe947cbaf0e435d822057a7b0fe414 192.168.200.140:8005
   replicates 49f40898d2d18cc162b00e5f69593f0b91af5766
Can I set the above configuration? (type ‘yes‘ to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join...
>>> Performing Cluster Check (using node 192.168.200.140:8000)
M: 6537ac1cb42f209600ea20b1d6310e7b789257d4 192.168.200.140:8000
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
M: 49f40898d2d18cc162b00e5f69593f0b91af5766 192.168.200.140:8002
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
S: bede8e5c11fe947cbaf0e435d822057a7b0fe414 192.168.200.140:8005
   slots: (0 slots) slave
   replicates 49f40898d2d18cc162b00e5f69593f0b91af5766
M: 5da79737cf493085724a689bdb5bd878c9121c07 192.168.200.140:8001
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: a92a3eb2cfff488e0af6ca07d9b612f2f3595c0a 192.168.200.140:8003
   slots: (0 slots) slave
   replicates 6537ac1cb42f209600ea20b1d6310e7b789257d4
S: ea41d1e7a41e5c5b19baa8e6cbac15b7d4d2bdb8 192.168.200.140:8004
   slots: (0 slots) slave
   replicates 5da79737cf493085724a689bdb5bd878c9121c07
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
You have new mail in /var/spool/mail/root
[ /home/software/redis-3.2.11]# redis-cli -c -p 8000
# 结果一样
127.0.0.1:8000> cluster nodes
49f40898d2d18cc162b00e5f69593f0b91af5766 192.168.200.140:8002 master - 0 1576153283083 3 connected 10923-16383
bede8e5c11fe947cbaf0e435d822057a7b0fe414 192.168.200.140:8005 slave 49f40898d2d18cc162b00e5f69593f0b91af5766 0 1576153285101 6 connected
5da79737cf493085724a689bdb5bd878c9121c07 192.168.200.140:8001 master - 0 1576153286109 2 connected 5461-10922
6537ac1cb42f209600ea20b1d6310e7b789257d4 192.168.200.140:8000 myself,master - 0 0 1 connected 0-5460
a92a3eb2cfff488e0af6ca07d9b612f2f3595c0a 192.168.200.140:8003 slave 6537ac1cb42f209600ea20b1d6310e7b789257d4 0 1576153283585 4 connected
ea41d1e7a41e5c5b19baa8e6cbac15b7d4d2bdb8 192.168.200.140:8004 slave 5da79737cf493085724a689bdb5bd878c9121c07 0 1576153284091 5 connected

redis cluster原生命令搭建

以上两种方法,都是依赖ruby脚本,其实它还是基于redis cluster的命令来搭建完成,使用这种方式搭建能更好的理解搭建中经历了什么。

(1)提前准备6个启动的节点,前提也需删除持久化文件,以及节点状态记录文件,跟上面一样。

(2)登录任何一个节点,和其他5个进行meet,这里使用8000的客户端来完成,meet完后任意一个节点都认为自己是主节点。

# 使用cluster meet命令192.168.200.140:8000> CLUSTER MEET 192.168.200.140 8001
OK
192.168.200.140:8000> CLUSTER MEET 192.168.200.140 8002
OK
192.168.200.140:8000> CLUSTER MEET 192.168.200.140 8003
OK
192.168.200.140:8000> CLUSTER MEET 192.168.200.140 8004
OK
192.168.200.140:8000> CLUSTER MEET 192.168.200.140 8005
OK# 登录任何一个节点查看,都能得到如下信息
192.168.200.140:8000> cluster nodes
fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 master - 0 1576155518818 0 connected
ced952bc6a47c38756bbc48d85a5f601db620b79 192.168.200.140:8000 myself,master - 0 0 4 connected
2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 master - 0 1576155517810 5 connected
231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 master - 0 1576155514284 1 connected
7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576155515791 2 connected
2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 master - 0 1576155516800 3 connected

(3)给8000,8001和8002分配槽道号,让其作为主节点。分配槽道号的命令为cluster addslots 槽道号...。分配完成后不管登录哪个节点都能看到槽道号信息,

# 槽道号之间有多个需要用空格隔开
[ /home/software/redis-3.2.11]# redis-cli -c -p 8000 -h 192.168.200.140
192.168.200.140:8000> CLUSTER ADDSLOTS 0 1 2 3 4 5
OK# 可以看到8000的节点分配了0-5这6个槽道号
192.168.200.140:8000> cluster nodes
fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 master - 0 1576155898158 0 connected
ced952bc6a47c38756bbc48d85a5f601db620b79 192.168.200.140:8000 myself,master - 0 0 4 connected 0-5
2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 master - 0 1576155896137 5 connected
231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 master - 0 1576155894119 1 connected
7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576155897147 2 connected
2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 master - 0 1576155895128 3 connected

一个个写来分配会比较费力气,可以写一个shell脚本来批量分配槽道号,使用for循环来实现。

shell脚本

#分配槽道号的脚本
#!/bin/bash
# 8000节点
for slot in {6..5460}
do redis-cli -c -p 8000 -h 192.168.200.140 cluster addslots $slot
done
# 8001节点
for slot in {5461..10922}
do redis-cli -c -p 8001 -h 192.168.200.140 cluster addslots $slot
done
# 8002节点
for slot in {10923..16383}
do redis-cli -c -p 8002 -h 192.168.200.140 cluster addslots $slot
done
# 执行完打印出来结果
echo "分配槽道号结束"

运行脚本执行完结果,ok。

Redis集群搭建的几种方式

登录一个节点查看集群槽道信息,发现跟分配的预想一样。

[ /home/software/redis-3.2.11]# redis-cli -c -p 8001 -h 192.168.200.140
192.168.200.140:8001> cluster nodes# 8001结果,与上面脚本一样结果
231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 myself,master - 0 0 1 connected 5461-10922 
fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 master - 0 1576157229386 0 connected
ced952bc6a47c38756bbc48d85a5f601db620b79 192.168.200.140:8000 master - 0 1576157228376 4 connected 0-5460
7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576157229386 2 connected 10923-16383
2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 master - 0 1576157227368 3 connected
2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 master - 0 1576157230393 5 connected

(4)将8003、8004和8005角色转换为从节点,使用cluster replicate 主节点id 命令。最后查看集群信息,发现成功的分配了槽道号。

[ /home/software/redis-3.2.11]# redis-cli -c -p 8003 -h 192.168.200.140
# 成为8000的从节点
192.168.200.140:8003> cluster replicate ced952bc6a47c38756bbc48d85a5f601db620b79
OK
192.168.200.140:8003> quit
[ /home/software/redis-3.2.11]# redis-cli -c -p 8004 -h 192.168.200.140
# 成为8001的从节点
192.168.200.140:8004> cluster replicate 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83
OK
192.168.200.140:8004> quit
You have new mail in /var/spool/mail/root
[ /home/software/redis-3.2.11]# redis-cli -c -p 8005 -h 192.168.200.140
# 成为8002的从节点
192.168.200.140:8005> cluster replicate 7ce388bde879f686fc3c8491175397ca20405565
OK
# 结果ok
192.168.200.140:8005> cluster nodes
2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 myself,slave 7ce388bde879f686fc3c8491175397ca20405565 0 0 5 connected
7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576157733223 2 connected 10923-16383
2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 slave ced952bc6a47c38756bbc48d85a5f601db620b79 0 1576157732216 4 connected
ced952bc6a47c38756bbc48d85a5f601db620b79 192.168.200.140:8000 master - 0 1576157735238 4 connected 0-5460
231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 master - 0 1576157736242 1 connected 5461-10922
fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 slave 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 0 1576157734230 1 connected

最后集群状态为ok,槽道全部分配完成,形成了三主三从的结构。

192.168.200.140:8005> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0# 6个节点
cluster_known_nodes:6
# 3个主cluster_size:3
cluster_current_epoch:5
cluster_my_epoch:2
cluster_stats_messages_sent:5093
cluster_stats_messages_received:5093

redis cluster高可用

现在在8000节点上保存数据,然后将8000的主节点宕机,8003从节点会顶替上来成为主节点,并获取到8000的槽道信息。

[ /home/software/redis-3.2.11]# redis-cli -c -p 8000 -h 192.168.200.140
192.168.200.140:8000> set age 28
OK
192.168.200.140:8000> get age
"28"
192.168.200.140:8000> quit
[ /home/software/redis-3.2.11]# redis-cli -c -p 8003 -h 192.168.200.140
192.168.200.140:8003> get age
-> Redirected to slot [741] located at 192.168.200.140:8000
"28"
192.168.200.140:8000> shutdown
not connected> quit
You have new mail in /var/spool/mail/root
[ /home/software/redis-3.2.11]# redis-cli -c -p 8003 -h 192.168.200.140
# 8000宕机后,8003暂时还是从
192.168.200.140:8003> cluster nodes
fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 slave 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 0 1576158134892 1 connected
2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 myself,slave ced952bc6a47c38756bbc48d85a5f601db620b79 0 0 3 connected
ced952bc6a47c38756bbc48d85a5f601db620b79 192.168.200.140:8000 master - 1576158120865 1576158119755 4 disconnected 0-5460
2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 slave 7ce388bde879f686fc3c8491175397ca20405565 0 1576158132873 5 connected
231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 master - 0 1576158133883 1 connected 5461-10922
7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576158131864 2 connected 10923-16383
# 过一会,8003变为主,并且获得槽道信息
192.168.200.140:8003> cluster nodes
fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 slave 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 0 1576158149019 1 connected
2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 myself,master - 0 0 6 connected 0-5460
ced952bc6a47c38756bbc48d85a5f601db620b79 192.168.200.140:8000 master,fail - 1576158120865 1576158119755 4 disconnected
2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 slave 7ce388bde879f686fc3c8491175397ca20405565 0 1576158148008 5 connected
231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 master - 0 1576158146999 1 connected 5461-10922
7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576158150028 2 connected 10923-16383
# 数据也备份了
192.168.200.140:8003> get age
"28"

redis cluster添加和删除节点

再补充添加和删除节点的操作,关于删除节点,它只能删除从节点或没有槽道管理权的节点,由于8000已经没有了槽道管理权,将8000重启后,下面使用ruby脚本删除,命令为del-node ip:port 节点id。

# 确认8000开启[ /home/software/redis-3.2.11]# ps -ef|grep redis
root      4170     1  0 20:51 ?        00:00:13 redis-server *:8001 [cluster]
root      4174     1  0 20:51 ?        00:00:11 redis-server *:8002 [cluster]
root      4178     1  0 20:51 ?        00:00:08 redis-server *:8003 [cluster]
root      4182     1  0 20:51 ?        00:00:08 redis-server *:8004 [cluster]
root      4186     1  0 20:51 ?        00:00:08 redis-server *:8005 [cluster]
root      7231     1  0 22:09 ?        00:00:00 redis-server *:8000 [cluster]
root      7298 18054  0 22:13 pts/2    00:00:00 grep redis# 删除8000
[ /home/software/redis-3.2.11]# src/redis-trib.rb del-node 192.168.200.140:8000 94f634b2d364d560d3f8fb6e88fa874cf2b493f6
>>> Removing node 94f634b2d364d560d3f8fb6e88fa874cf2b493f6 from cluster 192.168.200.140:8000
>>> Sending CLUSTER FORGET messages to the cluster...# 关闭节点
>>> SHUTDOWN the node.
You have new mail in /var/spool/mail/root

发现删除节点后,还自动将节点关闭了,因为8000节点的状态记录文件nodes-8000.conf还记录着8000是连接状态,但是其他节点的这个文件已经没了8000的信息,为了集群数据的不混乱,需要将8000立即关闭。

既然删除了,尝试启动8000后添加到集群,需要使用add-node  newHost:newPort existHost:existPort命令,类似添加从节点命令只是没有选项。发现启动8000后再添加进集群会报错。

# 确认开启8000
[ /home/software/redis-3.2.11]# ps -ef|grep redis
root      4170     1  0 20:51 ?        00:00:14 redis-server *:8001 [cluster]
root      4174     1  0 20:51 ?        00:00:11 redis-server *:8002 [cluster]
root      4178     1  0 20:51 ?        00:00:08 redis-server *:8003 [cluster]
root      4182     1  0 20:51 ?        00:00:08 redis-server *:8004 [cluster]
root      4186     1  0 20:51 ?        00:00:08 redis-server *:8005 [cluster]
root      7400     1  0 22:21 ?        00:00:00 redis-server *:8000 [cluster]
root      7422 18054  0 22:22 pts/2    00:00:00 grep redis
You have new mail in /var/spool/mail/root
# 添加8000到集群
[ /home/software/redis-3.2.11]# src/redis-trib.rb add-node 192.168.200.140:8000 192.168.200.140:8001
>>> Adding node 192.168.200.140:8000 to cluster 192.168.200.140:8001
>>> Performing Cluster Check (using node 192.168.200.140:8001)
M: 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004
   slots: (0 slots) slave
   replicates 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83
M: 7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
M: 2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003
   slots:0-5460 (5461 slots) master
   0 additional replica(s)
S: 2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005
   slots: (0 slots) slave
   replicates 7ce388bde879f686fc3c8491175397ca20405565
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
# 提示报错,提示8000可能有数据,或者8000里有其他集群的信息
[ERR] Node 192.168.200.140:8000 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

这里删除了8000的rdb和aof持久化文件,以及状态文件后,重启8000节点再次尝试添加进去集群发现成功,但是添加进去的节点为没有槽道管理权的主节点。

[ /home/software/redis-3.2.11]# src/redis-trib.rb add-node 192.168.200.140:8000 192.168.200.140:8001
>>> Adding node 192.168.200.140:8000 to cluster 192.168.200.140:8001
>>> Performing Cluster Check (using node 192.168.200.140:8001)
M: 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004
   slots: (0 slots) slave
   replicates 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83
M: 7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
M: 2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003
   slots:0-5460 (5461 slots) master
   0 additional replica(s)
S: 2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005
   slots: (0 slots) slave
   replicates 7ce388bde879f686fc3c8491175397ca20405565
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.200.140:8000 to make it join the cluster.
[OK] New node added correctly.

8000的信息

fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 slave 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 0 1576161337438 1 connected
# 没有槽道管理权的主节点
c41dbe9595ae83725d1322b032736fd198b26c49 192.168.200.140:8000 myself,master - 0 0 0 connected
2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 slave 7ce388bde879f686fc3c8491175397ca20405565 0 1576161336432 2 connected
231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 master - 0 1576161336432 1 connected 5461-10922
2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 master - 0 1576161339456 6 connected 0-5460
7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576161338448 2 connected 10923-16383

以上是redis-cluster集群搭建的基本知识,后面继续补充槽道知识。

参考博文:

(1)《Redis设计与实践》 

(2)https://blog.csdn.net/qq_26710443/article/details/82724268 ruby安装

(3)https://www.jianshu.com/p/c38369097448 

(4)https://www.cnblogs.com/xuliangxing/p/7146868.html ruby脚本不能执行解决,报错主要参考文

(5)https://blog.csdn.net/ck3207/article/details/90404952 openssl无法编译

相关推荐