使用 Docker 一键搭建 Redis Cluster

2021年01月18日2021年01月18日尘缘Redis

Redis Cluster 6.0.10 已经发布了，还没有找到一键搭建 Mac 测试环境的命令。Goframe 框架的 Redis 好像还不支持 Cluster，于是准备装一个集群进行测试。

集群模式介绍

Redis Cluster是社区版推出的Redis分布式集群解决方案，主要解决Redis分布式方面的需求，比如，当遇到单机内存，并发和流量等瓶颈的时候，Redis Cluster能起到很好的负载均衡的目的。

Redis Cluster着眼于提高并发量。

群集至少需要3主3从，且每个实例使用不同的配置文件。

在redis-cluster架构中，redis-master节点一般用于接收读写，而redis-slave节点则一般只用于备份，其与对应的master拥有相同的slot集合，若某个redis-master意外失效，则再将其对应的slave进行升级为临时redis-master。

在redis的官方文档中，对redis-cluster架构上，有这样的说明：在cluster架构下，默认的，一般redis-master用于接收读写，而redis-slave则用于备份，当有请求是在向slave发起时，会直接重定向到对应key所在的master来处理。但如果不介意读取的是redis-cluster中有可能过期的数据并且对写请求不感兴趣时，则亦可通过readonly命令，将slave设置成可读，然后通过slave获取相关的key，达到读写分离。具体可以参阅redis官方文档等相关内容

自己整理了一下，命令如下：

建立配置模板

port ${PORT}
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
cluster-announce-ip ${IP}
cluster-announce-port ${PORT}
cluster-announce-bus-port 1${PORT}
appendonly yes

将上述内容保存为 redis-cluster.tmpl，参数名如下

# redis端口
port ${PORT}
# 关闭保护模式
protected-mode no
# 开启集群
cluster-enabled yes
# 集群节点配置
cluster-config-file nodes.conf
# 超时
cluster-node-timeout 5000
# 集群节点IP host模式为宿主机IP
cluster-announce-ip 192.168.124.5
# 集群节点端口 7001 - 7006
cluster-announce-port ${PORT}
cluster-announce-bus-port 1${PORT}
# 开启 appendonly 备份模式
appendonly yes
# 每秒钟备份
appendfsync everysec
# 对aof文件进行压缩时，是否执行同步操作
no-appendfsync-on-rewrite no
# 当目前aof文件大小超过上一次重写时的aof文件大小的100%时会再次进行重写
auto-aof-rewrite-percentage 100
# 重写前AOF文件的大小最小值 默认 64mb
auto-aof-rewrite-min-size 64mb

创建 Docker 虚拟网络

docker network create redis
docker network inspect redis

创建一键启动脚本

p=$(pwd)
ip=$(ifconfig en0 | grep "inet " | awk '{print $2}')

shell="/usr/local/bin/redis-cli --cluster create "

for port in $(seq 8010 8015); do
    echo "create config file for " $port

    rm -rf ./${port}
    mkdir -p ./${port}/conf  \
    && IP=${ip} PORT=${port} envsubst < ./redis-cluster.tmpl > ./${port}/conf/redis.conf \
    && mkdir -p ./${port}/data;

    shell="$shell $ip:$port"
done

for port in $(seq 8010 8015); do
    echo "create pod for " $port

    docker stop redis-${port} >> /dev/null
    docker rm redis-${port} >> /dev/null

    docker run -it -d -p ${port}:${port} -p 1${port}:1${port} \
    --privileged=true -v $p/${port}/conf/redis.conf:/usr/local/etc/redis/redis.conf \
    --privileged=true -v $p/${port}/data:/data \
    --restart always --name redis-${port} --net redis \
    --sysctl net.core.somaxconn=1024 redis redis-server /usr/local/etc/redis/redis.conf;
done

echo "you should run this script to starting cluster"
echo "$shell" 
docker exec -it redis-8010 bash

将上述内容保存为 start.sh即可运行，其中 nc0 需要修改为你本机的宿主网卡，如 WIFI 或者有线网卡。

执行结果如下：

create config file for  8010
create config file for  8011
create config file for  8012
create config file for  8013
create config file for  8014
create config file for  8015
create pod for  8010
cc68f32f836ac845c5515269f9371c50ca62e4d4aaf28c7479f2eb2a046cb605
create pod for  8011
018fbc1b557a01aa147eb5ccdde73c2d355071e5447739998b7b605e1a5e0d80
create pod for  8012
6c87629efdde55afc0a093f1ea180204a9475db5071bd45df1b2a534096727ad
create pod for  8013
f2a83070e11f5fbabd8728d39a076e5befb2bdc7efd2bedb8c294bf3fc80957b
create pod for  8014
6320501af030000e383aac1731ca9d908e184c1057a801c51f6676eba78b45cd
create pod for  8015
3fe4f8f4f3acf5b3ba44cd50ef01056a89281a1e046b44e04f9c4c13996197d0
you should run this script to starting cluster
/usr/local/bin/redis-cli --cluster-replicas 1 --cluster create  192.168.17.117:8010 192.168.17.117:8011 192.168.17.117:8012 192.168.17.117:8013 192.168.17.117:8014 192.168.17.117:8015
117:8013 192.168.17.117:8014 192.168.17.117:8015 192.168.17.117:8012 192.168.17.1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.17.117:8014 to 192.168.17.117:8010
Adding replica 192.168.17.117:8015 to 192.168.17.117:8011
Adding replica 192.168.17.117:8013 to 192.168.17.117:8012
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: f79cf8e7f6578e4fcf605629492d24ee5a28c342 192.168.17.117:8010
   slots:[0-5460] (5461 slots) master
M: 63278b067faa1b7a247cfed1267acbcd4b5febd1 192.168.17.117:8011
   slots:[5461-10922] (5462 slots) master
M: 1ed31b17403eb1739fe19e76eba9f143dbd6db64 192.168.17.117:8012
   slots:[10923-16383] (5461 slots) master
S: 1b93607a971564a3f743d3862556af95a4d9f3d0 192.168.17.117:8013
   replicates 1ed31b17403eb1739fe19e76eba9f143dbd6db64
S: 27b0febdaa85e3f1fa268eea0236eacbf0d79e4a 192.168.17.117:8014
   replicates f79cf8e7f6578e4fcf605629492d24ee5a28c342
S: 082198a46fee0e941e446b5e00d994a5d569f975 192.168.17.117:8015
   replicates 63278b067faa1b7a247cfed1267acbcd4b5febd1
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join

>>> Performing Cluster Check (using node 192.168.17.117:8010)
M: f79cf8e7f6578e4fcf605629492d24ee5a28c342 192.168.17.117:8010
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 1b93607a971564a3f743d3862556af95a4d9f3d0 192.168.17.117:8013
   slots: (0 slots) slave
   replicates 1ed31b17403eb1739fe19e76eba9f143dbd6db64
M: 1ed31b17403eb1739fe19e76eba9f143dbd6db64 192.168.17.117:8012
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 082198a46fee0e941e446b5e00d994a5d569f975 192.168.17.117:8015
   slots: (0 slots) slave
   replicates 63278b067faa1b7a247cfed1267acbcd4b5febd1
M: 63278b067faa1b7a247cfed1267acbcd4b5febd1 192.168.17.117:8011
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 27b0febdaa85e3f1fa268eea0236eacbf0d79e4a 192.168.17.117:8014
   slots: (0 slots) slave
   replicates f79cf8e7f6578e4fcf605629492d24ee5a28c342
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
root@cc68f32f836a:/data#

请注意 /usr/local/bin/redis-cli --cluster-replicas 1 --cluster create 192.168.17.117:8010 192.168.17.117:8011 192.168.17.117:8012 192.168.17.117:8013 192.168.17.117:8014 192.168.17.117:8015 117:8013 192.168.17.117:8014 192.168.17.117:8015 192.168.17.117:8012 192.168.17.1 命令是创建 cluster 的命令，有个--cluster-replicas 1 参数，回车以后输入 Yes即可完成创建。

集群模式下的操作方法

自动跳转

由于Redis Cluster会根据key进行hash运算，然后将key分散到不同slots，name的hash运算结果在不同节点上的slots中。读写的时候则会进行提示，Key 已经 moved 到指点的节点

root@cc68f32f836a:/data# redis-cli -p 8010 get aaa
(error) MOVED 10439 192.168.17.117:8011

通过命令行读写的时候，可以加参数 -c 自动跟随

root@cc68f32f836a:/data# redis-cli -p 8010 -c get aaa
"ok"

查看集群状态

root@cc68f32f836a:/data# redis-cli -p 8010 -c cluster nodes
1b93607a971564a3f743d3862556af95a4d9f3d0 192.168.17.117:8013@18013 slave 1ed31b17403eb1739fe19e76eba9f143dbd6db64 0 1610963851866 3 connected
1ed31b17403eb1739fe19e76eba9f143dbd6db64 192.168.17.117:8012@18012 master - 0 1610963851000 3 connected 10923-16383
082198a46fee0e941e446b5e00d994a5d569f975 192.168.17.117:8015@18015 slave 63278b067faa1b7a247cfed1267acbcd4b5febd1 0 1610963850545 2 connected
f79cf8e7f6578e4fcf605629492d24ee5a28c342 192.168.17.117:8010@18010 myself,master - 0 1610963851000 1 connected 0-5460
63278b067faa1b7a247cfed1267acbcd4b5febd1 192.168.17.117:8011@18011 master - 0 1610963851558 2 connected 5461-10922
27b0febdaa85e3f1fa268eea0236eacbf0d79e4a 192.168.17.117:8014@18014 slave f79cf8e7f6578e4fcf605629492d24ee5a28c342 0 1610963851152 1 connected

查看slots分片

root@cc68f32f836a:/data# redis-cli -p 8010 -c cluster slots
1) 1) (integer) 10923
   2) (integer) 16383
   3) 1) "192.168.17.117"
      2) (integer) 8012
      3) "1ed31b17403eb1739fe19e76eba9f143dbd6db64"
   4) 1) "192.168.17.117"
      2) (integer) 8013
      3) "1b93607a971564a3f743d3862556af95a4d9f3d0"
2) 1) (integer) 0
   2) (integer) 5460
   3) 1) "192.168.17.117"
      2) (integer) 8010
      3) "f79cf8e7f6578e4fcf605629492d24ee5a28c342"
   4) 1) "192.168.17.117"
      2) (integer) 8014
      3) "27b0febdaa85e3f1fa268eea0236eacbf0d79e4a"
3) 1) (integer) 5461
   2) (integer) 10922
   3) 1) "192.168.17.117"
      2) (integer) 8011
      3) "63278b067faa1b7a247cfed1267acbcd4b5febd1"
   4) 1) "192.168.17.117"
      2) (integer) 8015
      3) "082198a46fee0e941e446b5e00d994a5d569f975"

查看集群信息

root@cc68f32f836a:/data# redis-cli -p 8010 -c cluster info
root@cc68f32f836a:/data# redis-cli -p 8010 -c cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:683
cluster_stats_messages_pong_sent:692
cluster_stats_messages_sent:1375
cluster_stats_messages_ping_received:687
cluster_stats_messages_pong_received:683
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:1375

读写分离

在redis的官方文档中，对redis-cluster架构上，有这样的说明：在cluster架构下，默认的，一般redis-master用于接收读写，而redis-slave则用于备份，当有请求是在向slave发起时，会直接重定向到对应key所在的master来处理。

但如果不介意读取的是redis-cluster中有可能过期的数据并且对写请求不感兴趣时，则亦可通过readonly命令，将slave设置成可读，然后通过slave获取相关的key，达到读写分离。

容灾

在创建集群时，如果启用 --cluster-replicas 1 参数，集群将会自动在 master 异常时提升一个 slave 为 master。