# 030. redis cluster 通过 master 水平扩容来支撑更高的读写吞吐 + 海量数据
redis cluster 模式下，不建议做物理的读写分离了，我们建议通过 master 的水平扩容，来横向扩展读写吞吐量，还有支撑更多的海量数据
比如 redis 单机假设读吞吐是 5w/s，写吞吐 2w/s，如果有 5 台 master，读吞吐可以达到总量 25w/s QPS，写可以达到 10w/s QPS
redis 单机内存不建议过大（建议值 6G、8G），因为 fork 类操作的时候很耗时，会导致请求延时的问题
只要横向扩容更多的 master 就能达到支撑 1TB 数据量。
# 加入新的 master
暂时不添加机器了（比较耗时，主要是为了演示这个功能），在 eshop-cache03 上增加一个 7007 的 redis 节点，然后把该节点加入到 redis cluster 中
搭建方式和配置文件的修改参考
# 添加节点到集群
[root@eshop-cache01 ~]# redis-trib.rb add-node 192.168.99.172:7007 192.168.99.170:7001
>>> Adding node 192.168.99.172:7007 to cluster 192.168.99.170:7001
>>> Performing Cluster Check (using node 192.168.99.170:7001)
S: 3807711e01cd28509d7ba9839e601058bf2a30cf 192.168.99.170:7001
slots: (0 slots) slave
replicates a174fe6613862db8985f82caac58bde91dfbd664
M: 9247dfb394441619da9da5b75b62b034c3f420e5 192.168.99.172:7006
slots:10923-16383 (5461 slots) master
1 additional replica(s)
M: a174fe6613862db8985f82caac58bde91dfbd664 192.168.99.171:7004
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: b0d66be3b15e117696c50a781ff24a842456733d 192.168.99.170:7002
slots: (0 slots) slave
replicates 0ff0e8ab05a8b032aeacf24e0c7fea77be3f5c55
S: cb2256b653cf5b7b3f9d1478cfa2953cc334c5bf 192.168.99.172:7005
slots: (0 slots) slave
replicates 9247dfb394441619da9da5b75b62b034c3f420e5
M: 0ff0e8ab05a8b032aeacf24e0c7fea77be3f5c55 192.168.99.171:7003
slots:5461-10922 (5462 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.99.172:7007 to make it join the cluster.
# 检查集群信息
[root@eshop-cache01 ~]# redis-trib.rb check 192.168.99.170:7001
>>> Performing Cluster Check (using node 192.168.99.170:7001)
S: 3807711e01cd28509d7ba9839e601058bf2a30cf 192.168.99.170:7001
slots: (0 slots) slave
replicates a174fe6613862db8985f82caac58bde91dfbd664
M: 9247dfb394441619da9da5b75b62b034c3f420e5 192.168.99.172:7006
slots:10923-16383 (5461 slots) master
1 additional replica(s)
M: a174fe6613862db8985f82caac58bde91dfbd664 192.168.99.171:7004
slots:0-5460 (5461 slots) master
1 additional replica(s)
M: 5d236f5ff996f0d378914b01159a8b7c5dee383d 192.168.99.172:7007
slots: (0 slots) master
0 additional replica(s)
S: b0d66be3b15e117696c50a781ff24a842456733d 192.168.99.170:7002
slots: (0 slots) slave
replicates 0ff0e8ab05a8b032aeacf24e0c7fea77be3f5c55
S: cb2256b653cf5b7b3f9d1478cfa2953cc334c5bf 192.168.99.172:7005
slots: (0 slots) slave
replicates 9247dfb394441619da9da5b75b62b034c3f420e5
M: 0ff0e8ab05a8b032aeacf24e0c7fea77be3f5c55 192.168.99.171:7003
slots:5461-10922 (5462 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
可以看到 7007 变成了一个 master，但是只有 0 个 slots ，表示此事的 7007 还不能对外服务，需要手动把执行迁移命令
# reshard 一些数据到新添加的 master 上
注：由于第一次迁移的时候输入完 done 后，就被刷屏了，记录没有保存下来，我把分配的 4096 个 slots 迁移到了 7006 上，这里再分配一次。记录下命令的反应（集群创建的时候，它会均匀分配 slots 到每个机器上的）
# 任意连接一台集群内的集群即可
[root@eshop-cache01 ~]# redis-trib.rb reshard 192.168.99.170:7001
>>> Performing Cluster Check (using node 192.168.99.170:7001)
S: 3807711e01cd28509d7ba9839e601058bf2a30cf 192.168.99.170:7001
slots: (0 slots) slave
replicates a174fe6613862db8985f82caac58bde91dfbd664
M: 9247dfb394441619da9da5b75b62b034c3f420e5 192.168.99.172:7006
slots:0-1364,5461-6826,10923-16383 (8192 slots) master
1 additional replica(s)
M: a174fe6613862db8985f82caac58bde91dfbd664 192.168.99.171:7004
slots:1365-5460 (4096 slots) master
1 additional replica(s)
M: 5d236f5ff996f0d378914b01159a8b7c5dee383d 192.168.99.172:7007
slots: (0 slots) master
0 additional replica(s)
S: b0d66be3b15e117696c50a781ff24a842456733d 192.168.99.170:7002
slots: (0 slots) slave
replicates 0ff0e8ab05a8b032aeacf24e0c7fea77be3f5c55
S: cb2256b653cf5b7b3f9d1478cfa2953cc334c5bf 192.168.99.172:7005
slots: (0 slots) slave
replicates 9247dfb394441619da9da5b75b62b034c3f420e5
M: 0ff0e8ab05a8b032aeacf24e0c7fea77be3f5c55 192.168.99.171:7003
slots:6827-10922 (4096 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
# 询问你要迁移多少个 slots 到目标机器去? 一共是 16384/4 = 4096
How many slots do you want to move (from 1 to 16384)? 4096
# 要迁移到的目标机器 id，这个id就在上面的 M/S 后面这一串
What is the receiving node ID? 5d236f5ff996f0d378914b01159a8b7c5dee383d
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
# 输入被迁移的 id，这里是要把之前的 3 台上的 node 均匀的迁移 4096 到 新机器上
Source node #1:9247dfb394441619da9da5b75b62b034c3f420e5
Source node #2:a174fe6613862db8985f82caac58bde91dfbd664
Source node #3:0ff0e8ab05a8b032aeacf24e0c7fea77be3f5c55
# 需要输入 done 命令完成输入源
Source node #4:done
... 接下来会一大串的日志信息，
Moving slot 7850 from 0ff0e8ab05a8b032aeacf24e0c7fea77be3f5c55
# 是否要继续执行集群计划
Do you want to proceed with the proposed reshard plan (yes/no)? yes
又是一大堆日志刷屏，完成后
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
报错问题
TIP
难道他们不是均匀分配的吗？对于下面这种问题，报错后，我没有回复成功， 只能把所有的 /etc/redis/ 和 /etc/redis-cluster/ 下的 conf 文件删除后， 并且重启所有的 redis 后，登录上节点手动执行 flushall 和 cluster reset 后， 才能重新创建集群，重新回复到现在这个章节的过程中来的
这里我怀疑只要把所有节点都执行 flushall 和 cluster reset 就可以吧集群打散。再重建
报错问题：在下面这种 slots 分布下，7006 有 8192 个节点，7007 0 个， 本来以为 redis 会自动迁移到 7007 上面去，指定了 3 个源节点，结果执行过程中报错了
S: 3807711e01cd28509d7ba9839e601058bf2a30cf 192.168.99.170:7001
slots: (0 slots) slave
replicates a174fe6613862db8985f82caac58bde91dfbd664
M: 9247dfb394441619da9da5b75b62b034c3f420e5 192.168.99.172:7006
slots:0-1364,5461-6826,10923-16383 (8192 slots) master
1 additional replica(s)
M: a174fe6613862db8985f82caac58bde91dfbd664 192.168.99.171:7004
slots:1365-5460 (4096 slots) master
1 additional replica(s)
M: 5d236f5ff996f0d378914b01159a8b7c5dee383d 192.168.99.172:7007
slots: (0 slots) master
0 additional replica(s)
S: b0d66be3b15e117696c50a781ff24a842456733d 192.168.99.170:7002
slots: (0 slots) slave
replicates 0ff0e8ab05a8b032aeacf24e0c7fea77be3f5c55
S: cb2256b653cf5b7b3f9d1478cfa2953cc334c5bf 192.168.99.172:7005
slots: (0 slots) slave
replicates 9247dfb394441619da9da5b75b62b034c3f420e5
M: 0ff0e8ab05a8b032aeacf24e0c7fea77be3f5c55 192.168.99.171:7003
slots:6827-10922 (4096 slots) master
[ERR] Calling MIGRATE: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
再次尝试迁移，发现还是报错
[root@eshop-cache01 ~]# redis-trib.rb reshard 192.168.99.170:7001
>>> Performing Cluster Check (using node 192.168.99.170:7001)
S: 3807711e01cd28509d7ba9839e601058bf2a30cf 192.168.99.170:7001
slots: (0 slots) slave
replicates a174fe6613862db8985f82caac58bde91dfbd664
M: 9247dfb394441619da9da5b75b62b034c3f420e5 192.168.99.172:7006
slots:6144-6826,10923-16383 (6144 slots) master
1 additional replica(s)
M: a174fe6613862db8985f82caac58bde91dfbd664 192.168.99.171:7004
slots:1860-5460 (3601 slots) master
1 additional replica(s)
M: 5d236f5ff996f0d378914b01159a8b7c5dee383d 192.168.99.172:7007
slots:0-1859,5461-6143 (2543 slots) master
0 additional replica(s)
S: b0d66be3b15e117696c50a781ff24a842456733d 192.168.99.170:7002
slots: (0 slots) slave
replicates 0ff0e8ab05a8b032aeacf24e0c7fea77be3f5c55
S: cb2256b653cf5b7b3f9d1478cfa2953cc334c5bf 192.168.99.172:7005
slots: (0 slots) slave
replicates 9247dfb394441619da9da5b75b62b034c3f420e5
M: 0ff0e8ab05a8b032aeacf24e0c7fea77be3f5c55 192.168.99.171:7003
slots:6827-10922 (4096 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
[WARNING] Node 192.168.99.171:7004 has slots in migrating state (1860).
[WARNING] Node 192.168.99.172:7007 has slots in importing state (1860).
[WARNING] The following slots are open: 1860
>>> Check slots coverage...
[OK] All 16384 slots covered.
*** Please fix your cluster problems before resharding
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
手动 kill 掉 7004 和 7007 后再次重启，发现还是有问题
[ERR] Nodes don't agree about configuration!
>>> Check for open slots...
[WARNING] Node 192.168.99.172:7007 has slots in importing state (1860).
[WARNING] The following slots are open: 1860
1
2
3
4
2
3
4
这个时候百度到了，使用 fix 来修复，但是没有成功，由于我把 /etc/redis-cluster/node-7007.conf 内容清空了。
redis-trib.rb fix 192.168.99.172:7007
1
# 添加 node 作为 slave
在 eshop-cache03 上增加一个 7008 的 redis 节点
# redis-trib.rb add-node --slave --master-id 要挂载的 masterid ，被挂载的 slave ，随便给一个集群节点即可
# 我这里是吧 7008 挂载到了 7001 上
redis-trib.rb add-node --slave --master-id b49a4562f0164bbf9c8fc9c6059e09e420ee7abf 192.168.99.172:7008 192.168.99.170:7001
1
2
3
2
3
# 删除 node
先用 resharding 将数据都移除到其他节点，确保 node 为空之后，才能执行 remove 操作
这里删除 7007 这个，现在集群中 4 个 master 都是 4096 个 slots
需要手动计算把 4096 平均分配给另外 3 个 master ，2 个分配 1365，1个分配 1366
# 手动执行 3 次，把 7007 上的 slots 分布迁移到其他 三个 master 上
redis-trib.rb reshard 192.168.99.170:7001
# 再删除 node-id，也就是协商 7007 的 id
redis-trib.rb del-node 192.168.99.170:7001 <node-id>
1
2
3
4
5
2
3
4
5
当你清空了一个 master 的 hashslot 时，redis cluster 就会自动将其 slave 挂载到其他 master 上去，这个时候就只要删除掉 master 就可以了