Spring Boot 2.3.0 新特性Redis 拓扑动态感应 4年前

Spring Boot 2.3 新特性优雅停机详解

Spring Boot 2.3 新特性分层 JAR

本篇是 spring boot v2.3 系列第三篇，来分享一下 v2.3 关于 spring data redis 的故障转移优化。

背景

关于 Redis 在生产中我们一般情况下都会选择 redis cluster 高可用架构部署，既能保证数据分片并且实现节点的故障自动转移。 基本部署拓扑如下：

创建测试集群

• 这里通过我封装的 pig4cloud/redis-cluster:4.0 镜像，即可构建一个 6 个节点的 redis cluster 测试环境。

  docker run --name redis-cluster -d -e CLUSTER_ANNOUNCE_IP=宿主机IP
  -p 7000-7005:7000-7005 -p 17000-17005:17000-17005 pig4cloud/redis-cluster:4.0

• 查看集群节点信息

  ⋊> ./redis-cli -h 172.17.0.111 -p 7000 -c 16:09:48 172.17.0.111:7000> cluster nodes 3d882206d40935beef84ff564b538d57369e4fd9 172.17.0.111:7003@17003 slave b8d24150df4a221c1045cd9a0696bd1972912d52 0 1591344590000 4 connected b8d24150df4a221c1045cd9a0696bd1972912d52 172.17.0.111:7001@17001 master - 0 1591344590513 2 connected 5461-10922 c21167a6da7f8af31d2dd612d449cdf92ad2e7e9 172.17.0.111:7005@17005 slave 810baa140db6e008a137708f09d4335f5207ede3 0 1591344591000 6 connected 810baa140db6e008a137708f09d4335f5207ede3 172.17.0.111:7000@17000 myself,master - 0 1591344590000 1 connected 0-5460 05d2f9884d350a50ac9e38f575b57f19e864e74c 172.17.0.111:7004@17004 slave b3cf24a918d96a1949f49a1d7b3a965ff9dc858c 0 1591344590011 5 connected b3cf24a918d96a1949f49a1d7b3a965ff9dc858c 172.17.0.111:7002@17002 master - 0 1591344591617 3 connected 10923-16383

应用层接入集群

• 这里使用 spring boot 2.2 演示， 默认的连接池使用 lettuce

  spring: redis: cluster: nodes: - 172.17.0.111:7000 - 172.17.0.111:7001 - 172.17.0.111:7002 - 172.17.0.111:7003 - 172.17.0.111:7004 - 172.17.0.111:7005

• 简单使用 redisTemplate 操作集群

  @RestController public class DemoController { @Autowired private RedisTemplate redisTemplate; @GetMapping("/add") public String redis() { redisTemplate.opsForValue().set("k1", "v1"); return "ok"; } }

• 调用查看日志

  ⋊> curl http://localhost:8080/add ok⏎

我们会发现操作 k1 是在 7000 节点进行操作写入

copy
[channel=0x5ff7aa8f, /172.17.0.156:50783 -> /172.17.0.111:7000, epid=0x8] write() writeAndFlush command ClusterCommand [command=AsyncCommand [type=SET, output=StatusOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command], redirections=0, maxRedirections=5]
[channel=0x5ff7aa8f, /172.17.0.156:50783 -> /172.17.0.111:7000, epid=0x8] write() done

模拟单点故障

• 关闭 7000 节点

  ./redis-cli -h 172.17.0.111 -p 7000 -c 172.17.0.111:7000> SHUTDOWN

• 查看 redis cluster 集群日志 docker logs -f redis-cluster

我们可以看到此时集群选举完毕，完成故障转移

copy
23:S 05 Jun 08:24:49.387 # Starting a failover election for epoch 7.
29:M 05 Jun 08:24:49.388 # Failover auth granted to c21167a6da7f8af31d2dd612d449cdf92ad2e7e9 for epoch 7
26:M 05 Jun 08:24:49.388 # Failover auth granted to c21167a6da7f8af31d2dd612d449cdf92ad2e7e9 for epoch 7
23:S 05 Jun 08:24:49.389 # Failover election won: I'm the new master.
23:S 05 Jun 08:24:49.389 # configEpoch set to 7 after successful failover
23:M 05 Jun 08:24:49.389 # Setting secondary replication ID to 5253748ecf5bd7ab3536058fba8cad62d2d5e825, valid up to offset: 1622. New replication ID is 21d6a0b199a1ba655c0279d9c78f9682477ac9a3
23:M 05 Jun 08:24:49.389 * Discarding previously cached master state.
23:M 05 Jun 08:24:49.390 # Cluster state changed: ok

• 此时集群状态。7005 从 slave 变更为 master

应用层日志

• 大量输出连接 7000 节点异常

  io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.17.0.111:7000 Caused by: java.net.ConnectException: Connection refused

  io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.17.0.111:7000 Caused by: java.net.ConnectException: Connection refused

  io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.17.0.111:7000 Caused by: java.net.ConnectException: Connection refused

• 再次操作 redisTemplate 会发现卡死，等待结果返回

  ⋊> curl http://localhost:8080/add

• 原因分析

此时还是操作 k1, 根据 slot 对应连接到 7000 节点，已经连接不到无限尝试重连的问题。 lettuce 客户端并未和 redis cluster 集群状态同步刷新，把宕机节点移除，完成故障转移。

集群拓扑动态感应

拓扑动态感应即客户端能够根据 redis cluster 集群的变化，动态改变客户端的节点情况，完成故障转移。

我们只需要在 spring boot 2.3.0 版本中 开启此特性即可。

copy
spring:
  redis:
    lettuce:
      cluster:
        refresh:
          adaptive: true

• 其实 lettuce 官方一直有这个功能，但 spring data redis 并未跟进，具体内容可以参考 user-content-refreshing-the-cluster-topology-view 章节

旧版本兼容

我们只需要参考 adaptive 开关打开后做了哪些事情，给自己的项目配置上 topology-view 即可

copy
@Bean
    public LettuceConnectionFactory redisConnectionFactory(RedisProperties redisProperties) {
        RedisClusterConfiguration redisClusterConfiguration = new RedisClusterConfiguration(redisProperties.getCluster().getNodes());

        // https://github.com/lettuce-io/lettuce-core/wiki/Redis-Cluster#user-content-refreshing-the-cluster-topology-view
        ClusterTopologyRefreshOptions clusterTopologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
                .enablePeriodicRefresh()
                .enableAllAdaptiveRefreshTriggers()
                .refreshPeriod(Duration.ofSeconds(5))
                .build();

        ClusterClientOptions clusterClientOptions = ClusterClientOptions.builder()
                .topologyRefreshOptions(clusterTopologyRefreshOptions).build();

        // https://github.com/lettuce-io/lettuce-core/wiki/ReadFrom-Settings
        LettuceClientConfiguration lettuceClientConfiguration = LettuceClientConfiguration.builder()
                .readFrom(ReadFrom.REPLICA_PREFERRED)
                .clientOptions(clusterClientOptions).build();

        return new LettuceConnectionFactory(redisClusterConfiguration, lettuceClientConfiguration);
    }