[FAQ] Jedis使用过程中踩过的那些坑

4. 一个大坑:若实例化 JedisShardInfo 时不设置节点名称(name属性),那么当Redis节点列表的顺序发生变化时,会发生“键 rehash 现象

使用BTrace追踪redis.clients.util.Sharded的实时状态,验证“Jedis分片机制的一致性哈希算法”实现;

发现一个致命坑:若JedisShardInfo不设置节点名称(name属性),那么当Redis节点列表的顺序发生变化时,会发生“键 rehash 现象”。见Sharded的initialize(...)方法实现:

(I) this.algo.hash("SHARD-" + i + "-NODE-" + n)

【缺点】 大坑:将节点的顺序索引i作为hash的一部分! 当节点顺序被无意识地调整了,会触发”键 rehash 现象”,那就杯具啦!("因节点顺序调整而引发rehash"的问题)

(II) this.algo.hash(shardInfo.getName() + "*" + shardInfo.getWeight() + n)

【优点】 这样设计避免了上面"因节点顺序调整而引发rehash"的问题。

【缺点】 坑:"节点名称+权重"必须是唯一的,否则节点会出现重叠覆盖! 同时,"节点名称+权重"必须不能被中途改变!

(III) 节点IP:端口号+编号

Memcached Java Client,就是采用这种策略。

【缺点】 因机房迁移等原因,可能导致节点IP发生改变!

(IIII) 唯一节点名称+编号

较好地一致性hash策略是:唯一节点名称+编号,不要考虑权重因素!

long hash = algo.hash(shardInfo.getName() + "*" + n)

所以,在配置Redis服务列表时,必须要设置节点逻辑名称(name属性)

redis.server.list=192.168.6.35:6379:Shard-01,192.168.6.36:6379:Shard-02,192.168.6.37:6379:Shard-03,192.168.6.38:6379:Shard-04

相关代码如下所示:

public class Sharded<R, S extends ShardInfo<R>> {

  public static final int DEFAULT_WEIGHT = 1;
  private TreeMap<Long, S> nodes;
  private final Hashing algo;
  private final Map<ShardInfo<R>, R> resources = new LinkedHashMap<ShardInfo<R>, R>();

  public Sharded(List<S> shards) {
    this(shards, Hashing.MURMUR_HASH); // MD5 is really not good as we works with 64-bits not 128
  }

  public Sharded(List<S> shards, Hashing algo) {
    this.algo = algo;
    initialize(shards);
  }

  private void initialize(List<S> shards) {
    nodes = new TreeMap<Long, S>();

    for (int i = 0; i != shards.size(); ++i) {
      final S shardInfo = shards.get(i);
      if (shardInfo.getName() == null) for (int n = 0; n < 160 * shardInfo.getWeight(); n++) {
        nodes.put(this.algo.hash("SHARD-" + i + "-NODE-" + n), shardInfo);
      }
      else for (int n = 0; n < 160 * shardInfo.getWeight(); n++) {
        nodes.put(this.algo.hash(shardInfo.getName() + "*" + shardInfo.getWeight() + n), shardInfo);
      }
      resources.put(shardInfo, shardInfo.createResource());
    }
  }

  ...

}

3. "Redis客户端连接数一直降不下来"的问题

这个问题发生有两方面的原因:

  1. 未正确使用对象池的空闲队列行为LIFO“后进先出”栈方式)
  2. 关闭集群链接时异常导致连接泄漏”问题(见本文的第一个问题)

具体分析过程,详见《[线上问题] "Redis客户端连接数一直降不下来"的问题解决》。

2. Jedis “Socket读取超时”导致“返回值类型错误

异常信息如下所示:

[2015-02-07 09:17:47] WARN  c.f.f.b.s.r.i.CustomShardedJedisFactory -quit jedis connection for server fail: xxx.xxx.xxx.xxx:xxx
java.lang.ClassCastException: java.lang.Long cannot be cast to [B   (强制类型转换异常)
	at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:181) ~[jedis-2.6.2.jar:na]
	at redis.clients.jedis.BinaryJedis.quit(BinaryJedis.java:136) ~[jedis-2.6.2.jar:na]
	at cn.fraudmetrix.forseti.biz.service.redis.impl.CustomShardedJedisFactory.destroyObject(CustomShardedJedisFactory.java:116) ~[forseti-biz-service-1.0-SNAPSHOT.jar:na]
	at org.apache.commons.pool2.impl.GenericObjectPool.destroy(GenericObjectPool.java:848) [commons-pool2-2.0.jar:2.0]
	at org.apache.commons.pool2.impl.GenericObjectPool.invalidateObject(GenericObjectPool.java:626) [commons-pool2-2.0.jar:2.0]
	at redis.clients.util.Pool.returnBrokenResourceObject(Pool.java:83) [jedis-2.6.2.jar:na]
	at cn.fraudmetrix.forseti.biz.service.redis.impl.CustomShardedJedisPool.returnBrokenResource(CustomShardedJedisPool.java:121) [forseti-biz-service-1.0-SNAPSHOT.jar:na]
	at cn.fraudmetrix.forseti.biz.service.redis.impl.RedisServiceImpl.zadd(RedisServiceImpl.java:337) [forseti-biz-service-1.0-SNAPSHOT.jar:na]
	at cn.fraudmetrix.forseti.biz.service.redis.impl.RedisServiceImpl.zadd(RedisServiceImpl.java:319) [forseti-biz-service-1.0-SNAPSHOT.jar:na]
	...
[2015-02-07 09:17:47] ERROR c.f.f.b.s.r.i.RedisServiceImpl -'zadd' key fail, key: xxx, score: xxx, member: xxx

[2015-02-07 09:17:47] ERROR c.f.f.b.s.r.i.RedisServiceImpl -java.net.SocketTimeoutException: Read timed out

redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out    (Socket读取超时异常)
	at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:201) ~[jedis-2.6.2.jar:na]     ('limit = in.read(buf);' at java.io.InputStream.read(InputStream.java:100) - 这里出现阻塞导致"Socket读取超时"!)
	at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.6.2.jar:na]
	at redis.clients.jedis.Protocol.process(Protocol.java:128) ~[jedis-2.6.2.jar:na]
	at redis.clients.jedis.Protocol.read(Protocol.java:192) ~[jedis-2.6.2.jar:na]
	at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:282) ~[jedis-2.6.2.jar:na]
	at redis.clients.jedis.Connection.getIntegerReply(Connection.java:207) ~[jedis-2.6.2.jar:na]
	at redis.clients.jedis.Jedis.zadd(Jedis.java:1293) ~[jedis-2.6.2.jar:na]
	at redis.clients.jedis.ShardedJedis.zadd(ShardedJedis.java:364) ~[jedis-2.6.2.jar:na]
	at cn.fraudmetrix.forseti.biz.service.redis.impl.RedisServiceImpl.zadd(RedisServiceImpl.java:328) [forseti-biz-service-1.0-SNAPSHOT.jar:na]
	at cn.fraudmetrix.forseti.biz.service.redis.impl.RedisServiceImpl.zadd(RedisServiceImpl.java:319) [forseti-biz-service-1.0-SNAPSHOT.jar:na]
	...
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.7.0_51]
	at java.net.SocketInputStream.read(SocketInputStream.java:152) ~[na:1.7.0_51]
	at java.net.SocketInputStream.read(SocketInputStream.java:122) ~[na:1.7.0_51]
	at java.net.SocketInputStream.read(SocketInputStream.java:108) ~[na:1.7.0_51]
	at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:195) ~[jedis-2.6.2.jar:na]
	... 38 common frames omitted
 

从异常信息来看,首先是在'zadd'操作时出现"Socket读取超时异常",具体异常信息"JedisConnectionException: java.net.SocketTimeoutException: Read timed out"。

出现异常后,会销毁这个阻塞的Jedis连接池对象(CustomShardedJedisPool.returnBrokenResource(CustomShardedJedisPool.java:121)),但在请求Redis服务端关闭连接时,出现"强制类型转换异常",具体异常信息"ClassCastException: java.lang.Long cannot be cast to [B"。

这个问题已经有前辈遇到过了,其解释:

查看 Jedis 源码发现它的Connection中对网络输出流做了一个封装(RedisInputStream),其中自建了一个buffer。当发生异常的时候,这个buffer里还残存着上次没有发送或者发送不完整的命令。这个时候没有做处理,直接将该连接返回到连接池,那么重用该连接执行下次命令的时候,就会将上次没有发送的命令一起发送过去,所以才会出现上面的错误“返回值类型不对”

所以,正确的写法应该是:在发送异常的时候,销毁这个连接,不能再重用! 

参考自:

  • jedis 用连接池时超时返回值类型错误

1. CustomShardedJedisFactory.destroyObject(PooledObject<ShardedJedis> pooledShardedJedis) 存在“客户端连接泄露”问题

异常信息如下所示:

[2015-01-28 15:33:51] ERROR c.f.f.b.s.r.i.RedisServiceImpl -ShardedJedis close fail   

redis.clients.jedis.exceptions.JedisException: Could not return the resource to the pool 
at redis.clients.util.Pool.returnBrokenResourceObject(Pool.java:85) ~[jedis-2.6.2.jar:na]       
at cn.fraudmetrix.forseti.biz.service.redis.impl.CustomShardedJedisPool.returnBrokenResource(CustomShardedJedisPool.java:120) ~[forseti-biz-service-1.0-SNAPSHOT.jar:na]       
at cn.fraudmetrix.forseti.biz.service.redis.impl.CustomShardedJedisPool.returnBrokenResource(CustomShardedJedisPool.java:26) ~[forseti-biz-service-1.0-SNAPSHOT.jar:na]       
at redis.clients.jedis.ShardedJedis.close(ShardedJedis.java:638) ~[jedis-2.6.2.jar:na]       
at cn.fraudmetrix.forseti.biz.service.redis.impl.RedisServiceImpl.close(RedisServiceImpl.java:90) [forseti-biz-service-1.0-SNAPSHOT.jar:na]       
at cn.fraudmetrix.forseti.biz.service.redis.impl.RedisServiceImpl.zadd(RedisServiceImpl.java:380) [forseti-biz-service-1.0-SNAPSHOT.jar:na]       
at cn.fraudmetrix.forseti.biz.service.redis.impl.RedisServiceImpl.zadd(RedisServiceImpl.java:346) [forseti-biz-service-1.0-SNAPSHOT.jar:na]       
...    
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_51]       
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_51]       
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]       
Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to [B 
at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:181) ~[jedis-2.6.2.jar:na]       
at redis.clients.jedis.BinaryJedis.quit(BinaryJedis.java:136) ~[jedis-2.6.2.jar:na]       
at redis.clients.jedis.BinaryShardedJedis.disconnect(BinaryShardedJedis.java:35) ~[jedis-2.6.2.jar:na]
at cn.fraudmetrix.forseti.biz.service.redis.impl.CustomShardedJedisFactory.destroyObject(CustomShardedJedisFactory.java:106) ~[forseti-biz-service-1.0-SNAPSHOT.jar:na]       
at org.apache.commons.pool2.impl.GenericObjectPool.destroy(GenericObjectPool.java:848) ~[commons-pool2-2.0.jar:2.0]       
at org.apache.commons.pool2.impl.GenericObjectPool.invalidateObject(GenericObjectPool.java:626) ~[commons-pool2-2.0.jar:2.0]       
at redis.clients.util.Pool.returnBrokenResourceObject(Pool.java:83) ~[jedis-2.6.2.jar:na]       
... 37 common frames omitted

从异常信息来看,是由于应用程序无法捕获运行时的强制类型转换异常(“java.lang.ClassCastException: java.lang.Long cannot be cast to [B”)导致关闭操作异常中断,问题的根源代码位于“BinaryShardedJedis.disconnect(BinaryShardedJedis.java:35)

CustomShardedJedisFactory.destroyObject(CustomShardedJedisFactory.java:106)”。

原实现代码只捕获了 JedisConnectionException 异常,如下所示:

public void destroyObject(PooledObject<ShardedJedis> pooledShardedJedis) throws Exception {
        final ShardedJedis shardedJedis = pooledShardedJedis.getObject();

        shardedJedis.disconnect(); // "链接资源"无法被释放,存在泄露
    }
public void disconnect() {
    for (Jedis jedis : getAllShards()) {
      try {
        jedis.quit();
      } catch (JedisConnectionException e) {
        // ignore the exception node, so that all other normal nodes can release all connections.
      }
      try {
        jedis.disconnect();
      } catch (JedisConnectionException e) {
        // ignore the exception node, so that all other normal nodes can release all connections.
      }
    }
  }

修复后代码捕获了所有的 Exception,就不存在释放链接时由于异常未捕获而导致链接释放中断。如下所示:

public void destroyObject(PooledObject<ShardedJedis> pooledShardedJedis) throws Exception {
        final ShardedJedis shardedJedis = pooledShardedJedis.getObject();

        // shardedJedis.disconnect(); // "链接资源"无法被释放,存在泄露
        for (Jedis jedis : shardedJedis.getAllShards()) {
            try {
                // 1. 请求服务端关闭连接
                jedis.quit();
            } catch (Exception e) {
                // ignore the exception node, so that all other normal nodes can release all connections.

                // java.lang.ClassCastException: java.lang.Long cannot be cast to [B
                // (zadd/zcard 返回 long 类型,而 quit 返回 string 类型。从这里看,上一次的请求结果并未读取)
                logger.warn("quit jedis connection for server fail: " + toServerString(jedis), e);
            }

            try {
                // 2. 客户端主动关闭连接
                jedis.disconnect();
            } catch (Exception e) {
                // ignore the exception node, so that all other normal nodes can release all connections.

                logger.warn("disconnect jedis connection fail: " + toServerString(jedis), e);
            }
        }
    }

相关推荐