Jedis异常处理方式

Question

我用Jedis的SharedJedisPool，配置为 MaxActive=500，MaxIdle=100，MaxWait=2000，testOnBorrow=true。redis服务器是Linux CentOS 6.0 内存98G，redis本身所占内存1.9G，Maxclients=655328，Linux的fs.file-max=9876450。服务器机构是 一主两从，主实例只负责写和持久化，两个从实例负责提供客户端读操作。主从在不同的机器，两个从实例在同一台机器上，主从实例在同一局域网。客户端其实也 是服务端程序，为40个Tomcat的实例，每20个访问一个从实例，客户端和redis服务不在同一个机器但是在同一个局域网。当有大量并发请求 redis客户端服务时，Jedis经常报错：

    redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool

    at redis.clients.util.Pool.getResource(Pool.java:22)

    …………………………

    Caused by: java.util.NoSuchElementException: Could not create a validated object, cause: ValidateObject failed

           at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1233)

           at redis.clients.util.Pool.getResource(Pool.java:20)

           ... 21 more

或者

    redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool

    at redis.clients.util.Pool.getResource(Pool.java:22)

    …………………………

    Caused by: java.util.NoSuchElementException:
Timeout waiting for idle object

           at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1233)

           at redis.clients.util.Pool.getResource(Pool.java:20)

           ... 21 more

其实发生这个错误的时候redis实例的connected_clients不足10000，大概就4000左右。使用redis-cli也连不上redis服务，报错：

    Could not connect to Redis at 127.0.0.1:6381: Connection timed out

使用netstat -anlpt|grep 127.0.0.1:6381查看连接到redis服务的tcp连接，发现如下问题：

     tcp        0      0 192.168.0.87:6381           192.168.0.144:48063         ESTABLISHED 15328/redis-server

tcp       14      0 192.168.0.87:6381           192.168.0.16:60948          ESTABLISHED 15328/redis-server

tcp        0      0 192.168.0.87:6381           192.168.0.144:48811         ESTABLISHED 15328/redis-server

tcp       55      0 192.168.0.87:6381           192.168.0.144:55022         ESTABLISHED 15328/redis-server

tcp       47      0 192.168.0.87:6381           192.168.0.144:23939         ESTABLISHED 15328/redis-server

tcp        0      0 192.168.0.87:6381           192.168.0.144:12101         ESTABLISHED 15328/redis-server

tcp       28      0 192.168.0.87:6381           192.168.0.144:45477         ESTABLISHED -

tcp       14      0 192.168.0.87:6381           192.168.0.144:47559         ESTABLISHED -

tcp        0      0 192.168.0.87:6381           192.168.0.16:2219           ESTABLISHED 15328/redis-server

tcp        0      0 192.168.0.87:6381           192.168.0.144:9343          ESTABLISHED 15328/redis-server

tcp       14      0 192.168.0.87:6381           192.168.0.144:41640         ESTABLISHED 15328/redis-server

tcp        0      0 192.168.0.87:6381           192.168.0.144:17647         ESTABLISHED 15328/redis-server

tcp       34      0 192.168.0.87:6381           192.168.0.144:55616         ESTABLISHED 15328/redis-server

tcp       34      0 192.168.0.87:6381           192.168.0.16:61066          ESTABLISHED 15328/redis-server

tcp        0      0 192.168.0.87:6381           192.168.0.144:12193         ESTABLISHED 15328/redis-server

tcp        0      0 192.168.0.87:6381           192.168.0.16:60982          ESTABLISHED 15328/redis-server

tcp        0      0 192.168.0.87:6381           192.168.0.144:11833         ESTABLISHED 15328/redis-server

tcp        0      0 192.168.0.87:6381           192.168.0.144:60907         ESTABLISHED 15328/redis-server

tcp       34      0 192.168.0.87:6381           192.168.0.144:59617         ESTABLISHED 15328/redis-server

tcp       14      0 192.168.0.87:6381           192.168.0.144:9831          ESTABLISHED 15328/redis-server

tcp        0      0 192.168.0.87:6381           192.168.0.144:58487         ESTABLISHED 15328/redis-server

tcp       34      0 192.168.0.87:6381           192.168.0.144:44767         ESTABLISHED 15328/redis-server

tcp       55      0 192.168.0.87:6381           192.168.0.144:10469         ESTABLISHED 15328/redis-server

tcp       14      0 192.168.0.87:6381           192.168.0.144:54119         ESTABLISHED 15328/redis-server

tcp        0      0 192.168.0.87:6381           192.168.0.144:10207         ESTABLISHED 15328/redis-server

tcp        0      0 192.168.0.87:6381           192.168.0.16:56750          ESTABLISHED 15328/redis-server

tcp       34      0 192.168.0.87:6381           192.168.0.16:61138          ESTABLISHED 15328/redis-server

tcp       34      0 192.168.0.87:6381           192.168.0.16:56897          ESTABLISHED 15328/redis-server

tcp        0      0 192.168.0.87:6381           192.168.0.16:56862          ESTABLISHED 15328/redis-server

tcp        0      0 192.168.0.87:6381           192.168.0.144:16855         ESTABLISHED 15328/redis-server

tcp       34      0 192.168.0.87:6381           192.168.0.144:60538         ESTABLISHED 15328/redis-server

tcp        0      0 192.168.0.87:6381           192.168.0.144:8619          ESTABLISHED 15328/redis-server

tcp       55      0 192.168.0.87:6381           192.168.0.16:61052          ESTABLISHED 15328/redis-server

tcp        0      0 192.168.0.87:6381           192.168.0.16:2391           ESTABLISHED 15328/redis-server

tcp        0      0 192.168.0.87:6381           192.168.0.16:2248           ESTABLISHED 15328/redis-server

tcp        0      0 192.168.0.87:6381           192.168.0.144:17951         ESTABLISHED 15328/redis-server

tcp        0      0 192.168.0.87:6381           192.168.0.16:61067          ESTABLISHED 15328/redis-server

tcp       28      0 192.168.0.87:6381           192.168.0.16:30130          ESTABLISHED -

tcp        0      0 192.168.0.87:6381           192.168.0.144:10339         ESTABLISHED 15328/redis-server

tcp        0      0 192.168.0.87:6381           192.168.0.144:9826          ESTABLISHED 15328/redis-server

tcp        0      0 192.168.0.87:6381           192.168.0.16:56731          ESTABLISHED 15328/redis-server

tcp        0      0 192.168.0.87:6381           192.168.0.144:55006         ESTABLISHED 15328/redis-server

tcp       34      0 192.168.0.87:6381           192.168.0.144:11421         ESTABLISHED 15328/redis-server

tcp        0      0 192.168.0.87:6381           192.168.0.16:30215          ESTABLISHED -

tcp        0      0 192.168.0.87:6381           192.168.0.144:30921         ESTABLISHED 15328/redis-server

tcp        0   1343 192.168.0.87:6381           192.168.0.144:8715          ESTABLISHED 15328/redis-server

tcp        0      0 192.168.0.87:6381           192.168.0.144:24553         ESTABLISHED 15328/redis-server

tcp       34      0 192.168.0.87:6381           192.168.0.144:9021          ESTABLISHED 15328/redis-server

tcp       34      0 192.168.0.87:6381           192.168.0.144:16338         ESTABLISHED 15328/redis-server

tcp       14      0 192.168.0.87:6381           192.168.0.144:41356         ESTABLISHED 15328/redis-server

有些连接的Recv_Q和Send_Q不能很快的清空，redis服务的client list列表如下：

addr=192.168.0.7:44002 fd=452 name= age=222 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping

addr=192.168.0.7:44005 fd=459 name= age=222 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping

addr=192.168.0.7:44006 fd=461 name= age=222 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=sismember

addr=192.168.0.7:43593 fd=599 name= age=522 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get

addr=192.168.0.7:43596 fd=602 name= age=522 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping

addr=192.168.0.7:44199 fd=737 name= age=206 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get

addr=192.168.0.7:43609 fd=619 name= age=522 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=1378 oll=0 omem=0 events=rw cmd=hgetall

addr=192.168.0.7:43610 fd=620 name= age=522 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=4 oll=0 omem=0 events=rw cmd=sismember

addr=192.168.0.7:43611 fd=621 name= age=522 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping

addr=192.168.0.7:43615 fd=627 name= age=522 idle=14 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get

addr=192.168.0.7:43247 fd=231 name= age=662 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping

addr=192.168.0.7:43252 fd=238 name= age=662 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping

addr=192.168.0.7:42598 fd=494 name= age=984 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=118 oll=0 omem=0 events=rw cmd=hgetall

addr=192.168.0.7:43828 fd=390 name= age=333 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=hgetall

addr=192.168.0.7:43829 fd=391 name= age=333 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping

addr=192.168.0.7:43979 fd=406 name= age=222 idle=8 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sismember

addr=192.168.0.7:43980 fd=407 name= age=222 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping

addr=192.168.0.7:43983 fd=417 name= age=222 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping

addr=192.168.0.7:43986 fd=422 name= age=222 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get

addr=192.168.0.7:43989 fd=429 name= age=222 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get

addr=192.168.0.7:43990 fd=430 name= age=222 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping

addr=192.168.0.7:43992 fd=435 name= age=222 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall

addr=192.168.0.7:43708 fd=58 name= age=344 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping

addr=192.168.0.7:43570 fd=575 name= age=522 idle=14 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sismember

addr=192.168.0.7:43572 fd=577 name= age=522 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping

addr=192.168.0.7:43573 fd=578 name= age=522 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall

addr=192.168.0.7:43686 fd=29 name= age=377 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get

addr=192.168.0.7:43230 fd=202 name= age=662 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping

addr=192.168.0.7:43236 fd=211 name= age=662 idle=14 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sismember

addr=192.168.0.7:43744 fd=160 name= age=334 idle=14 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get

addr=192.168.0.7:43745 fd=161 name= age=334 idle=14 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get

1. 如果谁在使用redis的时候出现过同类的问题，如果解决？
2. redis服务端和Jedis客户端如何配置才合适？

Answer 1

建议去官网查查，或是直接给作者发邮件问问，答案最直接。

Answer 2

http://blog.csdn.net/michaeljy1991/article/details/50315591

看看是不是这个原因

Answer 3

算是解决了， 我把JedisPool定义在本地，为static final对象

Answer 4

回复
定义成这样？ private static final JedisPool pool;

Answer 5

引用来自“yangkf”的评论

其实后来发现时没有正确的释放jedis resource 造成的web内存泄露和redis线程阻塞造成.

Answer 6

其实后来发现时没有正确的释放jedis resource 造成的web内存泄露和redis线程阻塞造成.

Answer 7

引用来自“tcltang”的评论

建议去官网查查，或是直接给作者发邮件问问，答案最直接。

Answer 8

引用来自“char1st”的评论

testOnBorrow=false 再看下有没有异常。

Answer 9

testOnBorrow=false 再看下有没有异常。

Answer 10

在客户端代码：

try{

SharedJedis j = sharedJedisPool.getResource();
j.get(key);

} catch(JedisConnectionException je){

sharedJedisPool.returnBrokenResource(j);

} catch(Exception e){

log.err(e.getMessage(),e);

} finally{

sharedJedisPool.returnResource(j);

}

客户端报这个错误：

redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
    at redis.clients.util.Pool.getResource(Pool.java:22)
    …………………………
    Caused by: java.util.NoSuchElementException: Could not create a validated object, cause: ValidateObject failed