hbase启动失败 zookeeper异常

Question

hbase启动失败，日志如下：

2015-02-13 16:03:15,264 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:java.library.path=/opt/hadoop/lib/native:/opt/hbase/lib/native/Linux-i386-32
2015-02-13 16:03:15,265 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:java.io.tmpdir=/tmp
2015-02-13 16:03:15,273 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:java.compiler=<NA>
2015-02-13 16:03:15,274 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:os.name=Linux
2015-02-13 16:03:15,274 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:os.arch=i386
2015-02-13 16:03:15,274 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:os.version=2.6.32-279.el6.i686
2015-02-13 16:03:15,274 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:user.name=root
2015-02-13 16:03:15,274 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:user.home=/root
2015-02-13 16:03:15,301 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:user.dir=/opt/hbase
2015-02-13 16:03:15,886 INFO org.apache.zookeeper.server.ZooKeeperServer: tickTime set to 3000
2015-02-13 16:03:15,887 INFO org.apache.zookeeper.server.ZooKeeperServer: minSessionTimeout set to -1
2015-02-13 16:03:15,887 INFO org.apache.zookeeper.server.ZooKeeperServer: maxSessionTimeout set to 180000
2015-02-13 16:03:18,917 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2181
2015-02-13 16:03:20,680 INFO org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot /tmp/hbase-root/zookeeper/version-2/snapshot.21
2015-02-13 16:03:20,927 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /172.20.5.21:58697
2015-02-13 16:03:22,298 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-02-13 16:03:22,299 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /172.20.5.21:58697 (no session established for client)
2015-02-13 16:03:23,123 INFO org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 0x54 to /tmp/hbase-root/zookeeper/version-2/snapshot.54
2015-02-13 16:03:24,028 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /172.20.5.21:58700
2015-02-13 16:03:24,106 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-02-13 16:03:24,106 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /172.20.5.21:58700 (no session established for client)
2015-02-13 16:03:26,199 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /172.20.5.21:58701
2015-02-13 16:03:26,233 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-02-13 16:03:26,233 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /172.20.5.21:58701 (no session established for client)
2015-02-13 16:03:27,486 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /172.20.5.21:58704
2015-02-13 16:03:27,572 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-02-13 16:03:27,572 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /172.20.5.21:58704 (no session established for client)
2015-02-13 16:03:29,378 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /172.20.5.21:58705
2015-02-13 16:03:29,398 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-02-13 16:03:29,398 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /172.20.5.21:58705 (no session established for client)
2015-02-13 16:03:31,493 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /172.20.5.21:58707
2015-02-13 16:03:31,523 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-02-13 16:03:31,523 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /172.20.5.21:58707 (no session established for client)
2015-02-13 16:03:33,639 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /172.20.5.21:58709
2015-02-13 16:03:33,667 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-02-13 16:03:33,667 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /172.20.5.21:58709 (no session established for client)
2015-02-13 16:03:35,770 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /172.20.5.21:58710
2015-02-13 16:03:36,127 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to renew session 0x14b81bed0eb0001 at /172.20.5.21:58710
2015-02-13 16:03:36,830 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14b81bed0eb0001 with negotiated timeout 180000 for client /172.20.5.21:58710
2015-02-13 16:03:52,934 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /172.20.5.21:58720
2015-02-13 16:03:53,227 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /172.20.5.21:58720
2015-02-13 16:03:53,232 INFO org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file: log.55
2015-02-13 16:03:53,680 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14b81f6b2ac0000 with negotiated timeout 180000 for client /172.20.5.21:58720
2015-02-13 16:04:19,461 INFO org.apache.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x14b81f6b2ac0000 type:delete cxid:0xc zxid:0x57 txntype:-1 reqpath:n/a Error Path:/hbase/backup-masters/MyDB,60000,1423814625610 Error:KeeperErrorCode = NoNode for /hbase/backup-masters/MyDB,60000,1423814625610
2015-02-13 16:04:30,303 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x14b81f6b2ac0000
2015-02-13 16:04:30,529 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x14b81f6b2ac0000, likely client has closed socket
 at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
 at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
 at java.lang.Thread.run(Thread.java:679)
2015-02-13 16:04:31,036 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /172.20.5.21:58720 which had sessionid 0x14b81f6b2ac0000

 

Answer 1

引用来自“震秦”的评论

看样子你是用的 HBase 内置 Zookeeper的.

2015-02-13 16:03:15,886 INFO org.apache.zookeeper.server.ZooKeeperServer: tickTime set to 3000
2015-02-13 16:03:15,887 INFO org.apache.zookeeper.server.ZooKeeperServer: minSessionTimeout set to -1
2015-02-13 16:03:15,887 INFO org.apache.zookeeper.server.ZooKeeperServer: maxSessionTimeout set to 180000
2015-02-13 16:03:18,917 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2181
2015-02-13 16:03:20,680 INFO org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot /tmp/hbase-root/zookeeper/version-2/snapshot.21
2015-02-13 16:03:20,927 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /172.20.5.21:58697
2015-02-13 16:03:22,298 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-02-13 16:03:22,299 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /172.20.5.21:58697 (no session established for client)
2015-02-13 16:03:23,123 INFO org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 0x54 to /tmp/hbase-root/zookeeper/version-2/snapshot.54
2015-02-13 16:03:24,028 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /172.20.5.21:58700
2015-02-13 16:03:24,106 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-02-13 16:03:24,106 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /172.20.5.21:58700 (no session established for client)
2015-02-13 16:03:26,199 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /172.20.5.21:58701

这些日志表明你的 ZooKeeper 还没有启动起来, HBase 就去访问 Zookeeper 了. 建议把 Zookeeper 的 data 和 logs 数据 copy 出来, 在单独的进程启动 Zookeeper, 然后启动, 也许可行.

Answer 2

看样子你是用的 HBase 内置 Zookeeper的.

2015-02-13 16:03:15,886 INFO org.apache.zookeeper.server.ZooKeeperServer: tickTime set to 3000
2015-02-13 16:03:15,887 INFO org.apache.zookeeper.server.ZooKeeperServer: minSessionTimeout set to -1
2015-02-13 16:03:15,887 INFO org.apache.zookeeper.server.ZooKeeperServer: maxSessionTimeout set to 180000
2015-02-13 16:03:18,917 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2181
2015-02-13 16:03:20,680 INFO org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot /tmp/hbase-root/zookeeper/version-2/snapshot.21
2015-02-13 16:03:20,927 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /172.20.5.21:58697
2015-02-13 16:03:22,298 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-02-13 16:03:22,299 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /172.20.5.21:58697 (no session established for client)
2015-02-13 16:03:23,123 INFO org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 0x54 to /tmp/hbase-root/zookeeper/version-2/snapshot.54
2015-02-13 16:03:24,028 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /172.20.5.21:58700
2015-02-13 16:03:24,106 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-02-13 16:03:24,106 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /172.20.5.21:58700 (no session established for client)
2015-02-13 16:03:26,199 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /172.20.5.21:58701

这些日志表明你的 ZooKeeper 还没有启动起来, HBase 就去访问 Zookeeper 了. 建议把 Zookeeper 的 data 和 logs 数据 copy 出来, 在单独的进程启动 Zookeeper, 然后启动, 也许可行.

Answer 3

1. 目测是用 root 用户启动的, 你确定 root 在 hdfs 上有访问的权限.

2. 请确定你的 Zookeeper 能连上, 自己用 zkCli.sh 试试可链接否