zookeeper集群（3个节点）中的leader宕机后，导致MQ集群的master也挂掉，请问为什么？

Question

实际项目：搭建基于zookeeper的Replicated leveldb ActiveMQ集群；搭建成功后进行测试，模拟测试当zookeeper的leader节点宕机后，会不会对集群造成影响。发现zookeeper剩余的两个节点进行了leader选举；但是问题是，MQ集群（3个）中的master节点不再运行，请问是什么原因导致的？

activemq日志报了一个这么错误：

2019-08-07 17:12:29,703 | WARN | Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect | org.apache.zookeeper.ClientCnxn | main-SendThread(10.166.18.129:2181)
java.net.ConnectException: Connection refused

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)[:1.8.0_211]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)[:1.8.0_211]
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)[zookeeper-3.4.6.jar:3.4.6-1569965]

2019-08-07 17:12:30,269 | WARN | SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/usr/local/activemq/activemq//conf/login.config'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. | org.apache.zookeeper.ClientCnxn | main-SendThread(10.166.18.130:2181)
2019-08-07 17:12:30,270 | INFO | Opening socket connection to server 10.166.18.130/10.166.18.130:2181 | org.apache.zookeeper.ClientCnxn | main-SendThread(10.166.18.130:2181)
2019-08-07 17:12:30,271 | WARN | unprocessed event state: AuthFailed | org.apache.activemq.leveldb.replicated.groups.ZKClient | main-EventThread
2019-08-07 17:12:30,273 | INFO | Socket connection established to 10.166.18.130/10.166.18.130:2181, initiating session | org.apache.zookeeper.ClientCnxn | main-SendThread(10.166.18.130:2181)
2019-08-07 17:12:30,449 | ERROR | Failed to start Apache ActiveMQ (localhost, null) | org.apache.activemq.broker.BrokerService | main
java.util.concurrent.TimeoutException: timeout reached while waiting on the lock: java.lang.Object@748fe51d

at org.linkedin.util.concurrent.ConcurrentUtils.awaitUntil(ConcurrentUtils.java:117)[org.linkedin.util-core-1.4.0.jar:]
at org.apache.activemq.leveldb.replicated.groups.ZKClient.waitForState(ZKClient.java:260)[activemq-leveldb-store-5.15.9.jar:5.15.9]
at org.apache.activemq.leveldb.replicated.groups.ZKClient.waitForConnected(ZKClient.java:248)[activemq-leveldb-store-5.15.9.jar:5.15.9]
at org.apache.activemq.leveldb.replicated.ElectingLevelDBStore.init(ElectingLevelDBStore.scala:189)[activemq-leveldb-store-5.15.9.jar:5.15.9]
at org.apache.activemq.broker.LockableServiceSupport.preStart(LockableServiceSupport.java:89)[activemq-broker-5.15.9.jar:5.15.9]
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:54)[activemq-client-5.15.9.jar:5.15.9]
at org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:687)[activemq-broker-5.15.9.jar:5.15.9]
at org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:671)[activemq-broker-5.15.9.jar:5.15.9]
at org.apache.activemq.broker.BrokerService.start(BrokerService.java:635)[activemq-broker-5.15.9.jar:5.15.9]
at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73)[activemq-spring-5.15.9.jar:5.15.9]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_211]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_211]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_211]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_211]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1763)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1700)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1630)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)[spring-context-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)[spring-context-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)[xbean-spring-4.2.jar:4.2]
at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)[xbean-spring-4.2.jar:4.2]
at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:104)[activemq-spring-5.15.9.jar:5.15.9]
at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:104)[activemq-spring-5.15.9.jar:5.15.9]
at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:67)[activemq-spring-5.15.9.jar:5.15.9]
at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)[activemq-broker-5.15.9.jar:5.15.9]
at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)[activemq-broker-5.15.9.jar:5.15.9]
at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:87)[activemq-console-5.15.9.jar:5.15.9]
at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.15.9.jar:5.15.9]
at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.15.9.jar:5.15.9]
at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.15.9.jar:5.15.9]
at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.15.9.jar:5.15.9]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_211]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_211]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_211]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_211]
at org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.15.9]
at org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.15.9]

2019-08-07 17:12:30,452 | INFO | Apache ActiveMQ 5.15.9 (localhost, null) is shutting down | org.apache.activemq.broker.BrokerService | main
2019-08-07 17:12:30,455 | INFO | Network Connector DiscoveryNetworkConnector:default-nc:BrokerService[localhost] stopped | org.apache.activemq.network.NetworkConnector | main
2019-08-07 17:12:30,457 | INFO | Connector openwire stopped | org.apache.activemq.broker.TransportConnector | main
2019-08-07 17:12:30,460 | INFO | Connector amqp stopped | org.apache.activemq.broker.TransportConnector | main
2019-08-07 17:12:30,462 | INFO | Connector stomp stopped | org.apache.activemq.broker.TransportConnector | main
2019-08-07 17:12:30,463 | INFO | Connector mqtt stopped | org.apache.activemq.broker.TransportConnector | main
2019-08-07 17:12:30,464 | INFO | Connector ws stopped | org.apache.activemq.broker.TransportConnector | main
2019-08-07 17:12:30,465 | INFO | StateChangeDispatcher terminated. | org.apache.activemq.leveldb.replicated.groups.ZKClient | ZooKeeper state change dispatcher thread
2019-08-07 17:12:31,040 | INFO | Session: 0x0 closed | org.apache.zookeeper.ZooKeeper | main
2019-08-07 17:12:31,042 | INFO | EventThread shut down | org.apache.zookeeper.ClientCnxn | main-EventThread
2019-08-07 17:12:31,042 | INFO | Apache ActiveMQ 5.15.9 (localhost, null) uptime 31.649 seconds | org.apache.activemq.broker.BrokerService | main
2019-08-07 17:12:31,046 | INFO | Apache ActiveMQ 5.15.9 (localhost, null) is shutdown | org.apache.activemq.broker.BrokerService | main
2019-08-07 17:12:31,049 | INFO | Closing org.apache.activemq.xbean.XBeanBrokerFactory$1@5aae8eb5: startup date [Wed Aug 07 17:11:59 CST 2019]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
2019-08-07 17:12:31,053 | WARN | Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in file [/usr/local/activemq/activemq/conf/activemq.xml]: Invocation of init method failed; nested exception is java.util.concurrent.TimeoutException: timeout reached while waiting on the lock: java.lang.Object@748fe51d | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main

.

Answer 1

问题以及解决！需要配置主机名称映射：
10.47.3.137 cluster1
10.47.3.132 cluster2
10.47.3.139 cluster3

server.1=cluster1:2888:3888
server.2=cluster2:2888:3888
server.3=cluster3:2888:3888

Answer 2

我在本地也有这个问题，改成localhost1~5之后仍然不行