weblogic故障，急，请各位大虾帮忙！

Question

客户的问题 ，很急 ，请各位大虾 指导 ！

从28号开始，到今天11点截止，共发生了4次故障，故障发生时主要是系统查询缓慢，查询服务器CPU占用率在60%-85%之间，Queue 逐渐上升，以下是从Weblogic上提取下来的部分日志：
####<2008-10-29 下午12时28分20秒 CST> <Error> <HTTP> <localhost.localdomain> <zxserver> <ExecuteThread: '6' for queue: 'weblogic.kernel.Default'> <<WLS Kernel>> <> <BEA-101017> <[ServletContext(id=37787602,name=besttone_zx,context-path=/besttone_zx)] Root cause of ServletException.
javax.servlet.jsp.JspException: Input/output error: java.net.SocketException: Connection reset
        at org.apache.struts.taglib.TagUtils.writePrevious(TagUtils.java:1264)
        at org.apache.struts.taglib.logic.IterateTag.doAfterBody(IterateTag.java:346)
        at jsp_servlet._qc.__qc_add_querytype._jspService(__qc_add_querytype.java:1483)
        at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
        at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
        at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
        at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:2
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
        at cn.sh.ideal.util.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:41)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
        at weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:110)
        at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1085)
        at org.apache.struts.action.RequestProcessor.processForwardConfig(RequestProcessor.java:39
        at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:172)
        at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
        at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:416)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
        at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
        at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
        at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:2
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
        at cn.sh.ideal.util.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:41)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7049)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
        at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3732)
        at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
        at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
        at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
>
####<2008-10-29 下午12时29分01秒 CST> <Error> <HTTP> <localhost.localdomain> <zxserver> <ExecuteThread: '25' for queue: 'weblogic.kernel.Default'> <<WLS Kernel>> <> <BEA-101020> <[ServletContext(id=37787602,name=besttone_zx,context-path=/besttone_zx)] Servlet failed with Exception
java.lang.StringIndexOutOfBoundsException: String index out of range: -2
        at java.lang.String.substring(II)Ljava.lang.String;(Unknown Source)
        at jsp_servlet._agent.__business_info_base._jspService(__business_info_base.java:1651)
        at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
        at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
        at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
        at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:2
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
        at cn.sh.ideal.util.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:41)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
        at weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:110)
        at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1085)
        at org.apache.struts.action.RequestProcessor.processForwardConfig(RequestProcessor.java:39
        at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:172)
        at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
        at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:416)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
        at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
        at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
        at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:2
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
        at cn.sh.ideal.util.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:41)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7049)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
        at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3732)
        at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
        at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
        at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

请各位 大虾 指导 ，谢谢 ！

Answer 1

性能问题，可能有多种原因引起。查原因最好在发生问题时做几次thread dump，即kill -3.然后根据输出才能进一步分析。

至于你贴在这里的第一个错误堆栈，可能是结果，不是原因，也就是说，在性能下降时，客户端终止了IE操作，断开了socket，所以connection reset。而第二个错误堆栈，属于另一个线程，是应用逻辑抛错，不会导致性能问题。