有谁看到过这个错误 oracle.ucp.UniversalConnectionPoolException: The endRequest call failed

Question

我有 APEX 在 tomcat 中运行。最近升级到最新的 ORDS 21.4.2 和 APEX 21.1 后 当尝试访问应用程序时，我收到错误。这种情况似乎只发生在隔夜不活动后的早晨。第一次访问该 URL 会导致该错误，后续则正常。

ORDS 无法连接到数据库。如果数据库不可用、已达到最大会话数或池配置不正确，则可能会发生这种情况。连接池名为：|apx100dev|pu|出现以下错误：获取连接时发生异常：oracle.ucp.UniversalConnectionPoolException：endRequest 调用失败

在 Tomcat 日志中，我可以看到相当大的堆栈：

2022-03-30T10:31:12.752Z WARNING     :::got unchecked exception from user-defined connection labeling callback
2022-03-30T10:31:12.753Z SEVERE      <Zn1NIBM1in5tsu0bK4FrYQ> GET ords-apex-dev.apps.adpd-np.k8s.bgintdev.com /ords/<dbname>/ws/r/application/home?session=<sessionid> 503 ORDS was unable to make a connection to the database.  This can occur if the database is unavailable, the maximum number of sessions has been reached or the pool is not correctly configured. The connection pool named: |apx100dev|pu| had the following error(s): Exception occurred while getting connection: oracle.ucp.UniversalConnectionPoolException: The endRequest call failed 
ServiceUnavailableException [statusCode=503, logLevel=SEVERE, reasons=[ORDS was unable to make a connection to the database.  This can occur if the database is unavailable, the maximum number of sessions has been reached or the pool is not correctly configured. The connection pool named: |apx100dev|pu| had the following error(s): Exception occurred while getting connection: oracle.ucp.UniversalConnectionPoolException: The endRequest call failed ]]
    at oracle.dbtools.http.errors.ServletResponseExceptionMapper.mapError(ServletResponseExceptionMapper.java:48)
    at oracle.dbtools.http.errors.ErrorLogger.log(ErrorLogger.java:26)
    at oracle.dbtools.http.errors.ErrorPageFilter.doFilter(ErrorPageFilter.java:106)
    at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
    at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
    at oracle.dbtools.http.secure.ForceHttpsFilter.doFilter(ForceHttpsFilter.java:74)
    at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
    at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
    at oracle.dbtools.http.auth.ForceAuthFilter.doFilter(ForceAuthFilter.java:44)
    at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
    at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
    at oracle.dbtools.http.filters.Filters.filter(Filters.java:67)
    at oracle.dbtools.http.entrypoint.EntryPoint.service(EntryPoint.java:70)
    at oracle.dbtools.http.entrypoint.EntryPointServlet.service(EntryPointServlet.java:125)
    at oracle.dbtools.entrypoint.WebApplicationRequestEntryPoint.service(WebApplicationRequestEntryPoint.java:50)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
    at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:769)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:359)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1735)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: oracle.dbtools.url.mapping.TargetNotAvailableException: ORDS was unable to make a connection to the database.  This can occur if the database is unavailable, the maximum number of sessions has been reached or the pool is not correctly configured. The connection pool named: |apx100dev|pu| had the following error(s): Exception occurred while getting connection: oracle.ucp.UniversalConnectionPoolException: The endRequest call failed 
    at oracle.dbtools.url.mapping.db.PoolInjector.inject(PoolInjector.java:64)
    at oracle.dbtools.url.mapping.db.DatabaseURLMappingBase.addServices(DatabaseURLMappingBase.java:338)
    at oracle.dbtools.url.mapping.URLMappingBase.doFilter(URLMappingBase.java:75)
    at oracle.dbtools.url.mapping.db.DatabaseTenantMapping.dispatchSelf(DatabaseTenantMapping.java:193)
    at oracle.dbtools.url.mapping.db.DatabaseTenantMappingBase.doFilter(DatabaseTenantMappingBase.java:50)
    at oracle.dbtools.url.mapping.defaultpool.LegacyDatabaseTenantMapping.doFilter(LegacyDatabaseTenantMapping.java:46)
    at oracle.dbtools.url.mapping.tenant.TenantMappingDispatcher.dispatch(TenantMappingDispatcher.java:52)
    at oracle.dbtools.url.mapping.db.DatabaseTenantMappingBase.dispatchChild(DatabaseTenantMappingBase.java:151)
    at oracle.dbtools.url.mapping.db.DatabaseTenantMappingBase.doFilter(DatabaseTenantMappingBase.java:48)
    at oracle.dbtools.url.mapping.defaultpool.LegacyDatabaseTenantMapping.doFilter(LegacyDatabaseTenantMapping.java:46)
    at oracle.dbtools.url.mapping.defaultpool.LegacyDatabaseTenantMapping.doFilter(LegacyDatabaseTenantMapping.java:44)
    at oracle.dbtools.url.mapping.tenant.TenantMappingDispatcher.dispatch(TenantMappingDispatcher.java:52)
    at oracle.dbtools.url.mapping.tenant.TenantMappingFilter.doFilter(TenantMappingFilter.java:75)
    at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
    at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
    at oracle.dbtools.http.forwarding.ForwardingFailedFilter.doFilter(ForwardingFailedFilter.java:41)
    at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
    at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
    at oracle.dbtools.http.auth.external.ExternalSessionFilter.doFilter(ExternalSessionFilter.java:59)
    at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
    at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
    at oracle.dbtools.rt.authentication.apex.ApexSessionQueryRewriteFilter.doFilter(ApexSessionQueryRewriteFilter.java:58)
    at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
    at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
    at oracle.dbtools.http.cors.CORSResponseFilter.doFilter(CORSResponseFilter.java:90)
    at oracle.dbtools.http.filters.HttpResponseFilter.doFilter(HttpResponseFilter.java:45)
    at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
    at oracle.dbtools.http.filters.AbsoluteLocationFilter.doFilter(AbsoluteLocationFilter.java:65)
    at oracle.dbtools.http.filters.HttpResponseFilter.doFilter(HttpResponseFilter.java:45)
    at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
    at oracle.dbtools.http.auth.external.ExternalAccessValidationFilter.doFilter(ExternalAccessValidationFilter.java:59)
    at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
    at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
    at oracle.dbtools.http.errors.ErrorPageFilter.doFilter(ErrorPageFilter.java:87)
    ... 36 more
Caused by: oracle.dbtools.common.jdbc.ConnectionPoolConfigurationException: ORDS was unable to make a connection to the database.  This can occur if the database is unavailable, the maximum number of sessions has been reached or the pool is not correctly configured. The connection pool named: |apx100dev|pu| had the following error(s): Exception occurred while getting connection: oracle.ucp.UniversalConnectionPoolException: The endRequest call failed 
    at oracle.dbtools.common.jdbc.ConnectionPoolExceptions.from(ConnectionPoolExceptions.java:47)
    at oracle.dbtools.common.jdbc.DataSourceConnection.getPooledConnection(DataSourceConnection.java:220)
    at oracle.dbtools.common.jdbc.DataSourceConnection.getPooledConnection(DataSourceConnection.java:186)
    at oracle.dbtools.common.jdbc.DataSourceConnection.getDefaultConnection(DataSourceConnection.java:144)
    at oracle.dbtools.common.jdbc.DataSourceConnection.getConnection(DataSourceConnection.java:58)
    at oracle.dbtools.common.pools.DataSourceTargetImpl.getConnection(DataSourceTargetImpl.java:57)
    at oracle.dbtools.common.config.db.SchemaConnectionFactoryBase.inject(SchemaConnectionFactoryBase.java:69)
    at oracle.dbtools.url.mapping.db.PoolInjector.inject(PoolInjector.java:58)
    ... 69 more
Caused by: java.sql.SQLException: Exception occurred while getting connection: oracle.ucp.UniversalConnectionPoolException: The endRequest call failed
    at oracle.ucp.util.UCPErrorHandler.newSQLException(UCPErrorHandler.java:499)
    at oracle.ucp.util.UCPErrorHandler.throwSQLException(UCPErrorHandler.java:176)
    at oracle.ucp.jdbc.PoolDataSourceImpl.getConnection(PoolDataSourceImpl.java:1900)
    at oracle.ucp.jdbc.PoolDataSourceImpl.access$300(PoolDataSourceImpl.java:225)
    at oracle.ucp.jdbc.PoolDataSourceImpl$3.build(PoolDataSourceImpl.java:4050)
    at oracle.ucp.jdbc.PoolDataSourceImpl.getConnection(PoolDataSourceImpl.java:1823)
    at oracle.dbtools.common.ucp.PoolDataSourceDelegate.getConnection(PoolDataSourceDelegate.java:109)
    at oracle.dbtools.common.config.db.RefreshablePoolDataSource.getConnection(RefreshablePoolDataSource.java:109)
    at oracle.dbtools.common.ucp.PoolDataSourceDelegate.getConnection(PoolDataSourceDelegate.java:109)
    at oracle.dbtools.common.ucp.PoolDataSourceDelegate.getConnection(PoolDataSourceDelegate.java:96)
    at oracle.dbtools.common.jdbc.DataSourceConnection.getPooledConnection(DataSourceConnection.java:210)
    ... 75 more
Caused by: oracle.ucp.UniversalConnectionPoolException: The endRequest call failed
    at oracle.ucp.util.UCPErrorHandler.newUniversalConnectionPoolException(UCPErrorHandler.java:379)
    at oracle.ucp.util.UCPErrorHandler.newUniversalConnectionPoolException(UCPErrorHandler.java:347)
    at oracle.ucp.jdbc.oracle.OracleConnectionConnectionPool.returnConnection(OracleConnectionConnectionPool.java:241)
    at oracle.ucp.common.UniversalConnectionPoolImpl.borrowConnectionAndValidateHelper(UniversalConnectionPoolImpl.java:225)
    at oracle.ucp.common.UniversalConnectionPoolImpl.borrowConnectionAndValidate(UniversalConnectionPoolImpl.java:166)
    at oracle.ucp.common.UniversalConnectionPoolImpl.borrowConnection(UniversalConnectionPoolImpl.java:139)
    at oracle.ucp.jdbc.JDBCConnectionPool.borrowConnection(JDBCConnectionPool.java:185)
    at oracle.ucp.jdbc.oracle.OracleJDBCConnectionPool.borrowConnection(OracleJDBCConnectionPool.java:611)
    at oracle.ucp.jdbc.oracle.OracleConnectionConnectionPool.borrowConnection(OracleConnectionConnectionPool.java:119)
    at oracle.ucp.jdbc.PoolDataSourceImpl.getConnection(PoolDataSourceImpl.java:1880)
    ... 83 more
Caused by: java.sql.SQLRecoverableException: IO Error
    at oracle.jdbc.driver.T4CConnection.closeProxySession(T4CConnection.java:3512)
    at oracle.jdbc.driver.PhysicalConnection.close(PhysicalConnection.java:2856)
    at oracle.jdbc.driver.PhysicalConnection.close(PhysicalConnection.java:2742)
    at oracle.jdbc.driver.PhysicalConnection.closeConnectionSafely(PhysicalConnection.java:10964)
    at oracle.jdbc.driver.T4CConnection.drainOnInbandNotification(T4CConnection.java:7112)
    at oracle.jdbc.driver.PhysicalConnection.checkAndDrain(PhysicalConnection.java:10846)
    at oracle.jdbc.driver.PhysicalConnection.endRequest(PhysicalConnection.java:10446)
    at oracle.jdbc.driver.T4CConnection.endRequest(T4CConnection.java:9760)
    at oracle.jdbc.driver.PhysicalConnection.endRequest(PhysicalConnection.java:10459)
    at oracle.ucp.jdbc.oracle.OracleConnectionConnectionPool.returnConnection(OracleConnectionConnectionPool.java:238)
    ... 90 more
Caused by: java.nio.channels.ClosedChannelException
    at java.base/sun.nio.ch.SocketChannelImpl.ensureOpenAndConnected(SocketChannelImpl.java:215)
    at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:527)
    at oracle.net.nt.TimeoutSocketChannel.write(TimeoutSocketChannel.java:467)
    at oracle.net.ns.NIOPacket.writeToSocketChannel(NIOPacket.java:361)
    at oracle.net.ns.NIONSDataChannel.writeDataToSocketChannel(NIONSDataChannel.java:206)
    at oracle.net.ns.NIONSDataChannel.writeDataToSocketChannel(NIONSDataChannel.java:151)
    at oracle.jdbc.driver.T4CMAREngineNIO.flush(T4CMAREngineNIO.java:804)
    at oracle.jdbc.driver.T4CMAREngineNIO.prepareForUnmarshall(T4CMAREngineNIO.java:822)
    at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:487)
    at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:622)
    at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:299)
    at oracle.jdbc.driver.T4C7Ocommoncall.doOLOGOFF(T4C7Ocommoncall.java:74)
    at oracle.jdbc.driver.T4CConnection.closeProxySession(T4CConnection.java:3501)
    ... 99 more

我一直在调整池限制的配置，并且连接重用但没有什么区别。