nexus3上传docker镜像报错EOF

Question

RT:
docker login 私服正常，网页打开nexus3管理端正常。
docker push的时候EOF异常：

1a6dd0fe761d: Preparing
1a6dd0fe761d: Pushing [==================================================>]  34.79MB/34.79MB
1731012b8e30: Layer already exists
0db1c93dd886: Layer already exists
2104e5b926f6: Preparing
2104e5b926f6: Layer already exists

EOF

查看nexus3的日志为：

org.sonatype.nexus.blobstore.api.BlobStoreException: BlobId: path$nexus-repository-docker/e379b6e1-7f89-4b22-b2d8-3a82c8deef6d/1596539444735, java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms, Cause: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

at org.sonatype.nexus.blobstore.file.FileBlobStore.tryCreate(FileBlobStore.java:372)

at org.sonatype.nexus.blobstore.file.FileBlobStore.create(FileBlobStore.java:308)

at org.sonatype.nexus.blobstore.file.FileBlobStore.doCreate(FileBlobStore.java:289)

at org.sonatype.nexus.blobstore.BlobStoreSupport.create(BlobStoreSupport.java:127)

at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)

at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)

at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)

at org.sonatype.nexus.blobstore.BlobStoreSupport.create(BlobStoreSupport.java:111)

at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)

at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)

at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)

at org.sonatype.nexus.repository.docker.internal.Upload.addBlob(Upload.java:125)

at org.sonatype.nexus.repository.docker.internal.UploadManagerImpl.add(UploadManagerImpl.java:77)

at org.sonatype.nexus.repository.docker.internal.DockerHostedFacetImpl.uploadBlobRange(DockerHostedFacetImpl.java:588)

at org.sonatype.nexus.repository.docker.internal.DockerHostedFacet$uploadBlobRange$3.call(Unknown Source)

at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure4.doCall(V2Handlers.groovy:137)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)

at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)

at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)

at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)

at groovy.lang.Closure.call(Closure.java:420)

at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)

at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)

at com.sun.proxy.$Proxy190.handle(Unknown Source)

at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)

at org.sonatype.nexus.repository.storage.UnitOfWorkHandler.handle(UnitOfWorkHandler.java:39)

at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)

at com.sonatype.nexus.clm.internal.FirewallContributedHandler.handle(FirewallContributedHandler.java:96)

at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)

at org.sonatype.nexus.repository.view.handlers.HandlerContributor.handle(HandlerContributor.java:67)

at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)

at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:51)

at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)

at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)

at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure18.doCall(V2Handlers.groovy:298)

at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)

at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)

at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)

at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)

at groovy.lang.Closure.call(Closure.java:420)

at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)

at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)

at com.sun.proxy.$Proxy190.handle(Unknown Source)

at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)

at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)

at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure1.doCall(V2Handlers.groovy:90)

at sun.reflect.GeneratedMethodAccessor100.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)

at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)

at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)

at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)

at groovy.lang.Closure.call(Closure.java:420)

at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)

at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)

at com.sun.proxy.$Proxy190.handle(Unknown Source)

at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)

at com.sonatype.analytics.internal.handler.AnalyticsMeteringHandler.handle(AnalyticsMeteringHandler.java:69)

at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)

at org.sonatype.nexus.repository.view.handlers.TimingHandler.handle(TimingHandler.java:58)

at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)

at org.sonatype.nexus.repository.view.Context.start(Context.java:114)

at org.sonatype.nexus.repository.view.Router.dispatch(Router.java:65)

at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:52)

at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:43)

at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.dispatchAndSend(ViewServlet.java:212)

at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.doService(ViewServlet.java:174)

at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.service(ViewServlet.java:126)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)

at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)

at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)

at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)

at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)

at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)

at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)

at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)

at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)

at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)

at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)

at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)

at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)

at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)

at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)

at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)

at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)

at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)

at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)

at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)

at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)

at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)

at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:85)

at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)

at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)

at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)

at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387)

at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)

at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:101)

at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at org.sonatype.nexus.repository.httpbridge.internal.ExhaustRequestFilter.doFilter(ExhaustRequestFilter.java:80)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:114)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:112)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:79)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)

at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)

at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)

at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:239)

at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)

at org.eclipse.jetty.server.Server.handle(Server.java:505)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)

at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

at org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1085)

at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:318)

at com.google.common.io.CountingInputStream.read(CountingInputStream.java:63)

at java.security.DigestInputStream.read(DigestInputStream.java:161)

at java.io.FilterInputStream.read(FilterInputStream.java:133)

at java.io.FilterInputStream.read(FilterInputStream.java:107)

at com.google.common.io.ByteStreams.copy(ByteStreams.java:109)

at org.sonatype.nexus.blobstore.file.internal.SimpleFileOperations.create(SimpleFileOperations.java:61)

at org.sonatype.nexus.blobstore.file.FileBlobStore.lambda$1(FileBlobStore.java:289)

at org.sonatype.nexus.blobstore.file.FileBlobStore.tryCreate(FileBlobStore.java:342)

... 158 common frames omitted

Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166)

at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

... 1 common frames omitted

Suppressed: java.lang.Throwable: HttpInput failure

at org.eclipse.jetty.server.HttpInput.failed(HttpInput.java:831)

at org.eclipse.jetty.server.HttpConnection$BlockingReadCallback.failed(HttpConnection.java:661)

at org.eclipse.jetty.io.FillInterest.onFail(FillInterest.java:138)

at org.eclipse.jetty.io.AbstractEndPoint.onIdleExpired(AbstractEndPoint.java:406)

... 9 common frames omitted

该nexus3是部署在k8s环境的，使用的pvc是nas存储。镜像版本是3.22.0，请教这个问题如何解决呢？

Answer 1

参考issue：https://community.sonatype.com/t/java-io-ioexception-java-util-concurrent-timeoutexception-idle-timeout-expired-30000-30000-ms-when-up-or-downloading-large-files/2891/2

官方nginx示例

解决方案 关闭proxy-buffering：

nginx.ingress.kubernetes.io/proxy-buffering: "off"