使用 AWS TransferManager 时出现 RejectedExecutionException

Question

我有以下方法可以对 amazonS3 进行分段上传：

public static void uploadToS3UserPath(String filePath, String bucket, String userPath) {
    ClientConfiguration config = new ClientConfiguration();
    config.setProxyHost("host");
    config.setProxyPort(3128);
    config.setProtocol(Protocol.HTTP);

    TransferManager tm = new TransferManager(new AmazonS3Client(credentials, config));
    TransferManagerConfiguration conf = new TransferManagerConfiguration();
    conf.setMinimumUploadPartSize(50 * 1024 * 1024); //use 50 megabytes parts;
    tm.setConfiguration(conf);

    PutObjectRequest req = new PutObjectRequest(bucket, userPath, new File(filePath));
    req.setCannedAcl(CannedAccessControlList.PublicRead);
    req.setStorageClass(StorageClass.ReducedRedundancy); //we do it to decrease storage costs
    Upload up = tm.upload(req);

    try {
        up.waitForCompletion();
        tm.shutdownNow();
        System.out.println("Upload completed successfully");
    } catch (AmazonClientException ex) {
        Logger.getLogger(Utilities.class.getName()).log(Level.SEVERE, ex.getLocalizedMessage(), ex);
        tm.shutdownNow();
    } catch (InterruptedException ex) {
        Logger.getLogger(Utilities.class.getName()).log(Level.SEVERE, ex.getLocalizedMessage(), ex);
    }
}

问题是有时我会得到：

    com.amazonaws.AmazonClientException: Unable to complete transfer: null
            at
    com.amazonaws.services.s3.transfer.Transfer.unwrapExecutionException(Transfer.java:226)
            at
    com.amazonaws.services.s3.transfer.Transfer.rethrowExecutionException(Transfer.java:210)
            at
    com.amazonaws.services.s3.transfer.Transfer.waitForCompletion(Transfer.java:116)
            at
   xx.xxx.xxx.xxx.agent.Utilities.uploadToS3UserPath(Utilities.java:373)
            at
    xx.xxx.xxx.xxx.agent.AbstractAgent.uploadOutputToS3(AbstractAgent.java:312)
            at
    uk.org.infectogenomics.agent.AbstractAgent.uploadOutputToS3(AbstractAgent.java:305)
            at xx.xxx.xxx.xxx.hostEl.HostEl.run(HostEl.java:598)
            at xx.xxx.xxx.xxx.agent.Agent.main(Agent.java:98)
    Caused by: java.util.concurrent.RejectedExecutionException
            at
    java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1759)
            at
    java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
            at
    java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:216)
            at
    java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:378)
            at
    com.amazonaws.services.s3.transfer.internal.UploadMonitor.reschedule(UploadMonitor.java:210)
            at
    com.amazonaws.services.s3.transfer.internal.UploadMonitor.upload(UploadMonitor.java:197)
            at
    com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:148)
            at
    com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:49)
            at
    java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
            at java.util.concurrent.FutureTask.run(FutureTask.java:138)
            at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
            at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
            at java.lang.Thread.run(Thread.java:619)

问题是我不知道为什么这仅适用于随机上传。显然原因是RejectedExecutionException？难道是因为我提前关闭了 TransferManager？

Answer 1

是否是因为我过早关闭了 TransferManager？

这确实是此行为的触发因素，但根本原因是 AWS SDK for Java，请参阅 Eric Milas 类似问题 Amazon AWS Java SDK TransferManager 问题：

TransferManager 包含一个 UploadMonitor，而 UploadMonitor 有一个
静态 ScheduledExecutorService 称为 timedThreadPool。
TransferManager.shutdownNow() 调用静态方法
UploadMonitor.shutdownNow() 调用
timedThreadPool.shutdownNow()。下次致电时
UploadMonitor.reschedule() 被创建，RejectedExecutionException 为
抛出。

他自己的回答引用了 AWS 团队对 TransferManager 线程问题 ，确认了此错误，并且已实施相应的修复： 版本 1.3.2）：

Amazon S3 TransferManager 类 已更新以解决两个错误：

从 InputStream 上传时可能会出现数据损坏的情况
在上传的某些部分遇到 IO 错误。干净地
当 shutdownNow() 被调用时关闭计时器线程
TransferManager 实例