如何使用AWS CLI S3API命令执行多部分上传？

发布于 2025-01-31 11:57:56 字数 1838 浏览 3 评论 0 原文

据我了解，S3对象存储应该能够恢复不完整的多部分上传。我正在尝试针对本地S3存储系统进行测试。

我知道AWS CLI将通过 AWS S3 CP 自动在较大文件上执行多部分上传，但是如何使用 AWS S3API 执行相同的操作？

我已经阅读了文档，并且知道这是一个三个步骤：

运行 aws s3api create-multipart-upload （ docs ）
运行 aws aws s3api s3api upload-part-part-part-part-part-part-part （ docs ）
通过运行重建对象（）

我试图针对7GB的文件执行这些步骤。运行第一个命令后，我收到了所有后续命令所需的预期 upload-id ：

aws --profile foo --endpoint-url=https://endpoint:9003 s3api create-multipart-upload --bucket mybucket1 --key 'some7gfile.bin/01' --output json
{
    "Bucket": "mybucket1", 
    "UploadId": "41a1462d-0d23-47f6-83aa-377e7aedbb8a", 
    "Key": "some7gfile.bin/01"
}

我假设 01 some some7gfile.bin/01 <的部分/代码>表示第一部分，但事实并非如此。 01 是什么意思？是任意的吗？

当我尝试运行第二个上传part-part 命令时，我收到一个错误：

aws --profile foo --endpoint-url=https://endpoint:9003 s3api upload-part --bucket mybucket1 --key 'some7gfile.bin/01' --part-number 1 --body part01 --upload-id "41a1462d-0d23-47f6-83aa-377e7aedbb8a" --output json

Error parsing parameter '--body': Blob values must be a path to a file.

在运行上传part-part-part-part 步骤之前，是否必须将源文件分为不同的部分？如果是这样，最有效的方法是什么？

原文

From my understanding, an S3 object-store should be able to resume incomplete multipart uploads. I am trying to test this against a local S3 storage system.

I'm aware the AWS CLI will automatically perform a multipart upload on larger files via aws s3 cp, but how do I perform the same operation using aws s3api?

I've read the documentation and know that it's a three step process:

Run aws s3api create-multipart-upload (docs)
Run aws s3api upload-part (docs)
Finish by running aws s3api complete-multipart-upload which reconstructs the object (docs)

I attempted to perform these steps against a file that was 7GB in size. After running the first command, I received the expected upload-id which is required for all subsequent commands:

aws --profile foo --endpoint-url=https://endpoint:9003 s3api create-multipart-upload --bucket mybucket1 --key 'some7gfile.bin/01' --output json
{
    "Bucket": "mybucket1", 
    "UploadId": "41a1462d-0d23-47f6-83aa-377e7aedbb8a", 
    "Key": "some7gfile.bin/01"
}

I assumed the 01 portion of some7gfile.bin/01 denoted the first part, but that doesn't appear to be the case. What does the 01 mean? Is it arbitrary?

When I tried running the second upload-part command, I received an error:

aws --profile foo --endpoint-url=https://endpoint:9003 s3api upload-part --bucket mybucket1 --key 'some7gfile.bin/01' --part-number 1 --body part01 --upload-id "41a1462d-0d23-47f6-83aa-377e7aedbb8a" --output json

Error parsing parameter '--body': Blob values must be a path to a file.

Does the source file have to be split into different parts prior to running the upload-part step? If so, what's the most efficient way to do this?