带有安全 tsv url 列表文件的 Google Cloud Storage Transfer

Question

我正在尝试在GCS存储桶上传输内容或公开提供的URL。

为此，我使用Google Cloud Storage Transfer API，它要求我执行两个步骤：

1. 创建一个包含我公共URL 列表的文件 .tsv 。
2. 创建传输（在此使用Python API）。

要启动脚本，我使用一个服务帐户，该帐户在包含 transfer.tsv 文件的存储对象admin许可以及水槽存储桶上。

当将 Transfer.tsv 文件上传到Internet上打开的存储桶时，我似乎只能使其起作用。

您是否知道是否有可能放在有担保的存储桶上，并允许创建转移的服务帐户？

到目前为止，我所有的尝试都会产生以下错误。

错误

PERMISSION_DENIED   1   
https://storage.googleapis.com/my-private-bucket/transfer.tsv   
Received HTTP error code 403.   

transper.tsv

TsvHttpData-1.0
https://image.shutterstock.com/image-photo/portrait-surprised-cat-scottish-straight-260nw-499196506.jpg

python脚本

from google.cloud import storage_transfer
from datetime import datetime

def create_one_time_http_transfer(
    project_id: str,
    description: str,
    list_url: str,
    sink_bucket: str,
):
    """Creates a one-time transfer job from Amazon S3 to Google Cloud
    Storage."""

    client = storage_transfer.StorageTransferServiceClient()

    # the same time creates a one-time transfer
    one_time_schedule = {"day": now.day, "month": now.month, "year": now.year}

    transfer_job_request = storage_transfer.CreateTransferJobRequest(
        {
            "transfer_job": {
                "project_id": project_id,
                "description": description,
                "status": storage_transfer.TransferJob.Status.ENABLED,
                "schedule": {
                    "schedule_start_date": one_time_schedule,
                    "schedule_end_date": one_time_schedule,
                },
                "transfer_spec": {
                    "http_data_source": storage_transfer.HttpData(list_url=list_url),
                    "gcs_data_sink": {
                        "bucket_name": sink_bucket,
                    },
                },
            }
        }
    )

    result = client.create_transfer_job(transfer_job_request)
    print(f"Created transferJob: {result.name}")

，我称之为功能

create_one_time_http_transfer(
        project_id="my-project-id",
        description="first transfer",
        list_url=tsv_url,
        sink_bucket="my-destination-bucket",
    )

Answer 1

找到了一种使它起作用的方法。

当我将 transfer.tsv 文件上传到存储时，我返回签名的URL 而不是 public URL

from datetime import datetime
from google.cloud import storage

def upload_to_storage(
    file_input_path: str, file_output_path: str, bucket_name: str
) -> str:
    gcs = storage.Client()

    # # Get the bucket that the file will be uploaded to.
    bucket = gcs.get_bucket(bucket_name)

    # # Create a new blob and upload the file's content.
    blob = bucket.blob(file_output_path)

    blob.upload_from_filename(file_input_path)

    return blob.generate_signed_url(datetime.now())

然后将此签名的URL传递到 create_one_time_http_transfer 上面提到的。

Answer 2

问题可能是 storage_transfer.StorageTransferServiceClient() 中的权限。创建一个角色来访问存储并将其附加到运行 Python 脚本的服务帐户。或者将您的凭据放在这里 storage_transfer.StorageTransferServiceClient(credentials=XXXX.json)