循环通过压缩GZIP文件抛出“错误[errno 2]没有这样的文件或目录：＆＃x27; part-r-00001.gz;＆quot;在第二次迭代 - python

发布于 2025-02-08 16:47:44 字数 1482 浏览 3 评论 0原文

我正在通过S3存储桶中的多个文件循环。第一次迭代效果很好，但是一旦跳到下一个迭代，我会收到“错误[errno 2]否此类文件或目录：'part-r-r-00001.gz'”。（正确访问了part-r-00000.gz）

我不确定为什么在存储桶中找到该文件。

这是代码：

BUCKET = 'bucket'
PREFIX = 'path'

now = datetime.utcnow()
today = (now - timedelta(days=2)).strftime('%Y-%m-%d')
folder_of_the_day = PREFIX + today + '/'
logger.info("map folder: %s", folder_of_the_day)

client = boto3.client('s3')
response = client.list_objects_v2(Bucket=BUCKET, Prefix=folder_of_the_day)
for content in response.get('Contents', []):
    bucket_file = os.path.split(content["Key"])[-1]
    if bucket_file.endswith('.gz'):
        logger.info("----- starting with file: %s -----", bucket_file)
        try:
            with gzip.open(bucket_file, mode="rt") as file:
                for line in file:
                    //do something

        except Exception as e:
            logger.error(e)
            logger.critical("Failed to open file!")
            sys.exit(4)

第二轮执行后，这是输出：

2022-06-18 12：14：48,027 [root]信息------从文件开始： part-r-00001.gz ----- 2022-06-18 12：14：48,028 [root]错误[errno 2] 没有这样的文件或目录：'part-r-00001.gz'

update 根据评论，我将代码更新为适当的GZIP方法，但仍然存在错误。完成第一次迭代后，就找不到第二个文件。

这是更新的代码：

try:
    with gzip.GzipFile(bucket_file) as gzipfile:
        decompressed_content = gzipfile.read()
        for line in decompressed_content.splitlines():
            //do something
            break

原文

I am looping through multiple files within an s3 bucket. The first iteration works perfectly fine, but once jumping to the next I receive an "ERROR [Errno 2] No such file or directory: 'part-r-00001.gz'". (part-r-00000.gz was accessed correctly)

I am not sure why the file is not found as it is available in the bucket.

This is the code:

BUCKET = 'bucket'
PREFIX = 'path'

now = datetime.utcnow()
today = (now - timedelta(days=2)).strftime('%Y-%m-%d')
folder_of_the_day = PREFIX + today + '/'
logger.info("map folder: %s", folder_of_the_day)

client = boto3.client('s3')
response = client.list_objects_v2(Bucket=BUCKET, Prefix=folder_of_the_day)
for content in response.get('Contents', []):
    bucket_file = os.path.split(content["Key"])[-1]
    if bucket_file.endswith('.gz'):
        logger.info("----- starting with file: %s -----", bucket_file)
        try:
            with gzip.open(bucket_file, mode="rt") as file:
                for line in file:
                    //do something

        except Exception as e:
            logger.error(e)
            logger.critical("Failed to open file!")
            sys.exit(4)

Once executed for the second round, this is the output:

2022-06-18 12:14:48,027 [root] INFO ----- starting with file:
part-r-00001.gz ----- 2022-06-18 12:14:48,028 [root] ERROR [Errno 2]
No such file or directory: 'part-r-00001.gz'

Update
Based on the comment I updated my code to a proper gzip method, but still the error remains. Once the first iteration is done, the second file is not being found.

This is the updated code:

try:
    with gzip.GzipFile(bucket_file) as gzipfile:
        decompressed_content = gzipfile.read()
        for line in decompressed_content.splitlines():
            //do something
            break

分享到QQ

分享到微博