如何解压缩2GB+在AWS lambda上使用python的文件?

发布于 2025-01-22 23:43:18 字数 2308 浏览 4 评论 0原文

我正在尝试将位于lambda(Python)上S3上的文件解压缩。一切都很好,直到原始文件超过2GB,那时我只得到“不支持这种压缩方法”。

我主要尝试使用Zipfile和Zipfile39,没有运气。...还有其他几个软件包,但我得到了类似的结果。

尝试使用Zipfile解压缩内容时

# Using ZipFile -- No problems here... (I have tried with/without 
# compression, compresslevel, allowZip64 as well as different values and 
# same output)
zip_content = zipfile.ZipFile(zip_content, 
                              'r', 
                              compression=8, # There is no 9 here
                              compresslevel=9,
                              allowZip64=True)
for filename in zip_content.namelist():
    print(zip_content.getinfo(filename))
    # Printing the zip_info I get: <ZipInfo filename='test_file.csv' 
    # compress_type=deflate64 external_attr=0x20 file_size=2505399449 
    # compress_size=853276056>

,尝试使用Zipfile39解压缩内容时,

# Using ZipFile39 -- No problems here... (I have tried with/without compression,
# compresslevel, allowZip64 as well as different values and same output)
# But something interesting is that I cannot use 'ZIP_DEFLATED64' as it says that 
# attribute cannot be found on ZipFile39, but it is there (using 9 also fails).
zip_content = zipfile39.ZipFile(zip_content, 
                                'r',
                                compression=9, 
                                compresslevel=9, 
                                allowZip64=True)
for filename in zip_content.namelist():
    print(zip_content.getinfo(filename))
    # Printing the zip_info I get: <ZipInfo filename='test_file.csv' 
    # compress_type=deflate64 external_attr=0x20 file_size=2505399449 
    # compress_size=853276056>

例外情况会在尝试时出现:

# Writing to S3 <<---- HERE is where the exception occurs, always: 
# "That compression method is not supported"
# That is happening on the "open" zip_content.open (I have tried with the 
# force_zip64 and without it)
zip_content.open(zip_info, force_zip64=True)

我已经看到了其他有关同一主题的问题,但是我找不到答案,我可以使它能够正常工作,到目前为止,我已经尝试过,我已经尝试过zipfile-deflate64(有周期性参考错误),stream_inflate(未对文件解压缩),stream_unzip(未对文件解压缩)。

很少有重要说明:

  1. 这仅在AWS lambda上发生(本地工作没有问题 - Windows笔记本电脑)。
  2. 它与lambda上的内存/空间无关,此时不超过20%。

任何想法/帮助/建议将不胜感激。

谢谢

I am trying to decompress a file which is located on S3 from a lambda (Python). Everything works great until the original file is over 2GB, at that point, I only get "That compression method is not supported".

I have mainly tried with ZipFile and and ZipFile39 and no luck.... there are a couple of other packages but I got similar results.

When try to unzip the content using ZipFile

# Using ZipFile -- No problems here... (I have tried with/without 
# compression, compresslevel, allowZip64 as well as different values and 
# same output)
zip_content = zipfile.ZipFile(zip_content, 
                              'r', 
                              compression=8, # There is no 9 here
                              compresslevel=9,
                              allowZip64=True)
for filename in zip_content.namelist():
    print(zip_content.getinfo(filename))
    # Printing the zip_info I get: <ZipInfo filename='test_file.csv' 
    # compress_type=deflate64 external_attr=0x20 file_size=2505399449 
    # compress_size=853276056>

when try to unzip the content using ZipFile39

# Using ZipFile39 -- No problems here... (I have tried with/without compression,
# compresslevel, allowZip64 as well as different values and same output)
# But something interesting is that I cannot use 'ZIP_DEFLATED64' as it says that 
# attribute cannot be found on ZipFile39, but it is there (using 9 also fails).
zip_content = zipfile39.ZipFile(zip_content, 
                                'r',
                                compression=9, 
                                compresslevel=9, 
                                allowZip64=True)
for filename in zip_content.namelist():
    print(zip_content.getinfo(filename))
    # Printing the zip_info I get: <ZipInfo filename='test_file.csv' 
    # compress_type=deflate64 external_attr=0x20 file_size=2505399449 
    # compress_size=853276056>

The exception comes when trying:

# Writing to S3 <<---- HERE is where the exception occurs, always: 
# "That compression method is not supported"
# That is happening on the "open" zip_content.open (I have tried with the 
# force_zip64 and without it)
zip_content.open(zip_info, force_zip64=True)

I have seen other question about the same topic but I can't find an answer I can make it work so far, I have tried zipfile-deflate64 (got a cyclical reference error), stream_inflate (did not decompressed the file), stream_unzip (did not decompressed the file).

Few important notes:

  1. This is happening only on the AWS Lambda (locally works without issues - windows laptop).
  2. It is not about the memory/space on the lambda, it is not used more than 20% at this point.

Any idea/help/suggestion will be appreciated.

Thanks

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。
列表为空,暂无数据
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文