如何解压缩2GB+在AWS lambda上使用python的文件?
我正在尝试将位于lambda(Python)上S3上的文件解压缩。一切都很好,直到原始文件超过2GB,那时我只得到“不支持这种压缩方法”。
我主要尝试使用Zipfile和Zipfile39,没有运气。...还有其他几个软件包,但我得到了类似的结果。
尝试使用Zipfile解压缩内容时
# Using ZipFile -- No problems here... (I have tried with/without
# compression, compresslevel, allowZip64 as well as different values and
# same output)
zip_content = zipfile.ZipFile(zip_content,
'r',
compression=8, # There is no 9 here
compresslevel=9,
allowZip64=True)
for filename in zip_content.namelist():
print(zip_content.getinfo(filename))
# Printing the zip_info I get: <ZipInfo filename='test_file.csv'
# compress_type=deflate64 external_attr=0x20 file_size=2505399449
# compress_size=853276056>
,尝试使用Zipfile39解压缩内容时,
# Using ZipFile39 -- No problems here... (I have tried with/without compression,
# compresslevel, allowZip64 as well as different values and same output)
# But something interesting is that I cannot use 'ZIP_DEFLATED64' as it says that
# attribute cannot be found on ZipFile39, but it is there (using 9 also fails).
zip_content = zipfile39.ZipFile(zip_content,
'r',
compression=9,
compresslevel=9,
allowZip64=True)
for filename in zip_content.namelist():
print(zip_content.getinfo(filename))
# Printing the zip_info I get: <ZipInfo filename='test_file.csv'
# compress_type=deflate64 external_attr=0x20 file_size=2505399449
# compress_size=853276056>
例外情况会在尝试时出现:
# Writing to S3 <<---- HERE is where the exception occurs, always:
# "That compression method is not supported"
# That is happening on the "open" zip_content.open (I have tried with the
# force_zip64 and without it)
zip_content.open(zip_info, force_zip64=True)
我已经看到了其他有关同一主题的问题,但是我找不到答案,我可以使它能够正常工作,到目前为止,我已经尝试过,我已经尝试过zipfile-deflate64(有周期性参考错误),stream_inflate(未对文件解压缩),stream_unzip(未对文件解压缩)。
很少有重要说明:
- 这仅在AWS lambda上发生(本地工作没有问题 - Windows笔记本电脑)。
- 它与lambda上的内存/空间无关,此时不超过20%。
任何想法/帮助/建议将不胜感激。
谢谢
I am trying to decompress a file which is located on S3 from a lambda (Python). Everything works great until the original file is over 2GB, at that point, I only get "That compression method is not supported".
I have mainly tried with ZipFile and and ZipFile39 and no luck.... there are a couple of other packages but I got similar results.
When try to unzip the content using ZipFile
# Using ZipFile -- No problems here... (I have tried with/without
# compression, compresslevel, allowZip64 as well as different values and
# same output)
zip_content = zipfile.ZipFile(zip_content,
'r',
compression=8, # There is no 9 here
compresslevel=9,
allowZip64=True)
for filename in zip_content.namelist():
print(zip_content.getinfo(filename))
# Printing the zip_info I get: <ZipInfo filename='test_file.csv'
# compress_type=deflate64 external_attr=0x20 file_size=2505399449
# compress_size=853276056>
when try to unzip the content using ZipFile39
# Using ZipFile39 -- No problems here... (I have tried with/without compression,
# compresslevel, allowZip64 as well as different values and same output)
# But something interesting is that I cannot use 'ZIP_DEFLATED64' as it says that
# attribute cannot be found on ZipFile39, but it is there (using 9 also fails).
zip_content = zipfile39.ZipFile(zip_content,
'r',
compression=9,
compresslevel=9,
allowZip64=True)
for filename in zip_content.namelist():
print(zip_content.getinfo(filename))
# Printing the zip_info I get: <ZipInfo filename='test_file.csv'
# compress_type=deflate64 external_attr=0x20 file_size=2505399449
# compress_size=853276056>
The exception comes when trying:
# Writing to S3 <<---- HERE is where the exception occurs, always:
# "That compression method is not supported"
# That is happening on the "open" zip_content.open (I have tried with the
# force_zip64 and without it)
zip_content.open(zip_info, force_zip64=True)
I have seen other question about the same topic but I can't find an answer I can make it work so far, I have tried zipfile-deflate64 (got a cyclical reference error), stream_inflate (did not decompressed the file), stream_unzip (did not decompressed the file).
Few important notes:
- This is happening only on the AWS Lambda (locally works without issues - windows laptop).
- It is not about the memory/space on the lambda, it is not used more than 20% at this point.
Any idea/help/suggestion will be appreciated.
Thanks
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论