pandas to_parquet因GZIP而失败
我有一个有关pandas pd.to_parquet功能的问题，compression ='gzip'选项。 GZIP实用程序未识别使用此选项创建的文件。 我正在运行AWSS EC2实例，并具有深度学习基础AMI（Ubuntu 18.04）版本53 Python 3.6.9，Pandas 1.1.5。
保存的文件
df.to_parquet(path, engine='pyarrow', compression='gzip')
具有6159个字节。
就会抛出一个错误“不用gzip格式”
保存的文件上使用gzip，
df.to_parquet(path, engine='pyarrow', compression=None)
gzip -dv如果我在没有压缩的情况下 1511字节的大小，不用说，gzip -dv工作正常并还原文件。
我试图谷歌搜索，但什么也没想到。 任何帮助都将受到赞赏
I have a question regarding pandas pd.to_parquet function with the compression = 'gzip' option. Files created with this option are not recognized by gzip utility.
I'm running AWSs ec2 instance with Deep Learning Base AMI (Ubuntu 18.04) Version 53
python 3.6.9, pandas 1.1.5.
A file saved with
df.to_parquet(path, engine='pyarrow', compression='gzip')
has the size 6159 bytes.
gzip -dv throws an error " not in gzip format"
If I used gzip on the file saved without compression, i.e. first run
df.to_parquet(path, engine='pyarrow', compression=None)
and then gzip the .parquet file, the resulting .parquet.gz file has the size of 1511 bytes, and, needless to say, gzip -dv works just fine and restores the file.
I tried to google it, but came up with nothing.
Any help is appreciated
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
到Gzip的Pandas Parquet不会Gzip the Parquet，而是制作一个内部的GZZ parquet，该文件的标题具有镶木木字节（不是GZ的字节！）。
如果您想gzip，则必须使用 gzip python库镶木。如果文件不太大，请在字节对象上创建镶木，因为它会更快。
另外，压缩水平时间与大小是对数，通常9非常慢，比7的优势几乎没有
。不认识魔术字节。您也应该在同一库中解开它们。
pandas parquet to GZip does not GZip the parquet, but makes an internally GZipped parquet, the header of the file has the Parquet magic bytes (not the GZ ones!).
If you want to GZip you will have to use GZip python library after creating the Parquet. If the file is not too big, create the parquet on a BytesIO object, as it will be much faster.
Also, compression level time vs size is logarithmic, usually 9 is very slow and offers little advantage over 7.
Note that if I recall good, similarly pandas cannot load an externally GZ'd parquet, as it does not recognize the magic bytes. You should unGZip them also, possible with the same library.