Apache Beam Fileio写下压缩文件
我想知道是否可以使用Apache Beam Python SDK的Fileio模块编写压缩文件。目前，我使用模块将文件写入GCP存储桶：
_ = (logs | 'Window' >> beam.WindowInto(window.FixedWindows(60*60))
| 'Convert to JSON' >> beam.ParDo(ConvertToJson())
| 'Write logs to GCS file' >> fileio.WriteToFiles(path = gsc_output_path, shards=1, max_writers_per_bundle=0))
压缩将有助于最大程度地降低存储成本。
根据
我对此是对的，还是有人知道该怎么做？
谢谢你！
I would like to know if it's possible to write compressed files using the fileio module from Apache Beam, Python SDK. At the moment I am using the module to write files to a GCP bucket:
_ = (logs | 'Window' >> beam.WindowInto(window.FixedWindows(60*60))
| 'Convert to JSON' >> beam.ParDo(ConvertToJson())
| 'Write logs to GCS file' >> fileio.WriteToFiles(path = gsc_output_path, shards=1, max_writers_per_bundle=0))
Compression would help in minimizing storage costs.
According to this doc and comment inside class _MoveTempFilesIntoFinalDestinationFn, developers still need to implement handling of compression.
Am I right about this or does someone know how to do it?
Thank you!
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
这是正确的。
虽然有开放的FRS：
目前，您可以编写DOFN：读取最终文件 - ＆gt;压缩 - ＆gt;写下压缩的最终文件并删除原始最终文件。
This is correct.
Though there are open FRs:
At the moment, you can write a DoFn: read the final files -> compress -> write the compressed final files and delete original final files.