Apache Beam FiredWindow不做GroupByKey转换后做任何事情

发布于 2025-01-26 17:07:55 字数 717 浏览 5 评论 0原文

我构建了一条管道，该管道从Confluent Kafka读取它处理记录，然后使用侧面输出将其分成被拒绝和批准的PCollections，然后批准的PCollections将其写入BigQuery，但我想坚持批准的记录并将其写入文件中在GCS上。

该代码是：

windowing=(aproved
| 'Create_window'  >> beam.WindowInto(window.FixedWindows(60))
| 'AddKey' >> beam.Map(lambda record: (none,record))
| 'GBK' >> beam.GroupByKey()
| 'remove_key' >> beam.FlatMap(ret_key)
| 'AddTimeStamp' >> beam.Map(lambda record: beam.window.TimestampValue(record,time.time()))
| 'Write' >> WriteToFiles(path=MY_BUCKET,file_naming=destination_prefix_naming('.ppl'))
)

当我测试它从文件读取并使用Direct Runner时，这起作用，但是当我使用dataflow和流式传输时，GroupByKey转换后没有做任何事情，它在图上说添加了20个元素，但是下一个转换（'remove_key'）在那之后再也没有元素

原文

I built a pipeline which reads from confluent kafka it processes the records and then use side outputs to split them into rejected and approved pcollections, then the approved pcollections gets written to bigquery, but I want to persist the approved records and write them into a file on gcs.

The code is:

windowing=(aproved
| 'Create_window'  >> beam.WindowInto(window.FixedWindows(60))
| 'AddKey' >> beam.Map(lambda record: (none,record))
| 'GBK' >> beam.GroupByKey()
| 'remove_key' >> beam.FlatMap(ret_key)
| 'AddTimeStamp' >> beam.Map(lambda record: beam.window.TimestampValue(record,time.time()))
| 'Write' >> WriteToFiles(path=MY_BUCKET,file_naming=destination_prefix_naming('.ppl'))
)

This works when I test it reading from a file and using direct runner, but when I use dataflow and streaming it just doesn't do anything after the GroupByKey transform, it says on the graph that 20 element were added, but the next transform ('remove_key') never gets an element after that

分享到QQ

分享到微博