pyflink的内部体系结构

发布于 2025-01-26 18:55:54 字数 774 浏览 4 评论 0 原文

我引用了下面显示的文章,该文章指示Pyflink在Python解释器和JVM上的工作方式。

我无法弄清楚他们是否在几个过程中执行工作,等等...

所以我留下了我的解释他们的内部系统在这里,想请您纠正它。

我的解释:

  1. 用户在Python中写入程序。
  2. Python解释器制作一个过程(过程1),该过程是控制与JVM Gatewayserver(过程2)的连接的过程。
  3. 进程1将使流媒体形成工作图,并将其通过套接字提交给JVM Gateway Server。 它也用于传递输入数据并接收输出数据。
  4. 下一步,JobManager(Process 3)计划实际执行并分配足够的任务插槽。(进程4),这是一个执行图,该执行图已对其进行平行并准备执行。
  5. 有几个需要致电Python,因为某些操作员使用UserDefined Function或Lambdafunc等。在这种情况下,操作员(例如地图,flatmap)(进程4)将状态和一些相关信息提交给Python Process(过程5),该进程是为执行UDF或Lambda等... JVM的操作员。

I referenced the article shown below which instruct how pyflink works on python interpreter and jvm.
https://www.alibabacloud.com/blog/the-flink-ecosystem-a-quick-start-to-pyflink_596150

And I couldn't figure out that whether they execute a job across the several processes or not,etc...

So I left my interpretation of their internal system here and want to ask you to correct it.

My Interpretation:

  1. User writes program in python.
  2. Python interpreter make an process(Process 1) which accounts for controlling connection to JVM gatewayserver(Process 2).
  3. Process 1 will make job graph of streaming and submit it to JVM gateway server through socket.
    Which is also used for passing input data and receive output data.
  4. Next step, a jobmanager(Process 3) plan an actual execution and allocate sufficient task slots.(Processes 4) Then here is an execution graph which is parallerized and ready to execute.
  5. There are several needs to call python because some operator use UserDefinedFunction or LambdaFunc etc.... which are not supported in java. In such situation, the operator(e.g. map, flatmap)(Processes 4) submit state and some related information to python process(Process 5) which is made for executing UDF or Lambda etc... and the result of it will be returned to the operator in JVM.

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(1

站稳脚跟 2025-02-02 18:55:54

我建议您观看 pyflink:使用Python Language 在上一个Flink Forward Conference中构建Flink应用程序以获取答案。您可以在

I can recommend watching the PyFlink: Building Flink applications with Python language talk from the last Flink Forward conference for your answers. You can find it at https://www.youtube.com/watch?v=SbLBtL6IPE4

~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文