我引用了下面显示的文章,该文章指示Pyflink在Python解释器和JVM上的工作方式。
我无法弄清楚他们是否在几个过程中执行工作,等等...
所以我留下了我的解释他们的内部系统在这里,想请您纠正它。
我的解释:
- 用户在Python中写入程序。
- Python解释器制作一个过程(过程1),该过程是控制与JVM Gatewayserver(过程2)的连接的过程。
- 进程1将使流媒体形成工作图,并将其通过套接字提交给JVM Gateway Server。
它也用于传递输入数据并接收输出数据。
- 下一步,JobManager(Process 3)计划实际执行并分配足够的任务插槽。(进程4),这是一个执行图,该执行图已对其进行平行并准备执行。
- 有几个需要致电Python,因为某些操作员使用UserDefined Function或Lambdafunc等。在这种情况下,操作员(例如地图,flatmap)(进程4)将状态和一些相关信息提交给Python Process(过程5),该进程是为执行UDF或Lambda等... JVM的操作员。
I referenced the article shown below which instruct how pyflink works on python interpreter and jvm.
https://www.alibabacloud.com/blog/the-flink-ecosystem-a-quick-start-to-pyflink_596150
And I couldn't figure out that whether they execute a job across the several processes or not,etc...
So I left my interpretation of their internal system here and want to ask you to correct it.
My Interpretation:
- User writes program in python.
- Python interpreter make an process(Process 1) which accounts for controlling connection to JVM gatewayserver(Process 2).
- Process 1 will make job graph of streaming and submit it to JVM gateway server through socket.
Which is also used for passing input data and receive output data.
- Next step, a jobmanager(Process 3) plan an actual execution and allocate sufficient task slots.(Processes 4) Then here is an execution graph which is parallerized and ready to execute.
- There are several needs to call python because some operator use UserDefinedFunction or LambdaFunc etc.... which are not supported in java. In such situation, the operator(e.g. map, flatmap)(Processes 4) submit state and some related information to python process(Process 5) which is made for executing UDF or Lambda etc... and the result of it will be returned to the operator in JVM.
发布评论
评论(1)
我建议您观看
pyflink:使用Python Language
在上一个Flink Forward Conference中构建Flink应用程序以获取答案。您可以在I can recommend watching the
PyFlink: Building Flink applications with Python language
talk from the last Flink Forward conference for your answers. You can find it at https://www.youtube.com/watch?v=SbLBtL6IPE4