Full disclaimer that I work for Prefect but I think Prefect can indeed slot in nicely here where you have an API that hits Prefect's API to create_flow_runs to process that data that came in through the frontend. It also sounds like you need a decentralized solution where the compute happens on different machines. Prefect lets you do this by having multiple agents on those different machines, and then you can invoke flow runs that get picked up by the respective agents.
Yes, you are also right non Python tasks can be abstracted with containers and by using tasks to spin up containers. There is some degree of resource management offered by the agents because you can have them on multiple machines. If you decide to go to Kubernetes also for better resource management, Prefect also supports that and the Kubernetes agent can control the resources per container.
发布评论
评论（1）
完全免责声明，我为 Prefect 工作，但我认为 Prefect 确实可以很好地融入其中，您有一个 API 可以点击 Prefect 的 API 来 create_flow_runs 来处理通过前端传入的数据。听起来您还需要一个分散的解决方案，其中计算发生在不同的机器上。 Prefect 允许您通过在这些不同的计算机上拥有多个代理来实现此目的，然后您可以调用由相应代理拾取的流程运行。
是的，你也是对的，非 Python 任务可以通过容器进行抽象，并使用任务来启动容器。代理提供了某种程度的资源管理，因为您可以将它们放在多台计算机上。如果您决定使用 Kubernetes 也是为了更好的资源管理，Prefect 也支持这一点，并且 Kubernetes 代理可以控制每个容器的资源。
这可能是一个很好的 资源 为您服务
Full disclaimer that I work for Prefect but I think Prefect can indeed slot in nicely here where you have an API that hits Prefect's API to create_flow_runs to process that data that came in through the frontend. It also sounds like you need a decentralized solution where the compute happens on different machines. Prefect lets you do this by having multiple agents on those different machines, and then you can invoke flow runs that get picked up by the respective agents.
Yes, you are also right non Python tasks can be abstracted with containers and by using tasks to spin up containers. There is some degree of resource management offered by the agents because you can have them on multiple machines. If you decide to go to Kubernetes also for better resource management, Prefect also supports that and the Kubernetes agent can control the resources per container.
This might be a good resource for you