是否可以使用 prefect 按不同的时间表运行不同的任务?

发布于 2025-01-19 02:51:24 字数 401 浏览 4 评论 0原文

我正在以各个州的身份移动我的第一步,并且正在尝试看看它的自由度是多少。为此,我正在研究在同一Python过程中,Perfect是否支持在不同的时间表上运行不同的任务。例如,任务A可能必须每5分钟运行一次,而任务B可能每天使用CRON调度程序两次。

在我看来,时间表与a flow 相关联,而不是与任务相关联,因此要执行以上操作,一个人必须创建两个不同的单任务 flow 有自己的时间表。但是,即使这样,鉴于运行流是一个阻止操作,我也看不出如何同时“启动”两个流(或伪续,我完全意识到这些流不会在单独的线程上执行)。

是否有一种内置的方式可以使任务在其独立时间表上运行?我的印象是,是实现这一目标的一种方法,但是鉴于我在州长方面的经验有限,我完全缺少它。

非常感谢任何指示。

I'm moving my first steps with prefect, and I'm trying to see what its degrees of freedom are. To this end, I'm investigating whether prefect supports running different tasks on different schedules in the same python process. For example, Task A might have to run every 5 minutes, while Task B might run twice a day with a Cron scheduler.

It seems to me that schedules are associated with a Flow, not with a task, so to do the above, one would have to create two distinct one-task Flows, each with its own schedule. But even as that, given that running a flow is a blocking operation, I can't see how to "start" both flows concurrently (or pseudo-concurrently, I'm perfectly aware the flows won't execute on separate threads).

Is there a built-in way of getting the tasks running on their independent schedules? I'm under the impression that there is a way to achieve this, but given my limited experience with prefect, I'm completely missing it.

Many thanks in advance for any pointers.

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(2

韬韬不绝 2025-01-26 02:51:24

您是对的,时间表与流量而非任务相关联,因此添加时间表的唯一位置是流程。如果您仅使用开源县核心核心,则运行流量是阻止操作。对于生产用例,建议使用Perfect Cloud或Prefect Server运行流动。云是托管产品,服务器是您自己托管的。请注意,云具有非常宽敞的自由层。

使用后端时,您将使用将在新过程中启动流程运行的代理。这不会阻止。

首先使用后端,您可以检查 docs

You are right that schedules are associated with Flows and not Tasks, so the only place to add a schedule is a Flow. Running a Flow is a blocking operation if you are using the open source Prefect core only. For production use cases, it's recommended running your Flows against Prefect Cloud or Prefect Server. Cloud is the managed offering and Server is when you host it yourself. Note that Cloud has a very generous free tier.

When using a backend, you will use an agent that will kick off the flow run in a new process. This will not be blocking.

To start with using a backend, you can check the docs here

亣腦蒛氧 2025-01-26 02:51:24

此省级话语主题讨论了一个非常相似的问题,并展示了如何使用流动流量编排图案来解决它。

接近它的一种方法是利用缓存,以避免比主要流程相比需要低频调度的某些任务的重新签约。

This Prefect Discourse topic discusses a very similar problem and shows how you could solve it using a flow-of-flows orchestrator pattern.

One way to approach it is to leverage Caching to avoid recomputation of certain tasks that require lower-frequency scheduling than the main flow.

~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文