当我的日程安排同时具有基于事件和基于时间的触发条件时,会发生什么情况?

发布于 2025-01-15 08:41:18 字数 307 浏览 3 评论 0原文

我们有以下场景。管道完整构建是由时间表触发的,时间表本身是由以下两个条件之一触发的:要么有十几个数据集更新了数据,要么是凌晨 3:15。我们有时间触发器来确保管道不会太晚开始构建,希望在上游数据被消耗时它已经更新。构建需要 2 到 5 小时。

今天凌晨2点左右触发了日程,并在6点45分成功完成。到目前为止,一切都很好。

但随后,同一计划触发了第二次构建,这一次是意外且不受欢迎的。

有没有可能,尽管已经执行了构建,但基于时间的触发器也触发了构建,并且它在正在进行的构建之后“排队”?只是想了解一下,以便我们可以调整触发条件。

预先感谢您的见解

We have the following scenario. A pipeline full build is triggered by a schedule, that is itself triggered by one of these two conditions: either a dozen datasets have updated data, or it's 3:15 in the morning. We have the time trigger to ensure the pipeline doesn't start building too late, hoping that by the time the upstream data is consumed it has been updated. The build takes between 2 and 5 hours.

Today, the schedule was triggered at around 2 am, and completed successfully at 6:45. So far so good.

But then, a second build was triggered by the same schedule, this time unexpected and unwanted.

Could it be, that the time based trigger also triggered a build, despite a build already being executed, and that it was "queued" after the ongoing one? Just trying to understand, so that we can tweak the triggering conditions.

Thanks in advance for your insights

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(2

忆离笙 2025-01-22 08:41:18

这个问题可以归结为以下问题:(a) 我的计划是否会触发以及 (b) 当计划触发时,build2 将如何解析构建:

(a) 你是对的。用您自己的话来说,基于时间的触发器也将触发构建,尽管构建已经被执行,并且它在正在进行的构建之后“排队”。

本质上,您可以期望运行计划的另一个实例。

请注意,您可以在 xform 中设计一些自定义逻辑来中止构建 - 如果构建已经在执行,则可以告诉计划不要重新触发。

(b) 可能还值得澄清的是,除非按计划启用强制构建,否则构建服务将永远不会重新构建与其所有祖先都是最新的数据集,因此如果计划只是 A -> ; B-> C 和 A 在 2:30 更新,计划在 3:00 运行如果 B 和 C 不陈旧,则不会重新构建它们

This question can be boiled down to the following: (a) will my schedule trigger and (b) what will build2 resolve the build to be when the schedule does trigger:

(a) You are correct. In your own words, the time based trigger will also trigger a build, despite a build already being executed, and that it was "queued" after the ongoing one.

Essentially, you can expect another instance of the schedule to run.

Do note you can design some custom logic in the xform to abort the build - for a way to tell the schedule to not retrigger if a build is already being executed.

(b) Might also be worth clarifying that unless Force build is enabled on the schedule, the build service will never re-build datasets that are up-to-date with all of its ancestors so if the schedule is simply A -> B -> C and A updates at 2:30, the scheduled run at 3:00 isn't going to re-build B and C if they're not stale

演出会有结束 2025-01-22 08:41:18

请注意,触发器仅确定计划的构建是否应尝试运行。如果触发器确实触发,则目标数据集的新鲜度将决定构建是否真正运行。因此,触发器可能会触发,但运行最终会被忽略,因为所有目标数据集都是新鲜的。

正是由于这个原因,您可能会看到触发的运行在正在进行的运行之后“排队”,但是该排队的运行是否真正构建完全取决于所有输出数据集是否都是最新的。重要的是要仔细研究输入数据集上的触发器以及按此类计划输出数据集的陈旧性,以避免常见的混淆。

Note that triggers only determine whether a scheduled build should attempt to run. If the trigger does fire, then the target datasets’ freshness will determine whether the build will actually run or not. So, it’s possible for a trigger to fire but the run would ultimately be ignored since all target datasets are fresh.

It’s for this reason that you might see a triggered runs “queued” after an ongoing one, but whether that queued run actually does build depends entirely on if all the output datasets are up-to-date. It’s important to closely investigate the triggers on your input datasets, and the staleness of your output datasets on schedules like these in order to avoid common confusions.

~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文