创建一个一次只执行一次的 Celery 任务
我已经阅读了 Celery 文档中的相应文章（Cookbook :: 创建任务），但是我并不完全喜欢建议的解决方案。我有很长的运行任务（从 15 分钟到 1 小时，甚至可能更长），因此选择一些 LOCK_EXPIRE 设置似乎并不可靠：如果它太小，我会同时运行两个任务，如果它太大并且芹菜进程由于某种原因终止，我需要手动删除锁定缓存键。
我想到的想法是，我可以使用两种不同的锁：一种是用 fcntl.lockf 创建的（针对不同的 celery 工作进程），另一种是用 threading.Lock() 创建的（当任务在同一个 celery 进程中执行时） 。然而拥有两把锁似乎有点多余。
所以，我的问题是：解决我的问题最简单的解决方案是什么？它不必是跨平台的，并且只能支持Linux。
I already read the respective article in Celery docs (Cookbook :: Creating Tasks), however I didn't completely like the solution suggested. I have very long running tasks (from 15 minutes to 1 hour or possibly even more) and thus choosing some LOCK_EXPIRE setting doesn't seem reliable: if it turns out too small, I get two tasks running simultaneously and if it is too big and a celery process dies for some reason, I'll need to manually delete the locking cache key.
The idea that comes to my mind is that I could use two different locks: one created with fcntl.lockf (for different celery worker processes) and another one with threading.Lock() (when the task gets executed in the same celery process). However having two locks seems somewhat excessive.
So, my question is: what would be the simplest solution for my problem? It doesn't have to be cross-platform and can support only Linux.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
只需使用一个没有过期的锁，并创建一个定期“调查器检查一切是否正常的任务，也就是说，如果另一个任务意外死亡，它会修复锁定问题。另外，请查看此处。
Just use one lock without expiration, and create a periodic "investigator" Task that checks whether everything is ok, that is, if another Task died unexpectedly, it fixes the lock issue. As a plus, take a look here.