排队系统 - 启动多个工作人员的好方法是什么？
- 您如何为面向队列的系统设置一个或多个工作脚本？
- 您如何根据需要安排启动（并在必要时重新启动）工作脚本？ （我正在考虑诸如 init.d/、基于 Ruby 的“god”、DJB 的 Daemontools 等工具）
我正在开发一个异步队列/工作系统，在本例中使用 PHP 和 Worker。 BeanstalkdD（尽管实际的语言和守护进程并不重要）。 任务本身并不太难 - 将包含命令和参数的数组编码为 JSON，以便通过 Beanstalkd 守护进程进行传输，在工作脚本中拾取它们以根据需要执行操作。
还有许多其他类似的队列/工作人员设置，例如 Starling、Gearman， Amazon 的 SQS 和其他更面向“企业”的系统，例如 IBM 的 MQ 和 RabbitMQ。 如果您运行 Gearman 或 SQS 之类的东西 - 您如何启动和控制工作池？ 问题在于最初的工作人员启动，然后能够添加额外的额外工作人员，随意关闭它们（尽管我可以通过队列发送消息来关闭它们 - 只要某些“观察者”不会自动关闭）重新启动它们）。 这不是一个 PHP 问题，它是关于设置一个或多个进程在启动时运行的直接 Unix 进程，或者向池中添加更多工作线程的问题。
用于循环脚本的 bash 脚本 已就位 - 这将调用 PHP 脚本，然后该脚本从队列，偶尔退出以便能够自行清理（它也可以在失败时暂停几秒钟，或通过计划的事件暂停）。 这工作得很好，并且在此基础上构建工作进程一点也不难。
获得一个好的工作控制器系统涉及灵活性，在机器启动时自动启动一两个，并且能够在队列繁忙时从命令行添加更多，在不再需要时关闭额外的。
- How have you set-up one or more worker scripts for queue-oriented systems?
- How do you arrange to startup - and restart if necessary - worker scripts as required? (I'm thinking about such tools as init.d/, Ruby-based 'god', DJB's Daemontools, etc, etc)
I'm developing an asynchronous queue/worker system, in this case using PHP & BeanstalkdD (though the actual language and daemon isn't important). The tasks themselves are not too hard - encoding an array with the commands and parameters into JSON for transport through the Beanstalkd daemon, picking them up in a worker script to action them as required.
There are a number of other similar queue/worker setups out there, such as Starling, Gearman, Amazon's SQS and other more 'enterprise' oriented systems like IBM's MQ and RabbitMQ. If you run something like Gearman, or SQS - how do you start and control the worker pool? The questions is on the initial worker startup, and then being able to add additional extra workers, shutting them down at will (though I can send a message through the queue to shut them down - as long as some 'watcher' won't automatically restart them). This is not a PHP problem, it's about straight Unix processes of setting up one or more processes to run on startup, or adding more workers to the pool.
A bash script to loop a script is already in place - this calls the PHP script which then collects and runs tasks from the queue, occasionally exiting to be able to clean itself up (it can also pause a few seconds on failure, or via a planned event). This works fine, and building the worker processes on top of that won't be very hard at all.
Getting a good worker controller system is about flexibility, starting one or two automatically on a machine start, and being able to add a couple more from the command line when the queue is busy, shutting down the extras when no longer required.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（4）
我一直在帮助一位朋友，他正在开发一个项目，该项目涉及一个基于 Gearman 的队列，该队列将各种异步作业分派到多个服务器池上的各种 PHP 和 C 守护进程。
由于 /etc/init.d/ 中的简单 shell 脚本以及如下命令，worker 的行为就像经典的 unix/linux 守护进程一样：
invoke-rc.d myWorker start|stop|restart|reload< /code>这种机制简单高效。 由于它依赖于标准的 Linux 功能，即使对应用程序了解有限的人也可以启动或停止守护进程，只要他们知道它如何被称为系统明智的（在上面的示例中又称为“myWorker”）。
这种机制的另一个优点是它也使您的工作人员池管理变得容易。 您的计算机上可以有 10 个守护进程（myWorker1、myWorker2，...），并让“工作管理器”根据队列长度启动或停止它们。 由于这些命令可以通过 ssh 运行，因此您可以轻松管理多台服务器。
这个解决方案可能听起来很便宜，但是如果您使用编码良好的守护进程和可靠的管理脚本来构建它，我不明白为什么对于任何平均（如“非关键”）项目，它会比大资金解决方案效率低。
I've been helping a friend who's working on a project that involves a Gearman-based queue that will dispatch various asynchronous jobs to various PHP and C daemons on a pool of several servers.
The workers have been designed to behave just like classic unix/linux daemons, thanks to simple shell scripts in /etc/init.d/, and commands like :
invoke-rc.d myWorker start|stop|restart|reloadThis mechanism is simple and efficient. And as it relies on standard linux features, even people with a limited knowledge of your app can launch a daemon or stop one, if they know how it's called system-wise (aka "myWorker" in the above example).
Another advantage of this mechanism is it makes your workers pool management easy as well. You could have 10 daemons on your machine (myWorker1, myWorker2, ...) and have a "worker manager" start or stop them depending on the queue length. And as these commands can be run through ssh, you can easily manage several servers.
This solution may sound cheap, but if you build it with well-coded daemons and reliable management scripts, I don't see why it would be less efficient than big-bucks solutions, for any average (as in "non critical") project.
真正的消息队列中间件（例如 WebSphere MQ 或 MSMQ）提供“触发器”，其中作为 MQM 一部分的服务将在新消息放入队列时启动工作线程。
AFAIK，从野兽的本质来看，没有任何“网络服务”排队系统可以做到这一点。 不过我只仔细看过SQS。 在那里你必须对队列进行轮询，而在亚马逊的情况下，过于急切的轮询将会花费你一些真正的美元。
Real message queuing middleware like WebSphere MQ or MSMQ offer "triggers" where a service that is part of the MQM will start a worker when new messages are placed into a queue.
AFAIK, no "web service" queuing system can do that, by the nature of the beast. However I have only looked hard at SQS. There you have to poll the queue, and in Amazon's case overly eager polling is going to cost you some real $$.
我最近一直在研究这样一个工具。 它还没有完全完成（我认为还需要几天的时间才能实现我称之为 1.0 的东西）并且显然还没有准备好进行生产，但是重要的部分已经编码了。 任何人都可以在此处查看代码：https://gitorious.org/workers_pool。
I've recently been working on such a tool. It's not entirely finished (thought it should take more than a few more days before I hit something I could call 1.0) and clearly not ready for production yet, but the important part are already coded. Anybody can have a look at the code here: https://gitorious.org/workers_pool.
Supervisor 是一个很好的监控工具。 它包括一个 Web UI，您可以在其中监视和管理工作人员。
这是一个工作人员的简单配置文件。
Supervisor is a good monitor tool. It includes a web UI where you can monitor and manage workers.
Here is a simple config file for a worker.