在 Windows 中使用修改后的 FIFO 语义构建工作项处理系统
我正在构建一个系统，该系统会生成排队等待后端处理的“工作项”。 我最近完成了一个具有相同要求的系统，并提出了一个我认为不是最佳的架构，并希望为这个新系统提供一些建议。
工作项集中排队，并且需要以基本上先进先出的顺序进行处理。 如果这是唯一的要求，那么我可能会喜欢 MSMQ 或 SQL Server Service Broker 解决方案。 然而，实际上，我需要按照修改后的 FIFO 顺序选择工作项。 一个工作项具有多个属性，并且需要按 FIFO 顺序分配这些属性，其中存在某些属性值组合。
作为示例，工作项可以具有以下属性：办公室、优先级、组号和序列号（组内）。 当多个项目针对同一组号排队时，保证它们按序列号顺序排队，并且具有相同的优先级。
有几个后端进程（当前作为 Windows 服务实现）在给定服务的某些配置参数的情况下，按照修改后的 FIFO 顺序拉动工作时间。 运行华盛顿特区的服务配置为仅处理华盛顿的工作项目，而纽约的服务可能配置为处理纽约和华盛顿的项目（主要是为了提高总体吞吐量）。 除了这种类型的选择性之外，还应首先处理优先级较高的项目，并且必须按序列号顺序处理包含相同“组编号”的项目。 因此，如果 NY 服务正在处理具有序列 1 的组 100 中的 DC 项目，我不希望 DC 服务拉取组 100 序列 2 中的 DC 项目，因为序列 1 尚未完成。 其他组中的项目应仍符合处理条件。
在上一个系统中，我用 SQL 表实现了队列。 我创建了存储过程来提交项目，更重要的是，将项目“分配”给负责处理它们的 Windows 服务。 赋值存储过程包含上面描述的选择逻辑I。 每个Windows 服务都会调用分配存储过程，向其传递该服务实例特有的参数（例如合格的办公室）。 此分配存储过程将工作项标记为已分配（正在进行中），并且当工作完成时，将调用最终存储过程以从“队列”（表）中删除该项目。
这个解决方案确实有一些优点，因为我可以通过简单的 SQL select 语句快速检查这些“队列”的状态。 我还能够轻松地操作队列（例如，我可以使用简单的 SQL 更新语句来提高优先级）。 然而，不利的一面是，我偶尔必须处理这些队列表上的死锁，并承担编写这些存储过程的负担（一段时间后就会变得乏味）。
不知何故，我认为 MSMQ（带或不带 WCS）或 Service Broker 应该能够提供更优雅的解决方案。 滚动我自己的排队/工作项处理系统感觉不对。 但据我所知，这些技术无法提供我在分配过程中所需的灵活性。 我希望我错了。 任何的建议都受欢迎。
I’m building a system that generates “work items” that are queued up for back-end processing. I recently completed a system that had the same requirements and came up with an architecture that I don’t feel is optimal and was hoping for some advice for this new system.
Work items are queued up centrally and need to be processed in an essentially FIFO order. If this were the only requirement, then I would probably favor an MSMQ or SQL Server Service Broker solution. However, in reality, I need to select work items in a modified FIFO order. A work item has several attributes, and they need to be assigned in FIFO order where certain combinations of attribute values exist.
As an example, a work item may have the following attributes: Office, Priority, Group Number and Sequence Number (within group). When multiple items are queued for the same Group Number, they are guaranteed to be queued in Sequence Number order and will have the same priority.
There are several back-end processes (currently implemented as Windows Services) that pull work times in modified FIFO order given certain configuration parameters for the given service. The service running Washington, DC is configured to process only work items for DC, while the service in NY may be configured to process both NY and DC items (mainly to increase overall throughput). In addition to this type of selectivity, higher priority items should be processed first, and items that contain the same “Group Number” must be processed in Sequence Number order. So if the NY service is working on a DC item in group 100 with sequence 1, I don’t want the DC service to pull off DC item in group 100 sequence 2 because sequence 1 is not yet complete. Items in other groups should remain eligible for processing.
In the last system, I implemented the queues with SQL tables. I created stored procedures to submit items and, more importantly, to “assign” items to the Windows Services that were responsible for processing them. The assignment stored procedures contain the selection logic I described above. Each Windows Service would call the assignment stored procedure, passing it the parameters that were unique to that instance of the service (e.g. the eligible offices). This assignment stored procedure stamps the work item as assigned (in process) and when the work is complete, a final stored procedure is called to remove the item from the “queue” (table).
This solution does have some advantages in that I can quickly examine the state of these “queues” by a simple SQL select statement. I’m also able to manipulate the queues easily (e.g. I can bump priorities with a simple SQL update statement). However, on the downside, I occasionally have to deal with deadlocks on these queue tables and have the burden of writing these stored procedures (which gets tedious after a while).
Somehow I think that either MSMQ (with or without WCS) or Service Broker should be able to provide a more elegant solution. Rolling my own queuing/work-item-processing system just feels wrong. But as far as I know, these technologies don’t offer the flexibility that I need in the assignment process. I am hoping that I am wrong. Any advice would be welcome.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
在我看来，你的原子工作单元的概念是一个组。 因此，我建议您只对标识组 Id 的消息进行排队，然后您的工作人员将必须转到将组 Id 映射到 1 个或多个工作项的表。
您可以通过使用多个队列来处理其他问题 - NY-High、NY-Low、DC-High、DC-Low 等。
不过，老实说，我认为您最好解决您的死锁问题当前的架构。 您应该使用更新锁定和读取过去提示从队列表中读取 TOP 1 消息，并按您的优先级逻辑和您想要的任何过滤条件（办公室/位置）排序。 然后，您处理您的 1 条消息，更改其状态或将其移动到另一个表。 您应该能够并行调用该存储过程，而不会出现死锁问题。
It seems to me that your concept of an atomic unit of work is a Group. So I would suggest that you only queue up a message that identified a Group Id, and then your worker will have to go to a table that maps Group Id to 1 or more Work Items.
You can handle your other problems by using more than one queue - NY-High, NY-Low, DC-High, DC-Low, etc.
In all honesty, though, I think you are better served to fix your deadlock issues in your current architecture. You should be reading the TOP 1 message from your queue table with Update Lock and Read Past hints, ordered by your priority logic and whatever filter criteria you want (Office/Location). Then you process your 1 message, change it's status or move it to another table. You should be able to call that stored procedure in parallel without a deadlock issue.
队列适用于 FIFO 顺序，而不是随机访问顺序。 即使您说需要 FIFO 顺序，您也需要相对于一组随机变量的 FIFO 顺序，这本质上是随机顺序。 如果要使用队列，则需要能够在消息进入队列之前而不是进入之后确定顺序。
Queues are for FIFO order, not random access order. Even though you are saying that you want FIFO order, you want FIFO order with respect to a random set of variables, which is essentially random order. If you want to use queues, you need to be able to determine order before the message goes in the queue, not after it goes in.