The master would write the jobs into a space (essentially an object repository). The consuming clients are always looking for jobs and as jobs become available, they will pull from the space, process, and write back the results to that space, or another (all under a transaction). You would tag jobs as belonging to a particular run in order to group results.
The advantage of this is that this scales very easily (simply by adding more consumers). The consumers would have to determine when they can work, and simply need to be configured with info about the space (how to find it). The producer is decoupled completely from the set of consumers.
Because work is processed under a transaction, if a consumer fails to complete, the work returns to the space and is available for processing by another consumer.
You can handle intermediate results easily. The producer takes results from the space and can derive intermediates as results become available. You can cancel jobs easily. Simply remove them from the space.
You can add more producers very easily. They simply write to the same space, and if the jobs are tagged appropriately, results are tied to the producer unambiguously.
I'm not sure what frameworks are available for .Net, unfortunately (I'm from the Java world and would use Javaspaces - these use dynamic discovery, and next to no configuration is needed). But worth some Googling. Perhaps (if this is powerful enough), you can write the C# producer/consumers to interface to a Javaspace infrastructure.
发布评论
评论（1）
我将研究为此使用基于空间的架构的可能性。
主节点会将作业写入一个空间（本质上是一个对象存储库）。 消费客户端总是在寻找工作，当工作可用时，他们将从空间中提取、处理并将结果写回到该空间或另一个空间（全部在事务下）。 您可以将作业标记为属于特定运行，以便对结果进行分组。
这样做的优点是很容易扩展（只需添加更多消费者）。 消费者必须确定他们何时可以工作，并且只需要配置有关空间的信息（如何找到它）。 生产者与消费者完全解耦。
由于工作是在事务下处理的，因此如果使用者未能完成，工作将返回到空间并可供另一个使用者处理。
您可以轻松处理中间结果。 生产者从空间中获取结果，并在结果可用时导出中间体。 您可以轻松取消工作。 只需将它们从空间中移除即可。
您可以非常轻松地添加更多生产者。 他们只是写入相同的空间，如果作业被适当标记，结果就会明确地与生产者联系在一起。
不幸的是，我不确定 .Net 有哪些框架可用（我来自 Java 世界，会使用 Javaspaces - 这些使用动态发现，并且几乎不需要配置）。 但值得谷歌搜索一下。 也许（如果这足够强大的话），您可以编写 C# 生产者/消费者来连接到 Javaspace 基础设施。
I would investigate the possibility of using a space-based architecture for this.
The master would write the jobs into a space (essentially an object repository). The consuming clients are always looking for jobs and as jobs become available, they will pull from the space, process, and write back the results to that space, or another (all under a transaction). You would tag jobs as belonging to a particular run in order to group results.
The advantage of this is that this scales very easily (simply by adding more consumers). The consumers would have to determine when they can work, and simply need to be configured with info about the space (how to find it). The producer is decoupled completely from the set of consumers.
Because work is processed under a transaction, if a consumer fails to complete, the work returns to the space and is available for processing by another consumer.
You can handle intermediate results easily. The producer takes results from the space and can derive intermediates as results become available. You can cancel jobs easily. Simply remove them from the space.
You can add more producers very easily. They simply write to the same space, and if the jobs are tagged appropriately, results are tied to the producer unambiguously.
I'm not sure what frameworks are available for .Net, unfortunately (I'm from the Java world and would use Javaspaces - these use dynamic discovery, and next to no configuration is needed). But worth some Googling. Perhaps (if this is powerful enough), you can write the C# producer/consumers to interface to a Javaspace infrastructure.