Disco/MapReduce：使用先前迭代的结果作为新迭代的输入
目前正在 Disco 上实施 PageRank。作为一种迭代算法，一次迭代的结果用作下一次迭代的输入。
我有一个代表所有链接的大文件，每一行代表一个页面，该行中的值代表它链接到的页面。
对于 Disco，我将此文件分成 N 个块，然后运行 MapReduce 一轮。结果，我得到了一组（页面，排名）元组。
我想将此排名提供给下一次迭代。但是，现在我的映射器需要两个输入：图形文件和页面排名。
- 我想“压缩”在一起 图形文件和页面排名， 这样每行代表一个 页面，它的排名，它的链接。
- 由于该图形文件被分成 N 个块，因此我需要将 pagerank 向量分成 N 个 并行块，并压缩区域 将 pagerank 向量添加到图表中 这一切
似乎比必要的更复杂，并且作为一个非常简单的操作（使用典型的映射缩减算法），似乎我错过了关于 Disco 的一些东西，这些东西可以真正简化该方法。
有什么想法吗？
Currently am implementing PageRank on Disco. As an iterative algorithm, the results of one iteration are used as input to the next iteration.
I have a large file which represents all the links, with each row representing a page and the values in the row representing the pages to which it links.
For Disco, I break this file into N chunks, then run MapReduce for one round. As a result, I get a set of (page, rank) tuples.
I'd like to feed this rank to the next iteration. However, now my mapper needs two inputs: the graph file, and the pageranks.
- I would like to "zip" together
the graph file and the page ranks,
such that each line represents a
page, it's rank, and it's out links. - Since this graph file is separated into N chunks, I need to split the pagerank vector into N
parallel chunks, and zip the regions
of the pagerank vectors to the graph
chunks
This all seems more complicated than necessary, and as a pretty straightforward operation (with the quintessential mapreduce algorithm), it seems I'm missing something about Disco that could really simplify the approach.
Any thoughts?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
看起来您需要在第一次传递中使用 init_map ，然后在后续的每次迭代中使用 iter_map 。
请参阅：http://discoproject.org/doc/faq.html#id7
可以您输出包含外链的 python 对象，而不仅仅是 (page,rank) 元组？
另一种选择是在某处（dict、memcache、kyotocabinet 等）按页面键入外链接，然后从映射函数中查找它们。如果您使用 Disco 链接事物，我认为您不会想在工作流程中将事物压缩在一起。
Looks like you'll want to use an init_map for the first pass and then a iter_map for each subsequent iteration.
See: http://discoproject.org/doc/faq.html#id7
Can you output python object that include the outlinks, instead of just the (page,rank) tuples?
Another option would be to have the outlinks keyed by page somewhere (dict, memcache, kyotocabinet, etc...) and look them up from the mapping function. If you're chaining things with Disco, I don't think you'll want to zip things together in the middle of the workflow.