实现活动源、消息队列、RDBMS 或 NoSQL DB 的最佳(可扩展、快速、可靠)方法是什么?
我需要为类似于许多流行社交网络平台的系统构建一个活动源(流?更准确地说是“生命流”。)。我最初的尝试是使用 RDBMS,但由于需要大量的 JOIN,很快就放弃了这个想法。在寻找其他可能的(并且更适合的)方法时,我偶然发现了以下帖子:
根据使用消息队列的建议,我花了一些时间研究 RabbitMQ 及其 PubSubHubbub 协议。我假设了以下方法:
1)每个用户都有一个“主题”
2)其他用户订阅该主题
3)当用户执行某些操作时,会发布一条消息,然后将其与相关(参考文献已解析)、格式化(人类友好的语言、链接等)和聚合(X、Y 和 Z 对帖子 P 进行了评论) PHP 脚本。
但是,我仍然必须仔细检查每条消息并处理它(除非我的方法完全错误)。那么,将所有内容存储在 RDBMS 中和使用消息队列(除了 PubSubHubbub 协议的实现)之间有什么区别?
有没有更有效的方法来构建这样的系统? (如有,请注明)
欢迎评论/建议/批评。 :)
先感谢您!
PS:有一篇有趣的文章介绍了 FriendFeed 如何实现它( http://bret .appspot.com/entry/how-friendfeed-uses-mysql)。然而,我觉得“黑客”将 MySQL 推出了它的舒适领域(这只是关系数据,没有关系数据使用 RDBMS 有何意义?)
PPS:我看到的另一个使用消息队列的问题(也许,由于我是这项技术的新手),一旦消息被“消费者”获取,它就会从队列中删除,但是,我希望它持续任意时间。
I need to build an activity feed (stream? A "lifestream" to be more accurate.) for a system similar (same) in resemblance to many popular social networking platforms. My initial attempt was to use an RDBMS but quickly dropped the idea due to the vast amounts of JOINs needed. Scavenging for other possible (and better-suited) approaches, I stumbled upon the following post:
How do social networking websites compute friend updates?
Taking the advise to make use of a message queue, I have spent some time studying RabbitMQ and its PubSubHubbub protocol. And I postulated the following approach:
1) Each user has a "topic"
2) Other users subscribe to the topic
3) When the user performs some action, a message is published which is then related (References resolved), formatted (Human-friendly language, links, etc.) and aggregated (X, Y and Z have commented on post P) with a PHP-script.
However, I would still have to go through each message and process it (unless my approach is completely wrong). So, what would the difference be between storing everything in a RDBMS and using a message queue (other than the implementation of the PubSubHubbub protocol)?
Are there more efficient ways to build such a system? (If so, please specify)
Comments / Suggestions / Criticisms are welcome. :)
Thank you in advance!
P.S.: There is an interesting article on how FriendFeed implements it ( http://bret.appspot.com/entry/how-friendfeed-uses-mysql ). However, I feel the "hackery" pushes MySQL out of it's comfortable domain (which is simply Relational Data and what would be the point of using an RDBMS without relational data?)
P.P.S.: Another issue using a message queue that I see (perhaps, due to me being new to this technology) is that once the message is fetched by the "Consumer", it is removed from the queue, however, I want it to persist for an arbitrary amount of time.
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(1)
我想给你一些提示:
不要使用 RDBMS,而是使用内存(FAST)数据库,例如 雷迪斯。希望您同意我在 redis 基准 中的观点,redis 非常快。作为另一个旁注,我想指出安装 redis 是小菜一碟:)。
make
一个 redis-client,它使用 C,所以速度也会非常快。
- 如果我理解正确的话,你认为 pubsubhubbub 与消息队列相同,但它们不是:
与消息队列对比:
您可能认为它们是相同的(它们有一些相似之处),但它们并不相同。对于我的消息队列,我会使用redis(redis非常强大,因为它也有一个基本的消息队列:))。您可以使用 rpush 将消息(工作单元)放入队列中。
接收来自队列的消息
然后,您可以使用 brpop(阻止 pop:))从工作进程 将从 cli 启动以保留在内存中,因此不会有一次又一次将 PHP 加载到内存中的开销。
我希望这对您来说是有希望的,如果您有任何问题,我非常愿意回答;)
Some tips I would like to give you:
Don't use a RDBMS, but an in-memory(FAST) database like for example redis. As hopefully you agree with me from the redis benchmarks, redis is pretty fast. As another sidenote I would like to point out installing redis is child's play :).
make
There is a redis-client for PHP which uses C so that is also going to be very fast.
- If I understand you correctly you think that pubsubhubbub is the same as a message queue but they aren't:
Versus message queue:
You might think they are the same(they have some similarities), but they aren't the same. For my message queue I would redis(redis is very powerfull because it also has a basic message queue :)). You could put message(unit of work) onto a queue using rpush.
Then from your worker processes you could receive messages from the queue using brpop(blocking pop :))
The workers process spawn are going to be started from the cli to stay in memory so aren't going to have overhead loading PHP in memory again and again.
I hope this is hopefully for you and if you might have any question I am very willing to answer them ;)