基于队列文件的持久性 - WebSphere MQ 与 SQLite 队列 (WAL)
我使用 Websphere MQ，它默认带有“基于文件的队列”。因此，每个队列通常在磁盘上有一个物理文件。
因此，可以说，对于这个 Websphere MQ 队列，我们有多个读取器和写入器......我假设读取和写入之间会有一些“毫秒锁定”开销，但我保证它仍然比使用 DB2 等完整数据库更快现在
我已经使用 SQLite 实现了一个队列，并且它在 WAL 模式下在生产中运行良好。
我的问题是，您真的认为 WebSphere MQ 等产品使用的基于文件的队列或基于 SQLite 的队列之间的速度存在巨大差异吗？毕竟两者都是基于文件的，而且 SQLIte 是纯 ANSI C 的这一事实在速度方面表现良好。
SQLite 确实提供了一些额外的功能，例如“更新挂钩”等......以及许多其他功能。
我想我想知道的是，如果您要在 C 中实现高速持久队列，您会采用与 WebSphere MQ 类似的方式来实现，并将消息放在单个文件中的不同偏移量等中，还是会使用 SQLite在 WAL 模式下？
感谢您的帮助;-)
I use Websphere MQ and it comes by default with a "file based queue". So each queue typically has ONE physical file on disk.
So lets say for this Websphere MQ queue we have multiple readers and writers....I assume there will be some "millisecond locking" overhead between reads and writes, yet I guarantee it will still be quicker than using a full database like DB2 etc.
Now I have implemented a queue using SQLite and it works in Production happily in WAL mode.
My question is do you really think there is a huge difference in SPEED between the file based queue used by products like WebSphere MQ or the SQLite based queue? Both are file based after all and the fact that SQLIte is pure ANSI C does it well for speed.
SQLIte does offer some extra functionality like "update hooks" etc....and numerous others.
I guess what I would like to know is if you were going to implement a high speed persistent queue in C would you do it in a similar way to WebSphere MQ and have messages in different offsets etc in a single file, or would you use SQLIte in WAL mode?
Thanks for the help ;-)
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
您对示例施加的限制越多，您就越有可能使数字看起来像您想要的那样。一个应用程序的单个队列？如果这是您唯一的要求，那么您有很多选择。
但看一下 WAL 模式的一些限制。检查点等待读者完成。因此，您拥有的读者越多，检查点就越困难，并且检查点的破坏性也越大。如果 WAL 文件变大，则读取性能会下降，因此您无法在繁忙的系统上执行惰性检查点并保持性能。
所以问题“如果您要在 C 中实现高速持久队列，您会采用与 Websphere MQ 类似的方式来实现，并在单个文件中使用不同偏移量等的消息，还是会在 WAL 中使用 SQLite模式？”没有涵盖所有要求或注意事项。随着并发用户数量的增加，进程架构开始掩盖您所询问的存储方法的差异。 WMQ 可以处理数千个并发读取器和写入器，日志文件非常大，性能曲线相当平坦，而 WAL 文档指出性能与 WAL 文件的大小成正比，即随着 WAL 变大，性能曲线呈下降趋势。
如果我要用C实现高速队列？如果你的意思是我要选择哪种现有产品，我会选择经过二十年调整和优化、每年投入数百万美元研发、并且专门专注于排队的产品。如果“实现”是指如果我从头开始编写一个新的排队引擎，我会怎么做，那么我会受到长期专注于排队的产品的严重影响，并尝试重用他们的解决方案到不平凡的问题。数据库和队列引擎并不是偶然形成各自的存储架构的。一种针对队列进行优化，另一种针对数据库语义进行优化，如果您将范围扩大到包括 WMQ 和 SQLite 之外的数据库和队列引擎，那么这在所有类别中通常都是正确的。
全面披露：WMQ 存在二十年来的大部分时间我都在使用它，最近加入了 IBM 的产品管理团队。我可能有点偏见，但我试图专注于这里的技术，而不是下意识的“我的产品比你的产品更好”的反应。赞成票表示同意，反对票表示不同意。如果得到太多反对票，我将撤回答案。
The more constraints you place on the example, the more you would probably be able to make the numbers look like whatever you wanted. A single queue with one app? If that's your only requirement then you have lots of choices.
But take a look at some of the restrictions on WAL mode. Checkpoints wait for readers to complete. Therefore more readers you have the harder it is to checkpoint and the more disruptive checkpointing becomes. If the WAL file gets large then read performance degrades so you cannot do lazy checkpointing on a busy system and remain performant.
So the question "if you were going to implement a high speed perstent queue in C would you do it in a similar way to Websphere MQ and have messages in different offsets etc in a single file, or would you use SQLIte in WAL mode?" doesn't capture all the requirements or considerations. As the number of concurrent users scales, the process architecture begins to overshadow the differences in storage approaches you are asking about. WMQ can handle thousands of concurrent readers and writers, with very large log files with fairly flat performance curves whereas the WAL docs state that performance is proportional to the size of the WAL file, i.e. performance curves trend down as the WAL gets large.
If I were going to implement a high speed queue in C? If you mean which existing product do I pick, I'd go for the one that has had twenty years of tuning and optimization with millions of dollars in R&D invested each of those years and which focuses exclusively on queuing. If by "implement" you mean how would I do it if I were to write a new queuing engine from scratch, then I'd be heavily influenced by the products that have focused on queuing for a really long time and try to reuse their solutions to the non-trivial problems. Databases and queuing engines didn't arrive at their respective storage architectures by accident. One is optimized for queues and one is optimized for database semantics and if you expand your scope to include DBs and queuing engines other than WMQ and SQLite this holds generally true across the categories.
Full disclosure: I've been working with WMQ for most of the twenty years it's been around and recently joined its product management team at IBM. I might be a bit biased but I've tried to focus on the technology here and not a knee-jerk "my product is better than your product" reaction. Feel free to signify agreement with up-votes, disagreement with down-votes. I'll withdraw the answer if it gets too many down-votes.
对于一个消息生成器，具有默认日志记录参数（双重或三重循环日志记录）的 WMQ 7 将使用 mqjms api 在我的笔记本电脑 7200 高清驱动器上每秒存储大约 300-400 条持久消息，同时使用人们实际期望的尽可能快的 jms 代码（一个会话，一个消息生产者，在通道上缓冲）和小消息大小（有效负载大小<1k的字节消息）。在这种情况下，HDD 是瓶颈。
通过在 QM 端使用较少的日志记录，您将获得更快的速度。这个比例相当线性。使用非持久消息也有帮助。
其他常见问题是网络层。消息传递通常是发送大小不超过几 kb 的小消息。使用来自一个用户的通用网络代码（打开连接/打开会话/打开生产者/以及返回），您将在 HDD 启动之前遇到 TCP 限制。为了避免此问题，WMQ 允许在通道上进行消息批处理。
我怀疑使用默认日志记录在 WMQ 中存储持久消息比在 DB2 中插入 blob 更快。底层机制基本相同（事务日志、rec/ack...）。
我从未尝试将它与 SQLLite 进行比较。
WMQ 并不是比子弹消息传送解决方案更快。它也不是最容易管理的解决方案。因此，它有光明的一面，在我看来，它是市面上更好的产品之一。
With one message producer, WMQ 7 with default logging parameters (double or triple circular logging) will store about 300-400 persistent message per second on my laptop 7200 hd drive using mqjms api while using as fast jms code as one can realistically expect (one session, one message producer, buffering on channels) and small message size (bytes message with payload size < 1k). In this scenario HDD is bottleneck.
You will get get faster speed by using lesser loging on QM side. This scales pretty linearly. Using unpersistent messages also helps.
Other common problem is network layer. Messaging is usually about sending small messages no more than few kb in size. With common network code (open connection/open session/open producer/and back) from one user you will hit TCP limitations before HDD kicks in. To avoid this problems WMQ allows message batching on channels.
I doubt that storing persistent messages in WMQ with default loging is faster than inserting blob in a DB2. Underlying mechanism is mostly the same (transactional logs, rec/ack, ...).
I never tried to compare it to SQLLite.
WMQ is not faster-than-bullet messaging solution. It is also not the easiest solution to manage. Thus said, it has bright sides and by my opinion it is one of better products out there.