使用.net进行实时数据存储和访问
有谁有接收和更新大量数据、存储数据、排序数据以及快速可视化数据的经验吗？
最好，我正在寻找 .NET 解决方案，但这可能不实用。
现在详细说明...
我每秒将收到大约 1000 次更新，其中一些更新、一些新的数据记录行。但是，它也可能是非常突发驱动的，有时有 5000 个更新和新行。
到一天结束时，我可能会拥有 4 到 5 百万行数据。
我必须存储它们并在 UI 中显示用户更新。用户界面允许用户对数据应用多个过滤器，以仅显示他们想要的内容。我需要更新所有记录并向用户显示这些更新。
我的视觉更新率为 1 fps。
有人对这个问题有任何指导或方向吗？我无法想象我是第一个必须处理这样的事情的人......
不过一开始，我会想到某种内存数据库，但是它是否足够快来查询接近结束时的更新当我获得足够大的数据集的那一天？或者这一切都依赖于智能索引和查询？
提前致谢。
Does anyone have any experience with receiving and updating a large volume of data, storing it, sorting it, and visualizing it very quickly?
Preferably, I'm looking for a .NET solution, but that may not be practical.
Now for the details...
I will receive roughly 1000 updates per second, some updates, some new rows of data records. But, it can also be very burst driven, with sometimes 5000 updates and new rows.
By the end of the day, I could have 4 to 5 million rows of data.
I have to both store them and also show the user updates in the UI. The UI allows the user to apply a number of filters to the data to just show what they want. I need to update all the records plus show the user these updates.
I have an visual update rate of 1 fps.
Anyone have any guidance or direction on this problem? I can't imagine I'm the first one to have to deal with something like this...
At first though, some sort of in memory database I would think, but will it be fast enough for querying for updates near the end of the day once I get a large enough data set? Or is that all dependent on smart indexing and queries?
Thanks in advance.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
这是一个非常有趣且具有挑战性的问题。
我会采用处理器实现排序、过滤、聚合等的管道设计。管道需要一个及时处理的异步（线程安全）输入缓冲区（根据您的 1fps 要求，低于一秒）。如果无法做到这一点，则需要根据问题的性质将数据在磁盘或内存中的某个地方排队。
因此，UI 需要以拉式方式实现，而不是推送式，您只想每秒更新一次。
对于数据存储，您有多种选择。使用数据库并不是一个坏主意，因为无论如何您都需要保留数据（而且我想也是可查询的）。如果您使用 ORM，您可能会发现 NHibernate 与其卓越的二级缓存相结合是一个不错的选择。
许多考虑因素也可能与 Ayende 在设计 NHibernate 实时分析器 NProf 时所做的考虑类似。他在他的博客上撰写了一系列有关它们的帖子。
It's a very interesting and also challenging problem.
I would approach a pipeline design with processors implementing sorting, filtering, aggregation etc. The pipeline needs an async (threadsafe) input buffer that is processed in a timely manner (according to your 1fps req. under a second). If you can't do it, you need to queue the data somewhere, on disk or in memory depending on the nature of your problem.
Consequently, the UI needs to be implemented in a pull style rather than push, you only want to update it every second.
For datastore you have several options. Using a database is not a bad idea, since you need the data persisted (and I guess also queryable) anyway. If you are using an ORM, you may find NHibernate in combination with its superior second level cache a decent choice.
Many of the considerations might also be similar to those Ayende made when designing NHProf, a realtime profiler for NHibernate. He has written a series of posts about them on his blog.
也许 Oracle 是更适合您的 RDBMS 解决方案。您的问题的问题在于，在这个“关键”级别，您需要处理太多的变量和条件。不仅是软件，还有您可以拥有的硬件（它需要花费:)）、连接速度、您预期的普通用户系统设置等等……
祝你好运。
May be Oracle is more appropriate RDBMS solution fo you. The problem with your question is that at this "critical" levels there are too much variables and condition you need to deal with. Not only software, but hardware that you can have (It costs :)), connection speed, your expected common user system setup and more and more and more...
Good Luck.