我可以使用什么技术来存储数据
我有一个应用程序，每 0.1 秒存储大约 1kbyte 的数据。即 36MByte/小时，或大约 600MByte/天。
数据具有高度可压缩性，因此其压缩比应在 10:1 到 100:1 之间。所有数据均通过时间戳引用。
我的问题是：我可以使用什么技术来存储这些数据？
约束：
- 将数据插入数据库的时间不能随着数据库大小的变大而增加。这个限制排除了 Microsoft SQL Server（我们尝试过，5 天后它就停止了，因为每个“插入”都需要一分钟）。
- 我们可以有效地每天暂停数据记录 4 小时，这将给我们时间进行压缩等。
- 我们希望与 LINQ for .NET 兼容，这意味着我们可能需要一个带有 LINQ 适配器（a MySQL 风格的界面就可以了）。
I have an application where I am storing about 1kbyte of data every 0.1 second. That's 36MByte/hour, or roughly 600MByte/day.
The data is highly compressible, so it should compress between 10:1 and 100:1. All the data is referenced by timestamp.
My question is this: what technique can I use to store this data?
Constraints:
- The time to insert the data into the database cannot increase as the database size gets larger. This constraint rules out Microsoft SQL Server (we tried it, and after 5 days it ground to a halt as each "insert" was taking a minute).
- We can effectively pause the data recording for 4 hours per day, which would give us time to do compression, etc.
- We would like to compatible with LINQ for .NET, which means that we would probably need a database that a LINQ adapter (a MySQL style interface would be ok).
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（3）
一种方法是将传入的数据简单地附加到磁盘上的文件中。一天后，切换到一个新文件，然后生成一个进程来压缩和存储前一天的文件。
您似乎假设您需要将数据存储在数据库中，但没有说明原因。你？
One approach is to simply append your incoming data to a file on disk. After a day, switch to a new file, and then spawn a process to compress and store the previous day's file.
You seem to assume that you need to store your data in a database, without stating a reason why. Do you?
如果您无法让 SQL Server 及时处理这么小的负载，那么我想知道如果您不探索如何插入数据，任何 RDBMS 是否对您有效。
您是否只是对没有其他索引/函数/进程阻塞/读取的单个表（带有主键）进行非常简单的插入？或者这个过程实际上比您正在谈论的这个简单/小插入更复杂一些？
如果您执意要使用 Linq，您是否会分析您的 linq 语句以确保您不会告诉 ORM 做一些愚蠢的事情？
If you can't get SQL Server to handle that small of a load in a timely fashion, then I wonder if any RDBMS will be effective for you if you don't explore how you are inserting the data.
Are you doing just a very simple insert into a single table (with a primary key) that has no other indexes/functions/process blocking/reads ? Or is this process actually a little more involved than this simple/small insert you are talking about?
If you are dead set on using Linq, are you profiling your linq statements to make sure you are not telling the ORM to do something stupid?
也许您可以将所有内容存储到二进制文件中，并将元数据存储到数据库中。
Perhaps you can store all the contents onto a binary file and the meta data to the DB.