逆索引二进制格式
我想弄清楚什么样的二进制文件可以支持我对反向索引的需求。假设我有一个可以用唯一 ID 识别的文档，每个文档可以有 0-65535 范围内的 360 个固定值。像这样的东西：
Document0: [1, 10, 123, ...] // 360 个值
Document1: [1, 10, 345, ...] // 360 个值
现在，反向索引很容易 - 我可以为每个值创建包含文档的可能值列表，并且可以快速执行查询，例如：
1: [Document0, Document1]
10: [Document0, Document1]
123: [Document0]
345: [Document1]
但我想存储大量文档在某种文件（二进制）中，并且能够快速查询，而且还可以添加新文档而无需重新创建整个结构。
现在我正在努力如何组织该文件。如果我想快速访问，我需要固定长度的文档数组来进行文件查找和读取。但固定大小意味着我将有很多空白空间用于文档列表。我唯一的想法是拥有某种存储桶系统，每个值都可以属于特定大小的存储桶，例如，有大小为 1, 2, 4, 8, 16, 32, ...（或类似的东西）的存储桶我需要某种标头来指出存储桶的起始位置和存储桶的大小。这个想法将优化商店规模，但我再次遇到添加新文档的问题。
知道如何组织我的“反向索引”文件吗？
最好的。
i'm trying to figure out what kind of binary file can support my needs for inverse index. Let say that i have document that i can identify with unique ID and each document can have 360 fixed values in range of 0-65535. Something like this:
Document0: [1, 10, 123, ...] // 360 values
Document1: [1, 10, 345, ...] // 360 values
Now, inverse index is easy - i can create for each possible value list of documents that contains, and query can be executed fast, e.g.:
1: [Document0, Document1]
10: [Document0, Document1]
123: [Document0]
345: [Document1]
But i wanna to store large number of documents in some kind of file (binary) and to have ability to query fast but also to add new documents without recreating whole structure.
Now i'm struggling how to organize that file. If I wanna fast access i need fixed length document arrays to do file seek and than read. But fixed size means that i will have a lot of empty spaces for document list. My only idea is to have some kind of bucketing system and each value can belong to bucket of specific size, e.g. there are buckets with size 1, 2, 4, 8, 16, 32, ... (or something like that) and i need some kind of header which will point me where bucket starts and size of bucket. This idea will optimize store size, but again i'm having problem with addition of new documents.
Any idea how to organize my 'inverse index' file?
Best.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
我会选择 65536 个文件，每个文件都有文档 ID。如果您想对文件系统温和一些，请将其分为 256 个目录，每个目录有 256 个文件。
I would go for 65536 files each having ID's of the documents. If you want to go gentle on the filesystem, divide that into 256 directories having 256 files each.
听起来很好。我的读取速度非常快，另一方面写入速度较慢 - 我需要确保每个文件中都有唯一的文档（现在我有一个简单的模型来在内存中存储恒定数量的文件，并将它们转储到当达到某个阈值时磁盘）。感谢您的回复。
That sounds good. I'm doing reads very fast, writes on other hand are slower - i need to make sure that each file has unique document in it (for now I'm having simple model to store constant number of files in memory, and dump them on disk when some threshold is reached). Thanks for response.