python c 扩展/opencl 守护进程
我正在研究内存数据库的概念证明。我确实了解像 redis 这样的东西，并且喜欢像相交集这样的东西，但这还不够。我想了解如何实现类似 parstream 数据库或 Drawingtoscale 解决方案的服务。 因此我喜欢有一个守护进程来将东西保存在内存中。 我用pyro做了一个原型，它做了它应该做的事情：生成一个随机位图索引并接受位图掩码作为来自pyro客户端的输入并返回与位图掩码匹配的所有主键。 但是迭代索引在某些程度上太慢了，这一点很明显，因为它是 python 的。我想使用 C 扩展来执行索引扫描。因此，位图索引应保存在 c 扩展内的内存中。 下一步是在 GPU 内存中保留相同的数据，并在我的 GPU 内进行迭代。
没有编写 C 扩展和 pyOpenCL 的经验，我的问题是：我可以将数据保存在我的 C 扩展中并使用 python （pyro daeomon 甚至 Flask/tornado ）来接收和返回查询，而不总是复制整个位图索引吗？
问候 基督教
I am working on a proof of concept for an in-memory database. I do know about stuff like redis and love things like intersecting sets but that is not enough. I want to find out how to implement a service which is like the parstream database or the drawntoscale solutions.
And therefore I like to have a daemon which keeps stuff in the memory.
I did a prototype with pyro which does what it should: generate a random bitmap index and accept an bitmap mask as input from a pyro client and return all primary keys which match the bitmap mask.
But iterating over the index is too slow by some magnitudes which is obvious since it is python. I would like to use an C extension which does the index scan. Therefore the bitmap index shoud be kept in memory inside the c extension.
The very next step would be to keep the same data in the gpu ram iterate over inside my gpu.
Having no experience writing c extensions and pyOpenCL my question is: can I keep data inside my C Extension and use python ( pyro daeomon or maybe even flask/tornado ) just for receiving and returning the queries without copying the whole bitmap index all the time?
Regards
Christian
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
是的，C 扩展可以分配内部结构，但请务必在需要时正确销毁数据。
您还可以将代码编写为 C/C++ DLL，并使用 ctypes python 扩展直接访问 C 函数并向它们传递 C 变量（int、char * 等）。
Yes, C extensions can have internal structures allocated, but be sure to destroy your data properly when needed.
You can also write your code as a C/C++ DLL and use the ctypes python extension to access C functions directly and pass them C variables (ints, char *, etc.).