具有内存压缩的分配器
我想知道，是否有关于内存压缩和内存分配器组合的项目/至少有一些研究（当然，以牺牲一些速度为代价）？
例如，想象一下这样的场景：我们有一棵巨大的树必须处理。这棵树不适合记忆。通过压缩分配器，我们几乎可以适应任何树。
当然，可以使用迭代方法而无需立即构建树，但我的问题纯粹是理论上的（对于今天而言）。
取消引用可能需要特殊的宏/模板，以便分配器能够解压选定的区域。但是，如果一个区域引用另一区域等怎么办？一定有一些非常复杂的算法，可能只能在托管语言中解决（但 Boehm 能够为 C++ 制作 GC！）
或者它可能太复杂/太慢（甚至与节省的内存相比）根本不值得？ 虚拟内存和交换可能非常慢，尤其是在垃圾收集环境中。最近，一个 1 GB 的应用程序使整个操作系统无响应...因此内核级机制不一定有效。
你可以把它想象成（我仍然试图向你保证这不是一个非常愚蠢的想法）作为快速用户模式互斥体与慢速本机互斥体、快速用户模式绿色线程的对立（如在 Erlang 中，在一台计算机上有多达 2000 万个并发进程）机）与较慢的本机线程等。
I am wondering, are there projects/at least some research on combination of in-memory compression and memory allocators (at the expense of some speed, of course)?
For example, imagine the scenario: we have a huge tree we must process. This tree doesn't fit into memory. With compressing allocator, we're able to fit pretty any tree.
Of course, one can use iterational approach without constructing the tree at once, but my question is purely theoretical (for today).
Probably dereferencing would need special macros/templates so that the allocator would be able to unpack the selected regions. But what if one region references another region, etc.? There must be some very complicated algorithm, probably resolved only in managed languages (but Boehm was able to make a GC for C++!)
Or maybe it's so complicated/slow (even compared to saved memory) that isn't worth at all?
Virtual memory & swapping may be very slow, especially in garbage-collected environments. Recently a 1 gb app was making the whole OS unresponsive... So kernel-level mechanisms aren't necessarily efficient.
You can think of it (i'm still trying to assure you that's not a very stupid idea) as of opposition fast usermode futexes vs. slow native mutexes, fast usermode green threads (as in Erlang with up to 20 million simultanous processes on a machine) vs. slower native threads etc.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
请参阅虚拟内存系统中的压缩缓存案例一些有关压缩虚拟内存的相关想法。这个想法有一个 Linux 实现 (compcache)。
我认为在分配器中进行压缩甚至没有意义。分配器的工作是返回我要求它分配的字节数（并避免堆碎片和所有其他爵士乐）。它不知道我将把什么数据放入该内存中。如果内存确实紧张，那么应用程序应该执行内存压缩。
内存压缩的一个很好的例子可以通过 Redis 使用记录的 Redis 的巧妙技巧来看到/redis.io/topics/memory-optimization" rel="nofollow">此处。
See The Case for Compressed Caching in Virtual Memory Systems for some related ideas that talk about compressing virtual memory. There's a Linux implementation of this idea (compcache).
I'm not it even makes sense to do compression in an allocator. An allocators jobs is to give back how many bytes I ask it to allocate (and avoid heap fragmentation and all that other jazz). It has no knowledge of what data I'm going to put in that memory. If memory is really tight then in-memory compression should be performed by the application.
A good example of in-memory compression can be seen by the clever tricks Redis uses documented here.