通过集合/列表/其他容器使用 AWE 内存进行数据存储
有没有人对自定义数据(delphi 集合、二叉树、DIContainer 等)的存储和处理有任何建议(产品、工具集、方法或其他),这些数据不将自身限制在标准 win32 内存地址空间?极端地说,有没有现成的东西可以相当于保存 10GB TList,从而突破 /3GB 开关障碍和 4GB“windows on windows”限制?
理想情况下,我们需要的是对 Delphi 应用程序程序员非常透明的东西,但允许非常快速地访问其结构中保存的数据,最好是通过键查找。相当于一个delphi集合容器就可以了,但是它的内存使用需要通过AWE。它还需要负责将其使用的物理空间映射和取消映射到使用它的 win32 进程中,即这将是透明位...
将数据移入数据库并不是答案 - 信息需要保留常驻内存以实现快速访问。我们尝试过的内存数据库/表没有使用 AWE,而且访问速度也很慢。我们当前的 Delphi 数据结构很好,但超出了 win32 地址空间的限制。
Does anyone have any suggestions (product, toolsets, methods or other) for the storage and processing of custom data (delphi collections, binary trees, DIContainers etc) that DOES NOT restrict itself to a standard win32 memory address space? To put that in the extreme, is there anything off the shelf that can do the equivalent of holding a 10GB TList, thereby blowing the /3GB switch barrier and the 4GB 'windows on windows' limit?
What we ideally need is something that is pretty transparent to the Delphi application programmer, but allows very fast access to the data held in its structures, preferably via key lookup. The equivalent of a delphi colletion container would be fine, but its memory usage needs to be via AWE. It would also need to take care of mapping and unmapping the physical space it uses into the win32 process making use of it i.e. that would be the transaprent bit...
Moving the data into a database is not the answer - the information needs to remain memory resident for very fast access. The in-memory databases/tables that we've tried do not make use of AWE and also are slow at accessing. Our current Delphi data structures are fine, but straining the limits of win32 address space.
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
![扫码二维码加入Web技术交流群](/public/img/jiaqun_03.jpg)
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(7)
我将成为一个十足的傻瓜,并告诉你我在工作中做了一些比你所描述的更先进的东西。所以我担心这都是闭源的。从来没有在任何地方见过这样的事情。我们将 VM、AWE、MMF 和(很快)32 64 位 IPC 组合成一台大型平均数据处理机器,寻址高达 64 GB 的内存,同时处理数百个数据集,每个数据集数十 GB /em>.
但我可以给你一些提示:AWE 视图交换相当慢,因为它在交换期间强制暂停所有正在运行的线程。因此,明智地选择窗口大小(窗口越小,交换速度越快,但当然,窗口大小越大,调用开销就越低)。我们已将 AWE 视图大小设置为等于 Windows 默认页面大小 (4 KB),但这只是因为随机访问以这种方式执行效果最佳。 Lineair 数据访问可以通过更大的视图尺寸运行得更快。
每个视图都可以映射到分配的 AWE 内存的任何部分,因此有帮助的一件事是仅将那些页面映射到需要访问的视图中 - 并尝试节省不必要的视图交换(想到优先级队列) )。
此外,您的设计中应该有一个注册机制来处理视图和其背后的 AWE 内存之间的链接。而且最好是线程安全的!
至于一般用法:不,这不适合常规的 Delphi 类。您应该完全切换到另一个概念 - 并以此为基础您的数据结构。
不管怎样,祝你好运,伙计!你会需要它...;-)
I'm going to be a complete dork, and tell you that I've made something even more advanced than what you're describing.... at work. So it's all closed source I'm afraid. Never saw anything like this anywhere. We combine VM, AWE, MMF and (soon) 32<>64 bit IPC into one big, mean data-processing machine, addressing up to 64 GB of memory, while processing hundreds of datasets, tens of GBs each.
But I can give you a few tips : AWE view-swapping is rather slow, because it forcibly pauses all running threads during the swap. Therefor, choose your window-sizes wisely (the smaller, the faster the swap - but call-overhead is lower with larger sizes ofcourse). We've settled with AWE view-sizes equal to the Windows default page-size (4 KB), but only because random-access performs best that way. Lineair data-access could run faster with bigger view-sizes.
Each view can map to any part of the allocated AWE memory, so one thing that can help is mapping only those pages into a view that need to be accessed - and try to save on unnessecary view-swaps (a priority-queue comes to mind).
Also, there should be a registration-mechanism somewhere in your design that handles the linkage between a view and the AWE memory behind this. And this better be thread-safe!
As for general usage : No, this doesn't fit in with regular Delphi classes. You should switch over to another concept altogether - and base your data-structures on that.
Anyway, good luck mate! You're going to need it... ;-)
有系统调用可以执行此操作但并非所有版本的 Windows 都支持它(特别是 Windows XP 不支持 AWE)。
透明度会成为一个问题,因为 API 无法返回指向对象的指针。将超过 4GB 的 RAM 映射到 4GB 地址空间意味着 32 位指针可能不明确 - 您可能会将不同的对象映射到同一位置。
这种歧义意味着您必须为持有可用于访问“记录”的句柄的对象生成代理。某些 SQL Server 版本使用此技术将磁盘缓冲区存储在 AWE 内存中。像这样的方法可能适用于矩阵中的行,其中操作是在整行上完成的。更细粒度的访问会更加繁琐。
为了提供对映射对象的直接访问,您必须实现一个协议,其中可以使用指向映射内存的临时指针。这还需要在使用时将对象锁定在内存中 - 再次,透明度会消失。
假设您现在可以获得 64 位版本的 Delphi,对于需要更多 RAM 的客户,您最好选择 64 位版本的 Windows。
There are system calls that can do this but it is not supported on all versions of Windows (in particular, Windows XP does not support AWE).
Transparency would be something of an issue as the API could not return pointers to objects. Mapping more than 4GB of RAM into a 4GB address space means that a 32 bit pointer could be ambiguous - you could potentially map different objects into the same location.
This ambiguity means that you would have to generate proxies for the objects which hold a handle that could be used to access the 'record'. Some SQL server versions use this technique to store disk buffers in AWE memory. An approach like this would probably work for something like rows in a matrix where the operations are done on the whole row. Finer grained access would be more fiddly.
In order to provide direct access to the mapped object you would have to implement a protocol where a temporary pointer to the mapped memory was made available. This would also require the object to be locked in memory while in use - again, bang goes your transparency.
Assuming you can get a 64 bit version of Delphi now you might be better off going to a 64 bit version of Windows for customers that need more RAM.
您声明您不想迁移到数据库,但是数据库怎么样 专门使用AWE?
我没有亲自尝试过,但会考虑在我自己的项目中使用该公司的产品。
[编辑]:NexusDB 是 Delphi 友好的:它起源于旧的 Turbopower FlashFiler 开发(但从那时起已经发展了很长一段路)。
You state that you do not want to move to a database, but what about a database that specifically uses AWE?
I've not tried it personally, but would consider using products from this company for my own projects.
[Edit]: NexusDB is Delphi-friendly: it originated from the old Turbopower FlashFiler development (but has moved on a long way since then).
AWE 的问题是,它的工作方式与旧的基于 DOS 的 EMS 和 XMS 非常相似 - 如果您曾经使用过它们的话。基本上,保留一定范围的可寻址内存,然后在需要时将可寻址范围之外的内存映射到可寻址范围,并在不再需要时取消映射,从而允许将其他内存映射到相同的地址。因此,大多数不支持 AWE 的数据结构或容器在这种情况下都无法工作——可能 TMemoryStream 后代更容易构建。构建一个在 AWE 内存中存储数据的 TList 等应该很容易,它应该跟踪数据的实际存储位置并在需要时调用它们,并在数据映射到可寻址内存时调整地址。我不知道有任何使用 AWE 的 Delphi 容器库,并且还有另一个问题:桌面 32 位操作系统不能使用超过 4GB 的物理 RAM,需要服务器版本,并且支持的物理 RAM 取决于所使用的版本,请参阅此处< /a> 以获得完整列表。
The issue with AWE it works very much alike the old, DOS-based EMS and XMS - if you ever used them. Basically, a range of addressable memory is reserved, and the memory outside the addressable range is then mapped to the addressable range when needed, and unmapped when no longer need, allowing other memory to be mapped at the same addresses. Thereby most non-AWE aware data structures or containers wouldn't work in such a scenario - probably a TMemoryStream descendant is easier to build. It should be easy enough to build a TList or the like that store data in AWE memory, it should keep track where the data are really stored and recall them when needed, adjusting addresses as well when data are mapped to addressable memory. I am not aware of any Delphi containers library using AWE, and there is another issue: desktop 32 bit operating systems can't use more than 4GB of physical RAM, a server version would be required, and the supported physical RAM depends on what version is used, see here for a complete list.
假设数据一次性加载并适合可用内存,NexusDB AWE 将非常非常快。该数据库可以创建为仅内存中的数据库,然后在操作时不需要任何进一步的硬盘驱动器访问。
Assuming the data is loaded once in bulk and fits available memory, NexusDB AWE will be very very fast. The database can be created as an in-memory only DB and will then not need any further harddrive access while manipulating.
在我看来,你们可能会考虑放弃当前的数据库 SQL 后端,转而采用 100% NexusDB + AWE 解决方案。
更确切地说,放弃对 SQL 后端的日常访问,并拥有一个导出/同步功能,可以将任何所需的 NexusDB 报告数据写入 MSSQL 报告数据库。)
(或者
Sounds to me like you guys might consider dropping the current database SQL backend and going to a 100% NexusDB + AWE solution.
(Or rather, dropping the day to day access to the SQL backend, and having an export/sync function that can write out any required NexusDB reporting data to an MSSQL reporting db.)
W
您的情况听起来与我们的类似,我们的应用程序使用存储在内存映射文件中的巨大数据文件。这些文件约为 750MB,我们从中分配数据结构,最多使用 1.5GB RAM。
遗憾的是,除了将部分内存移至 FPC/Lazarus 直至 Delphi 成为 64 位之前,我们没有找到任何解决 4GB 限制的解决方案。 AWE 不支持 Vista Home 版本,我们也无法让它支持 MMF。
您可以尝试使用滑动窗口的内存映射文件,这意味着您可以根据应用程序正在使用的文件部分来动态创建文件不同块的视图。听起来这行不通,因为您需要立即将整个文件存储在内存中。
Your situation sounds similar to ours, our application uses a huge datafile that we store in a memory-mapped file. The files are around 750MB, and we allocate data structures from them that use up to 1.5GB of RAM.
We have found no solution to the 4GB limit other than moving some of it off to FPC/Lazarus until Delphi is 64-bit, unfortunately. AWE does not work with Vista Home versions, also we couldn't get it to work with MMFs.
You could try memory-mapped files with a sliding window, meaning you dynamically create views of different chunks of the file depending on what part of it the application is using. Sounds like that won't work though because you need the entire file in memory at once.