我想分配并初始化相当大的连续内存块(~1GB),然后将其标记为只读并分叉多个(比如说几十个)将使用它的子进程,而不制作自己的内存副本(机器没有足够的内存来执行此操作)。
我是否正确地认为,如果我 malloc
内存为通常,然后使用 mprotect(addr, size, PROT_READ)
然后 fork
,这将允许子进程安全地使用内存而不导致内存被复制? (前提是我确保在 mprotect 调用之后没有任何内容尝试写入分配的内存)。
编辑:感谢您的所有回答。
后续问题 - 我计划使用 shmget
,但是我认为它使用 mm
因此将仅限于较小的分配(请参阅本页的限制部分)。例如 /proc/sys/kernel/shmmax
在我使用的服务器上是 32MB。但我想要 1GB 的连续内存。我对这个限制有错吗?
I want to allocate and initialise a fairly large chunk of contiguous memory (~1GB), then mark it as read-only and fork multiple (say several dozen) child processes which will use it, without making their own copies of the memory (the machine won't have enough memory for this).
Am I right in thinking that if I malloc
the memory as usual, then mark it as read-only with mprotect(addr, size, PROT_READ)
and then fork
, this will allow the child processes to safely use the memory without causing it to be copied? (Providing I ensure nothing tries to write to the allocated memory after the mprotect
call).
edit: Thanks for all the answers.
A followup question - I was planning on using shmget
, but I thought it used mm
and thus would be limited to smaller allocations (see the Restrictions section of this page). eg /proc/sys/kernel/shmmax
is 32MB on the server I'm using this one. But I want 1GB of contiguous memory. Am I wrong about this limitation?
发布评论
评论(6)
man mprotect
mprotect
仅适用于页面,而不适用于任意字节范围,因此一般来说malloc
是不合适的。posix_memalign
可能有帮助,但是......mprotect
任何您自己没有mmap
的内容。请改用mmap(0, pages*sysconf(_SC_PAGESIZE), PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS, -1, 0)
。man mprotect
mprotect
only works on pages, not arbitrary byte ranges, so in generalmalloc
is inappropriate.posix_memalign
may help, but...mprotect
anything that you didn'tmmap
yourself. Usemmap(0, pages*sysconf(_SC_PAGESIZE), PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS, -1, 0)
instead.您的理由是不正确的,因为任何子进程都可以调用
mprotect()
来删除保护并开始在那里写入。如果页面没有被复制,就会违反fork()
的原则。即使它以写时复制用于分叉进程的方式工作,我也不认为标准中有任何地方这么说(POSIX 没有说它是写时复制,例如)。
您可以使用标准措施来共享内存,而不是使用非标准行为。例如,带有
shm_open
的 POSIX 共享内存以及后续的mmap
(正如评论中指出并解释的那样)他的帖子 作者:ephemient)。文件描述符将通过分叉保留。You are not right by the reason than any of child processes can call
mprotect()
to remove protection and start writing there. If the pages have not been copied, it would violate the principles offork()
.Even if it works that way that copy-on-write is used for forked processes, I don't reckon any place in standards that says so (POSIX doesn't say it's copy-on-write, for instance).
Instead of using non-standard behavior, you may use standard measures to share memory. For example, the POSIX shared memory with
shm_open
and consequentmmap
(as was pointed out in comment and explained in his post by ephemient). The file descriptor will be preserved through forking.无需将其标记为只读,只需让您的子进程不理会它即可。
如果父级和子级都没有写入它,则它应该保持共享。如果你不想改变它,那也没关系。
如果你想写入它,你需要将 mmap 与 MAP_SHARED 一起使用。
There is no need to mark it read-only, just get your child processes to leave it alone.
If neither the parent nor child writes to it, it should remain shared. If you don't ever want to change it, that's fine.
If you want to write to it, you'll want to use mmap with MAP_SHARED.
您假设内核将进行写时复制优化,而不复制
mprotect
-ed页面。但我不会指望它。malloc
-ed 内存周围漂浮着各种元数据 - 保护页等,只有 Ulrich Drepper 知道 libc 内部发生了什么:)在中准备数据可能会更容易、更安全一个磁盘文件,然后将其
mmap
到所有进程中,或者只是走正常的 POSIXshm_open
路线。You are making an assumption that the kernel would do copy-on-write optimization and not copy the
mprotect
-ed pages. I would'n count on it though.malloc
-ed memory has all sorts of metadata floating around it - guard pages, etc. etc. and only Ulrich Drepper knows what's going on inside libc :)It would probably be easier and safer to prepare data in a disk file and them
mmap
it into all the processes, or just go normal POSIXshm_open
route.我的理解是肯定的,因为Linux使用写时复制机制来传递给子进程的内存页面。
My understanding is yes, since Linux uses a copy-on-write mechanism for memory pages passed to a child process.
你可以这样做。
另一种方法是使用 mmap()。
另一种选择是使用 POSIX 共享内存 (shm_open()) ;另一个主要替代方案是 System V 共享内存 (shmget(), shmat())。正式共享内存系统的优点之一是您的父进程可以创建内存,然后不相关的进程可以连接到它 - 如果这是有益的。
You could do it that way.
An alternative is to use mmap().
Another alternative is to use the POSIX shared memory (shm_open()); the other main alternative is System V shared memory (shmget(), shmat()). One advantage of the formal shared memory systems is that your parent process can create the memory and then unrelated process could connect to it - if that was beneficial.