Native objects don't get shared between processes (due to reference counting).
Instead, you can pickle them and share them using unix domain sockets, mmap, zeromq, or an intermediary such a sqlite3 that is designed for concurrent accesses.
After some test, I found that the following approach works for Linux using mmap.
Linux has /dev/shm. If you create a shared memory using POSIX shm_open, a new file is created in this folder.
Although python's mmap module does not provide the shm_open function. we can use a normal open to create a file in /dev/shm and it is actually similar and reside in memory. (Use os.unlink to remove it)
Then for IPC, we can use mmap to map that file to the different processes' virtual memory space. All the processes share that memory. Python can use the memory as buffer and create object such as bytes and numpy arrays on top of it. Or we can use it through the ctypes interface.
Of course, process sync primitives are still needed to avoid race conditions.
Parallel Python 可能值得一看，它可以在 Windows、OS X 和 Linux 上运行（我似乎记得我不久前在 UltraSPARC Solaris 10 计算机上使用过它）。我不知道它是否适用于 PyPy，但它 似乎确实可以与 Psyco 配合使用。
Parallel Python might be worth a look, it works on Windows, OS X, and Linux (and I seem to recall I used it on a UltraSPARC Solaris 10 machine a while back). I don't know if it works with PyPy, but it does seem to work with Psyco.
发布评论
评论（5）
本机对象不会在进程之间共享（由于引用计数）。
相反，您可以使用 unix 域套接字、mmap、zeromq 或专为并发访问设计的 sqlite3 等中介来腌制并共享它们。
Native objects don't get shared between processes (due to reference counting).
Instead, you can pickle them and share them using unix domain sockets, mmap, zeromq, or an intermediary such a sqlite3 that is designed for concurrent accesses.
使用多处理开始。
如果您需要多个 CPU，请查看 celery。
Use multiprocessing to start with.
If you need multiple CPU's, look at celery.
经过一些测试，我发现以下方法适用于使用
mmap的 Linux。Linux 有
/dev/shm。如果您使用 POSIXshm_open创建共享内存，则会在此文件夹中创建一个新文件。虽然python的
mmap模块没有提供shm_open函数。我们可以使用普通的open在/dev/shm中创建一个文件，它实际上是类似的并且驻留在内存中。 （使用os.unlink删除它）然后对于IPC，我们可以使用
mmap将该文件映射到不同进程的虚拟内存空间。所有进程共享该内存。 Python 可以使用内存作为缓冲区，并在其上创建字节和 numpy 数组等对象。或者我们可以通过ctypes接口来使用它。当然，仍然需要进程同步原语来避免竞争条件。
请参阅 mmap 文档、ctypes 文档 和
numpy.load其中有一个mmap_mode选项。After some test, I found that the following approach works for Linux using
mmap.Linux has
/dev/shm. If you create a shared memory using POSIXshm_open, a new file is created in this folder.Although python's
mmapmodule does not provide theshm_openfunction. we can use a normalopento create a file in/dev/shmand it is actually similar and reside in memory. (Useos.unlinkto remove it)Then for IPC, we can use
mmapto map that file to the different processes' virtual memory space. All the processes share that memory. Python can use the memory as buffer and create object such as bytes and numpy arrays on top of it. Or we can use it through thectypesinterface.Of course, process sync primitives are still needed to avoid race conditions.
See mmap doc, ctypes doc and
numpy.loadwhich has anmmap_modeoption.execnet 和 Pyro 提及
PyPy <-> CPython 通信。 Python Wiki 的 并行处理 页面中的其他包可能也适用。Both execnet and Pyro mention
PyPy <-> CPythoncommunication. Other packages from Python Wiki's Parallel Processing page are probably suitable too.Parallel Python 可能值得一看，它可以在 Windows、OS X 和 Linux 上运行（我似乎记得我不久前在 UltraSPARC Solaris 10 计算机上使用过它）。我不知道它是否适用于 PyPy，但它 似乎确实可以与 Psyco 配合使用。
Parallel Python might be worth a look, it works on Windows, OS X, and Linux (and I seem to recall I used it on a UltraSPARC Solaris 10 machine a while back). I don't know if it works with PyPy, but it does seem to work with Psyco.