使用多处理时出现奇怪的 unpickling 错误
使用多处理时出现以下错误：
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.6/threading.py", line 525, in __bootstrap_inner
self.run()
File "/usr/lib/python2.6/threading.py", line 477, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.6/multiprocessing/pool.py", line 282, in _handle_results
task = get()
UnpicklingError: NEWOBJ class argument has NULL tp_new
我完全不知道这意味着什么，尽管听起来像是 C 级别的问题。有人能解释一下吗？
更新：好的，我想出了如何解决这个问题。但我还是有点困惑。我正在返回此类的实例：
class SpecData(object):
def __init__(self, **kwargs):
self.__dict__.update(**kwargs)
def to_dict(self):
return self.__dict__
如果我返回此对象的实例，则会收到错误。但是，如果我调用 to_dict 并返回字典，它就可以工作。我做错了什么？
I get the following error when using multiprocessing:
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.6/threading.py", line 525, in __bootstrap_inner
self.run()
File "/usr/lib/python2.6/threading.py", line 477, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.6/multiprocessing/pool.py", line 282, in _handle_results
task = get()
UnpicklingError: NEWOBJ class argument has NULL tp_new
I have absolutely no idea what this means, although it sounds like something's wrong at the C level. Can anyone shed some light on this?
UPDATE: Ok, so I figured out how to fix this. But I'm still a bit perplexed. I'm returning an instance of this class:
class SpecData(object):
def __init__(self, **kwargs):
self.__dict__.update(**kwargs)
def to_dict(self):
return self.__dict__
If I return an instance of this object, I get the error. However, if I call to_dict and return a dictionary, it works. What am I doing wrong?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（3）
尝试使用
pickle模块而不是cPickle模块 -pickle是用纯 Python 编写的，通常它会提供比cPickle更有用的错误消息代码>cPickle。 （尽管有时我不得不制作pickle.py的本地副本，并在错误位置附近添加一些调试 printf 语句来找出问题。）一旦跟踪解决这个问题后，你可以切换回 cpickle。
（我对多处理模块不太熟悉，所以我不确定你是否在进行酸洗。如果是，那么让它使用 pickle 而不是 cpickle 的最简单方法可能是在导入多处理/线程模块之前进行一些猴子修补： import sys, pickle;
Try using the
picklemodule rather than thecPicklemodule --pickleis written in pure Python, and often it gives more useful error messages thancPickle. (Though sometimes I've had to resort to making a local copy ofpickle.py, and adding in a few debug printf statements near the location of the error to figure out the problem.)Once you track down the problem, you can switch back to cpickle.
(I'm not that familiar with the multiprocessing module, so I'm not sure whether you're doing the pickling or it is. If it is, then the easiest way to get it to use pickle rather than cpickle may be to do some monkey-patching before you import the multiprocessing/threading module:
import sys, pickle; sys.modules['cPickle']=pickle)我认为这是一些 python 函数的可挑选性/不可挑选性的问题。请参阅这篇文章：
http://khinsen.wordpress。 com/2012/02/06/teaching-parallel-computing-in-python/
我在使用 django-celery （使用多处理模块）时遇到了类似的问题。如果我的任务代码抛出本身不可pickelable的错误，则此多进程/pickle异常会掩盖信息。因为我还没有找到更好的方法来传播错误，所以我求助于任务代码中的调试日志记录来查找错误。我可能应该更聪明地了解要传递到队列的内容（防止将异常放入消息队列，因此多进程模块不会尝试腌制/取消腌制它们）。
在上面的情况下，您可能需要确保 SpecData.__dict__ 是可挑选的。请参阅 http://docs.python.org/library/pickle.html#pickle-协议
I think this a problem with pickability/unpickelability of some python functions. See this post:
http://khinsen.wordpress.com/2012/02/06/teaching-parallel-computing-in-python/
I have had similar problems using django-celery (which uses the multiprocessing module). If my task code thows errors which are not themselves pickelable, this multiprocess/pickle exception obscures the information. Because I haven't figured out a better way to propogate the errors, I resort to debug logging in my task code to hunt it down. I should probably get smarter about what I'm passing onto the queue (guard against putting exceptions onto the message queue, so the multiprocess module doesn't try to pickle/unpickle them).
In your case above, you might need to make sure
SpecData.__dict__is pickelable. See http://docs.python.org/library/pickle.html#pickle-protocol我已经在 C++、Java 和 Delphi 中完成了线程安全，但没有在 Python 中完成，所以请对我的评论持保留态度。
这个关于 Python 和线程安全的页面 特别提到了字典的原子分配并且线程安全。也许您对自定义类的引用不是线程安全的？如果您仍然希望在两个线程之间传递自定义容器类，请尝试添加一些推荐的锁定机制。
令我着迷的是，其他搜索结果都强调 Python 是完全线程安全的。 Python 文档本身声明提供了锁和其他机制来帮助对于线程应用程序，所以看起来是互联网出错的情况（这种情况真的会发生吗？）。
关于 Python 和线程安全的另一个 StackOverflow 问题。
I've done thread-safety in C++, Java, and Delphi, but not Python, so take my comments with a grain of salt.
This page on Python and Thread-Safety specifically mentions the assignment of a dictionary to be atomic and thread-safe. Perhaps your reference to your custom class is not thread-safe? Try adding some of the recommended locking mechanisms if you would still rather pass a custom container class between two threads.
I find it fascinating that other search results state emphatically that Python is completely thread-safe. The Python docs themself state that locks and other mechanisms are provided to help with threadded applications, so looks like it's the case of the internets being wrong (does that even happen??).
Another StackOverflow question on python and thread-safety.