当前位置：文江博客话题详情

如何让 unpickling 与 iPython 一起工作？

发布于 2024-09-13 11:12:57 字数 649 浏览 22 评论 0原文

我正在尝试在 iPython 中加载腌制对象。

我收到的错误是：

AttributeError：“FakeModule”对象没有属性“World”

有人知道如何让它工作，或者至少知道在 iPython 中加载对象以便交互式浏览它们的解决方法吗？

感谢

编辑添加：

我有一个名为 world.py 的脚本，它的作用基本上是：

import pickle
class World:
    ""
if __name__ == '__main__':
    w = World()
    pickle.dump(w, open("file", "wb"))

比在 REPL 中我所做的：

import pickle  
from world import World  
w = pickle.load(open("file", "rb"))

它在普通 python REPL 中工作，但不适用于 iPython。

我使用的是来自 Enthought Python Distribution 的 Python 2.6.5 和 iPython 0.10，但我也遇到了以前版本的问题。

原文

I'm trying to load pickled objects in iPython.

The error I'm getting is:

AttributeError: 'FakeModule' object has no attribute 'World'

Anybody know how to get it to work, or at least a workaround for loading objects in iPython in order to interactively browse them?

Thanks

edited to add:

I have a script called world.py that basically does:

import pickle
class World:
    ""
if __name__ == '__main__':
    w = World()
    pickle.dump(w, open("file", "wb"))

Than in a REPL I do:

import pickle  
from world import World  
w = pickle.load(open("file", "rb"))

which works in the vanilla python REPL but not with iPython.

I'm using Python 2.6.5 and iPython 0.10 both from the Enthought Python Distribution but I was also having the problem with previous versions.

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

紫瑟鸿黎 2024-09-20 11:12:57

看起来您在腌制数据和尝试取消腌制数据之间修改了 FakeModule ：具体来说，您已从该模块中删除了一些名为 的顶级对象World（也许是一个类，也许是一个函数）。

Pickling“按名称”序列化类和函数，因此它们需要在其模块的顶层命名并且该模块不得被修改（至少不能以严重影响这些名称的方式 - 绝对）不是通过在酸洗时间和取消酸洗时间之间从模块中删除这些名称！）。

一旦您准确地确定了所做的哪些更改阻碍了 unpickle，如果由于其他原因您无法恢复更改，则通常可以对其进行修改。例如，如果您刚刚将 World 从 FakeModule 移动到 CoolModule，请

import FakeModule
import CoolModule
FakeModule.World = CoolModule.World

在 unpickle 之前执行以下操作（并记住使用新的再次 pickle）结构，这样你就不必每次解封时都重复这些技巧；-)。

编辑：OP对Q的编辑使他的错误更容易理解。由于他现在正在测试 __name__ 是否等于 '__main__'，因此很明显，在写入时，pickle 将保存类 __main__.World< 的对象。 /代码>。由于他使用的是 ASCII pickles（顺便说一句，对于性能和磁盘空间来说，这是一个非常糟糕的选择），因此检查起来很简单：

$ cat file
(i__main__
World
p0
(dp1

正在查找的模块是（清楚明了）__main__.现在，甚至不用打扰 ipython，而是使用简单的 Python 交互式解释器：

$ py26
Python 2.6.5 (r265:79359, Mar 24 2010, 01:32:55) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import world
>>> import pickle
>>> pickle.load(open("file", "rb"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/pickle.py", line 1370, in load
    return Unpickler(file).load()
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/pickle.py", line 858, in load
    dispatch[key](self)
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/pickle.py", line 1069, in load_inst
    klass = self.find_class(module, name)
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/pickle.py", line 1126, in find_class
    klass = getattr(mod, name)
AttributeError: 'module' object has no attribute 'World'
>>>

可以轻松地重现该错误，其原因也同样明显：执行类名查找的模块（即 __main__) 确实没有名为“World”的属性。模块 world 确实有一个，但是 OP 没有“连接点”，正如我在答案的前一部分中解释的那样，在腌制文件需要的模块中放置具有正确名称的引用它。也就是说：

>>> World = world.World
>>> pickle.load(open("file", "rb"))
<world.World instance at 0xf5300>
>>>

当然，现在这工作得非常完美（正如我之前所说的）。也许OP没有看到这个问题，因为他使用了我讨厌的导入形式，from world import World（直接从模块内导入函数或类，而不是模块本身）。

在 ipython 中解决这个问题的方法在底层 Python 架构方面是完全相同的——只需要多几行代码，因为 ipython 提供所有额外的服务，不模块 __main__ 直接可用于直接记录交互式命令行中发生的情况，而是插入一个模块（称为 FakeModule，正如 OP 从错误消息中发现的那样；-）并按顺序对其执行黑魔法变得“酷”&c。尽管如此，每当你想直接访问具有给定名称的模块时，这在 Python 中都是相当简单的，当然：

In [1]: import world

In [2]: import pickle

In [3]: import sys

In [4]: sys.modules['__main__'].World = world.World

In [5]: pickle.load(open("file", "rb"))
Out[5]: <world.World instance at 0x118fc10>

In [6]:

要记住的教训，第一：避免黑魔法，至少除非并且直到你足够优秀作为巫师的学徒能够发现并修复它偶尔失控的情况（否则，那些提着水桶的扫帚可能会在你打盹时淹没世界；-）。

或者，另一种解读：要正确使用某个抽象层（例如 ipython 在 Python 之上放置的“酷”抽象层），您需要对底层有深入的了解（这里是 Python 本身及其核心机制，例如 pickling 和 sys .模块）。

第二课：由于您编写的方式，该 pickle 文件本质上已损坏，因为只有当模块 __main__ 具有名为 Word 的类时才能加载它，当然，如果没有像上面这样的一些技巧，它通常不会有。 pickle 文件应该将类记录为存在于模块 world 中。如果您绝对认为您必须在world.py中的if __name__ == '__main__':子句上生成文件，那么请使用一些冗余目的：

import pickle
class World:
    ""
if __name__ == '__main__':
    import world
    w = world.World()
    pickle.dump(w, open("file", "wb"))

这工作得很好，没有黑客（至少如果你遵循Python最佳实践，在模块顶层永远不要有任何实质性代码——只有导入、类、def和琐碎的赋值——其他一切都属于函数；如果您没有遵循此最佳实践，请编辑您的代码来执行此操作，这将使您在灵活性和性能方面更加满意）。

Looks like you've modified FakeModule between the time you pickled your data, and the time you're trying to unpickle it: specifically, you have removed from that module some top-level object named World (perhaps a class, perhaps a function).

Pickling serializes classes and function "by name", so they need to be names at their module's top level and that module must not be modified (at least not in such way to affect those names badly -- definitely not by removing those names from the module!) between pickling time and unpickling time.

Once you've identified exactly what change you've done that impedes the unpickling, it can often be hacked around if for other reasons you can't just revert the change. For example, if you've just moved World from FakeModule to CoolModule, do:

import FakeModule
import CoolModule
FakeModule.World = CoolModule.World

just before unpickling (and remember to pickle again with the new structure so you won't have to keep repeating these hacks every time you unpickle;-).

Edit: the OP's edit of the Q makes his error much easier to understand. Since he's now testing if __name__ equals '__main__', this makes it obvious that the pickle, when written, will be saving an object of class __main__.World. Since he's using ASCII pickles (a very bad choice for performance and disk space, by the way), it's trivial to check:

$ cat file
(i__main__
World
p0
(dp1

the module being looked up is (clearly and obviously) __main__. Now, without even bothering ipython but with a simple Python interactive interpreter:

$ py26
Python 2.6.5 (r265:79359, Mar 24 2010, 01:32:55) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import world
>>> import pickle
>>> pickle.load(open("file", "rb"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/pickle.py", line 1370, in load
    return Unpickler(file).load()
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/pickle.py", line 858, in load
    dispatch[key](self)
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/pickle.py", line 1069, in load_inst
    klass = self.find_class(module, name)
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/pickle.py", line 1126, in find_class
    klass = getattr(mod, name)
AttributeError: 'module' object has no attribute 'World'
>>>

the error can be easily reproduced, and its reason is just as obvious: the module in which the class name's lookup is performed (that is, __main__) does indeed have no attribute named "World". Module world does have one, but the OP has not "connected the dots" as I explained in the previous part of the answer, putting a reference with the right name in the module in which the pickled file needs it. That is:

>>> World = world.World
>>> pickle.load(open("file", "rb"))
<world.World instance at 0xf5300>
>>>

now this works just perfectly, of course (and as I'd said earlier). Perhaps the OP is not seeing this problem because he's using the form of import I detest, from world import World (importing directly a function or class from within a module, rather than the module itself).

The hack to work around the problem in ipython is exactly the same in terms of underlying Python architecture -- just requires a couple more lines of code because ipython, to supply all of its extra services, does not make module __main__ directly available to record directly what happens at the interactive command line, but rather interposes one (called FakeModule, as the OP found out from the error msg;-) and does black magic with it in order to be "cool" &c. Still, whenever you want to get directly to a module with a given name, it's pretty trivial in Python, of course:

In [1]: import world

In [2]: import pickle

In [3]: import sys

In [4]: sys.modules['__main__'].World = world.World

In [5]: pickle.load(open("file", "rb"))
Out[5]: <world.World instance at 0x118fc10>

In [6]:

Lesson to retain, number one: avoid black magic, at least unless and until you're good enough as a sorcerer's apprentice to be able to spot and fix its occasional runaway situations (otherwise, those bucket-carrying brooms may end up flooding the world while you nap;-).

Or, alternative reading: to properly use a certain layer of abstraction (such as the "cool" ones ipython puts on top of Python) you need strong understanding of the underlying layer (here, Python itself and its core mechanisms such as pickling and sys.modules).

Lesson number two: that pickle file is essentially broken, due to the way you've written it, because it can be loaded only when module __main__ has a class by name Word, which of course it normally will not have without some hacks like the above. The pickle file should instead record the class as living in module world. If you absolutely feel you must produce the file on an if __name__ == '__main__': clause in world.py, then use some redundancy for the purpose:

import pickle
class World:
    ""
if __name__ == '__main__':
    import world
    w = world.World()
    pickle.dump(w, open("file", "wb"))

this works fine and without hacks (at least if you follow the Python best practice of never having any substantial code at module top level -- only imports, class, def, and trivial assignments -- everything else belongs in functions; if you haven't followed this best practice, then edit your code to do so, it will make you much happier in terms of both flexibility and performance).

回复收藏 0 原文

木森分化 2024-09-20 11:12:57

当您使用 pickle.dump(w, open("file", "wb")) 在 __main__ 模块中 pickle w 时，事实来自 __main__ 模块的 w 记录在 file 的第一行：

% xxd file
0000000: 2869 5f5f 6d61 696e 5f5f 0a57 6f72 6c64  (i__main__.World
0000010: 0a70 300a 2864 7031 0a62 2e              .p0.(dp1.b.

当 IPython 尝试 unpickle file 时，它执行这些行：

/usr/lib/python2.6/pickle.pyc in find_class(self, module, name)
   1124         __import__(module)
   1125         mod = sys.modules[module]
-> 1126         klass = getattr(mod, name)
   1127         return klass
   1128

特别是，它尝试执行 __import__('__main__')。如果您在 REPL 中尝试这样做，您会得到

In [29]: fake=__import__('__main__')

In [32]: fake
Out[32]: <module '__main__' from '/usr/lib/pymodules/python2.6/IPython/FakeModule.pyc'>

This is the FakeModule IPython 在 AttributeError 中提到的。

如果您查看 fake.__dict__ 内部，您会发现它不包含 World，即使您在之前或之后说 from test import World __import__。

如果你运行

In [35]: fake.__dict__['World']=World

那么 pickle.load 将会工作：

In [37]: w = pickle.load(open("file", "rb"))

可能有一个更干净的方法；我不知道。您能想到的任何将 World 放入 fake 命名空间的方法都应该可行。

附言。 2008 年，IPython 的创建者 Fernando Perez 写了一点关于这个问题。他可能已经以某种方式解决了这个问题，以避免我的肮脏黑客行为。您可能想在 IPython 用户邮件列表上询问，或者，也许更简单，只是不要在 __main__ 命名空间内进行 pickle。

When you pickle w in the __main__ module with pickle.dump(w, open("file", "wb")), the fact that w comes from the __main__ module is recorded on the first line of file:

% xxd file
0000000: 2869 5f5f 6d61 696e 5f5f 0a57 6f72 6c64  (i__main__.World
0000010: 0a70 300a 2864 7031 0a62 2e              .p0.(dp1.b.

When IPython tries to unpickle file, it executes these lines:

/usr/lib/python2.6/pickle.pyc in find_class(self, module, name)
   1124         __import__(module)
   1125         mod = sys.modules[module]
-> 1126         klass = getattr(mod, name)
   1127         return klass
   1128

In particular, it tries to execute __import__('__main__'). If you try that in the REPL, you get

In [29]: fake=__import__('__main__')

In [32]: fake
Out[32]: <module '__main__' from '/usr/lib/pymodules/python2.6/IPython/FakeModule.pyc'>

This is the FakeModule that IPython mentions in the AttributeError.

If you look inside fake.__dict__ you'll see it doesn't include World even if you say from test import World before or after the __import__.

If you run

In [35]: fake.__dict__['World']=World

Then pickle.load will work:

In [37]: w = pickle.load(open("file", "rb"))

There might be a cleaner way; I don't know. Any way you can think of that puts World in the fake namespace should work.

PS. In 2008 Fernando Perez, the creator of IPython, wrote a little bit on this issue. He might have fixed this in some way that avoid my dirty hack. You might want to ask on the IPython-user mailing list, or, perhaps simpler, just don't pickle inside the __main__ namespace.

回复收藏 0 原文

~没有更多了~