Python 的“tarfile”模块是否将其构建的档案存储在内存中？

发布于 2024-10-21 04:31:25 字数 390 浏览 1 评论 0原文

我在内存受限的环境中工作，需要对 SQL 转储进行存档。如果我使用 python 内置的 tarfile 模块是 '. tar' 文件保存在内存中还是在创建时写入磁盘？

例如，在下面的代码中，如果huge_file.sql是2GB，那么tar变量会占用2GB内存吗？

import tarfile

tar = tarfile.open("my_archive.tar.gz")), "w|gz")
tar.add('huge_file.sql')
tar.close()

原文

I'm working in a memory constrained environment where I need to make archives of SQL dumps. If I use python's built in tarfile module is the '.tar' file held in memory or written to disk as it's created?

For instance, in the following code, if huge_file.sql is 2GB will the tar variable take up 2GB in memory?

import tarfile

tar = tarfile.open("my_archive.tar.gz")), "w|gz")
tar.add('huge_file.sql')
tar.close()

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

情绪操控生活 2024-10-28 04:31:25

不，它没有将其加载到内存中。您可以阅读 tarfile 源来查看它使用 < code>copyfileobj，它使用固定大小的缓冲区从文件复制到 tarball：

def copyfileobj(src, dst, length=None):
    """Copy length bytes from fileobj src to fileobj dst.
       If length is None, copy the entire content.
    """
    if length == 0:
        return
    if length is None:
        shutil.copyfileobj(src, dst)
        return

    BUFSIZE = 16 * 1024
    blocks, remainder = divmod(length, BUFSIZE)
    for b in xrange(blocks):
        buf = src.read(BUFSIZE)
        if len(buf) < BUFSIZE:
            raise IOError("end of file reached")
        dst.write(buf)

    if remainder != 0:
        buf = src.read(remainder)
        if len(buf) < remainder:
            raise IOError("end of file reached")
        dst.write(buf)
    return

No it is not loading it in memory. You can read the source for tarfile to see that it's using copyfileobj, which is using a fixed size buffer to copy from the file to the tarball:

def copyfileobj(src, dst, length=None):
    """Copy length bytes from fileobj src to fileobj dst.
       If length is None, copy the entire content.
    """
    if length == 0:
        return
    if length is None:
        shutil.copyfileobj(src, dst)
        return

    BUFSIZE = 16 * 1024
    blocks, remainder = divmod(length, BUFSIZE)
    for b in xrange(blocks):
        buf = src.read(BUFSIZE)
        if len(buf) < BUFSIZE:
            raise IOError("end of file reached")
        dst.write(buf)

    if remainder != 0:
        buf = src.read(remainder)
        if len(buf) < remainder:
            raise IOError("end of file reached")
        dst.write(buf)
    return

回复收藏 0 原文

~没有更多了~