如何在不读取压缩文件全部内容的情况下对其进行尾随？
我想模拟gzcat的功能 | 尾部-n.
当存在巨大文件（几 GB 左右）时，这会很有帮助。 我可以尾随此类文件的最后几行而不从头读取它吗？ 我怀疑这是不可能的，因为我猜测对于 gzip，编码将取决于之前的所有文本。
但我仍然想听听是否有人尝试过做类似的事情 - 也许调查可以提供此类功能的压缩算法。
I want to emulate the functionality of gzcat | tail -n.
This would be helpful for times when there are huge files (of a few GB's or so). Can I tail the last few lines of such a file w/o reading it from the beginning? I doubt that this won't be possible since I'd guess for gzip, the encoding would depend on all the previous text.
But still I'd like to hear if anyone has tried doing something similar - maybe investigating over a compression algorithm that could provide such a feature.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（7）
不，你不能。 压缩算法适用于流并调整其对流包含的内容进行内部编码以实现其高压缩比。
如果不知道某个点之前流的内容是什么，就不可能知道如何从该点开始解压缩。
任何允许您解压缩其任意部分的算法都需要多次传递数据才能压缩它。
No, you can't. The zipping algorithm works on streams and adapts its internal codings to what the stream contains to achieve its high compression ratio.
Without knowing what the contents of the stream are before a certain point, it's impossible to know how to go about de-compressing from that point on.
Any algorithm which allows you to de-compress arbitrary parts of it will require multiple passes over the data to compress it.
BGZF 用于创建 Samtools 创建的索引 gzip 压缩 BAM 文件。 这些是可以随机访问的。
http://samtools.sourceforge.net/
BGZF is used to created index gzip compressed BAM files created by Samtools. These are randomly accessible.
http://samtools.sourceforge.net/
如果您首先可以控制文件中的内容，如果它是类似 ZIP 文件的文件，您可以存储预定大小的块，文件名按递增的数字顺序排列，然后解压缩最后一个块/文件。
If you have control over what goes into the file in the first place, if it's anything like a ZIP file you could store chunks of predetermined size with filenames in increasing numerical order and then just decompress the last chunk/file.
如果可以选择，那么 bzip2 可能是用于此目的的更好的压缩算法。
Bzip2 使用块压缩方案。 因此，如果您确定文件末尾的一块足够大以包含最后一块的所有内容，那么您可以使用 bzip2recover 恢复它。
块大小可以在写入文件时选择。 事实上，当您将 -1（或 --fast）设置为 -9（或 --best）作为压缩选项（对应于 100k 到 900k 的块大小）时，就会发生这种情况。 默认值为 900k。
bzip2 命令行工具没有为您提供一种友好的管道方式来执行此操作，但考虑到 bzip2 不是面向流的，也许这并不奇怪。
If it's an option, then bzip2 might be a better compression algorithm to use for this purpose.
Bzip2 uses a block compression scheme. As such, if you take a chunk of the end of your file which you are sure is large enough to contain all of the last chunk, then you can recover it with bzip2recover.
The block size is selectable at the time the file is written. In fact that's what happens when you set -1 (or --fast) to -9 (or --best) as compression options, which correspond to block sizes of 100k to 900k. The default is 900k.
The bzip2 command line tools don't give you a nice friendly way to do this with a pipeline, but then given bzip2 is not stream oriented, perhaps that's not surprising.
zindex 以节省时间和空间的方式在压缩的、基于行的文本文件上创建和查询索引。
https://github.com/mattgodbolt/zindex
zindex creates and queries an index on a compressed, line-based text file in a time- and space-efficient way.
https://github.com/mattgodbolt/zindex
好吧，如果您之前为每个文件创建了一个索引，您就可以做到这一点...
我开发了一个命令行工具，它可以为 gzip 文件创建索引，该工具允许在其中进行非常快速的随机访问，并且它与操作（提取、尾部、连续尾部等）交错执行此操作：
https://github.com/circulosmeos/gztool
但是你可以做一个尾巴（
- t），并且索引将自动创建：如果您将来要做同样的事情，它会快得多，而且无论如何，第一次它会花费与gunzip相同的时间| 尾部：Well, you can do that if you previously creates an index for each file ...
I've developed a command line tool which creates indexes for gzip files which allow for very quick random access inside them, and it does this interleaved with actions (extract, tail, continuous tail, etc):
https://github.com/circulosmeos/gztool
But you can do a tail (
-t), and the index will be automatically created: if you're gonna do the same in the future it'll be much quicker, and anyway the first time it will take the same time as agunzip | tail:完全兼容 gzip 的伪随机访问格式的示例是
dictzip：An example of a fully gzip-compatible pseudo-random access format is
dictzip: