如何从 Perl 脚本非阻塞地写入 gzip 文件？
我目前正在编写一个脚本，该脚本采用数据库作为输入，并按照某些规则从 10 多个表中生成所有有效组合。由于输出非常巨大，我通过 gzip 将其转储到文件中，如下所示：
open( my $OUT, '|-', "gzip > file" );
for ( @data ) {
my $line = calculate($_);
print $OUT $line;
}
由于野兽的本质，尽管我最终不得不进行数十万次小写入，每一行一个。这意味着在每次计算之间，它都会等待 gzip 接收数据并完成压缩。至少我是这么认为的，也许我是错的。
如果我是对的，我想知道如何使此打印异步，即它在 gzip 上触发数据，然后继续处理数据。
I'm currently writing a script that takes a database as input and generates all valid combinations from the 10+ tables, following certain rules. Since the output is pretty darn huge, i'm dumping this through gzip into the file, like this:
open( my $OUT, '|-', "gzip > file" );
for ( @data ) {
my $line = calculate($_);
print $OUT $line;
}
Due to the nature of the beast though i end up having to make hundreds of thousands of small writes, one for each line. This means that between each calculation it waits for gzip to receive the data and get done compressing it. At least i think so, i might be wrong.
In case I'm right though, I'm wondering how i can make this print asynchronous, i.e. it fires the data at gzip and then goes on processing the data.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（3）
尝试
IO::Compress::Gzip。它接受要写入的文件句柄。您可以在该文件句柄上设置O_NONBLOCK。Give
IO::Compress::Gzipa try. It accepts a filehandle to write to. You can setO_NONBLOCKon that filehandle.管道已经使用了缓冲区，因此写入程序不必等待读取程序。然而，该缓冲区通常相当小（在 Linux 上通常只有 64KB）并且不容易更改（需要重新编译内核）。如果标准缓冲区不够，最简单的方法是在管道中包含一个缓冲程序：
bfr 只是将 STDIN 读入内存缓冲区，并以下一个程序允许的速度写入 STDOUT。默认值为 5MB 缓冲区，但您可以使用
-b选项更改该设置（例如，bfr -b10m表示 10MB 缓冲区）。Pipes already use a buffer so that the writing program doesn't have to wait for the reading program. However, that buffer is usually fairly small (it's normally only 64KB on Linux) and not easily changed (it requires recompiling the kernel). If the standard buffer is not enough, the easiest thing to do is include a buffering program in the pipeline:
bfr simply reads STDIN into an in-memory buffer, and writes to STDOUT as fast as the next program allows. The default is a 5MB buffer, but you can change that with the
-boption (e.g.bfr -b10mfor a 10MB buffer).当然，我会按照你的意愿用线或叉子来做。
http://hell.jedicoder.net/?p=82
naturally i'll will do it in a thread or with a fork as you wish.
http://hell.jedicoder.net/?p=82