我可以获得比 /dev/null 更快的输出管道吗?
我正在运行一项巨大的任务[使用 perl + 数据库等编写的自动翻译脚本],要不间断地运行大约 2 周。在思考如何加快速度时,我看到翻译器始终将所有内容(所有翻译的句子、所有信息)输出到 STDOUT。当我在控制台上获得输出时,这使得它的工作速度明显变慢。
显然,我将输出通过管道传输到 /dev/null
,但后来我想“还有更快的东西吗?”它的输出如此之多,确实会产生影响。
这就是我要问你的问题,因为据我所知,没有什么比这更快的了......(但我距离成为每天使用 Linux 仅过去 3 年的大师还很远)
I am running a huge task [automated translation scripted with perl + database etc.] to run for about 2 weeks non-stop. While thinking how to speed it up I saw that the translator outputs everything (all translated sentences, all info on the way) to STDOUT all the time. This makes it work visibly slower when I get the output on the console.
I obviously piped the output to /dev/null
, but then I thought "could there be something even faster?" It's so much output that it'd really make a difference.
And that's the question I'm asking You, because as far as I know there is nothing faster... (But I'm far from being a guru having used linux on a daily basis only last 3 years)
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(3)
输出到 /dev/null 是在内核中实现的,速度非常快。现在输出管道不是您的问题,而是构建发送到 /dev/null 的字符串所需的时间。我建议您仔细检查该程序并注释掉(或使用
if $be_verbose
保护)所有无用打印语句的行。我很确定这会给你带来明显的加速。Output to /dev/null is implemented in the kernel, which is pretty bloody fast. The output pipe isn't your problem now, it's the time it takes to build the strings that are getting sent to /dev/null. I would recommend you go through the program and comment out (or guard with
if $be_verbose
) all the lines that are useless print statements. I'm pretty sure that'll give you a noticeable speedup.我能够(通过 dd)每秒将 20 GB 的数据转储到 /dev/null 中。这不是你的瓶颈:-p
几乎唯一让它更快的方法就是首先不生成数据——完全删除日志语句。生成所有日志消息的成本可能远远超过丢弃它们的成本。
I'm able (via
dd
) to dump 20 gigabytes of data per second down /dev/null. This is not your bottleneck :-pPretty much the only way to make it faster is to not generate the data in the first place - remove the logging statements entirely. The cost of producing all the log messages likely exceeds the cost of throwing them away quite a bit.
与perl和标准输出无关,但有
null_blk
块设备,它甚至比/dev/null
更快。基本上,它受到系统调用性能的限制,并且对于大块,它可能会使内存总线饱和。Unrelated to perl and standard output, but there is
null_blk
block device, which is even faster than/dev/null
. Basically, it is bounded by syscall performance and with large blocks it can saturate memory bus.