Linux 非阻塞 fifo(按需记录)
我喜欢“按需”记录程序输出。例如。输出被记录到终端,但另一个进程可以随时挂接当前输出。
经典的方法是:
myprogram 2>&1 | tee /tmp/mylog
按需
tail /tmp/mylog
但是,即使在驱动器空间不足之前不使用,这也会创建不断增长的日志文件。所以我的尝试是:
mkfifo /tmp/mylog
myprogram 2>&1 | tee /tmp/mylog
并且按需
cat /tmp/mylog
现在我可以随时读取/tmp/mylog。但是,任何输出都会阻止程序,直到读取 /tmp/mylog。我喜欢 fifo 来刷新任何未读回的传入数据。怎么做呢?
I like to log a programs output 'on demand'. Eg. the output is logged to the terminal, but another process can hook on the current output at any time.
The classic way would be:
myprogram 2>&1 | tee /tmp/mylog
and on demand
tail /tmp/mylog
However, this would create a ever growing log file even if not used until the drive runs out of space. So my attempt was:
mkfifo /tmp/mylog
myprogram 2>&1 | tee /tmp/mylog
and on demand
cat /tmp/mylog
Now I can read /tmp/mylog at any time. However, any output blocks the program until the /tmp/mylog is read. I like the fifo to flush any incoming data not read back. How to do that?
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(10)
受您问题的启发,我编写了一个简单的程序,可以让您执行此操作:
$ myprogram 2>&1 | ftee /tmp/mylog
它的行为与
tee
类似,但将标准输入克隆到标准输出和命名管道(目前的要求),而不会阻塞。这意味着如果您想以这种方式记录,可能会丢失日志数据,但我想这在您的场景中是可以接受的。技巧是阻止
SIGPIPE
信号并忽略写入损坏的 fifo 时的错误。当然,此示例可以通过多种方式进行优化,但到目前为止,它确实做到了我猜是工作。您可以使用以下标准命令对其进行编译:
$ gcc ftee.c -o ftee
您可以通过运行以下命令快速验证它:
$ ping www.google.com | ftee /tmp/mylog
$ cat /tmp/mylog
另请注意 - 这不是多路复用器。一次只能让一个进程执行
$ cat /tmp/mylog
。Inspired by your question I've written a simple program that will let you do this:
$ myprogram 2>&1 | ftee /tmp/mylog
It behaves similarly to
tee
but clones the stdin to stdout and to a named pipe (a requirement for now) without blocking. This means that if you want to log this way it may happen that you're gonna lose your log data, but I guess it's acceptable in your scenario.The trick is to block
SIGPIPE
signal and to ignore error on writing to a broken fifo. This sample may be optimized in various ways of course, but so far, it does the job I guess.You can compile it with this standard command:
$ gcc ftee.c -o ftee
You can quickly verify it by running e.g.:
$ ping www.google.com | ftee /tmp/mylog
$ cat /tmp/mylog
Also note - this is no multiplexer. You can only have one process doing
$ cat /tmp/mylog
at a time.这是一个(非常)古老的线程,但我最近遇到了类似的问题。事实上,我需要的是将标准输入克隆到标准输出,并复制到非阻塞的管道。第一个答案中提出的 ftee 确实有帮助,但(对于我的用例)太不稳定了。这意味着我丢失了如果我及时处理的话本可以处理的数据。
我面临的场景是,我有一个进程(some_process),它聚合一些数据并每三秒将其结果写入标准输出。 (简化的)设置看起来像这样(在实际设置中我使用命名管道):
现在, raw_data.gz 必须被压缩并且必须是完整的。 ftee 很好地完成了这项工作。但是我在中间使用的管道太慢,无法获取冲出的数据 - 但它足够快,可以处理所有内容(如果它可以到达它),这是用普通三通进行测试的。但是,如果未命名管道发生任何情况,普通的 T 形结构会阻塞,并且由于我希望能够按需挂接,所以 T 形结构不是一个选项。回到主题:当我在中间放置一个缓冲区时,情况会变得更好,结果是:
但是仍然丢失了我可以处理的数据。因此,我继续将之前提出的 ftee 扩展为缓冲版本 (bftee)。它仍然具有所有相同的属性,但使用(低效?)内部缓冲区以防写入失败。如果缓冲区已满,它仍然会丢失数据,但对于我的情况来说它工作得很好。一如既往,还有很大的改进空间,但当我从这里复制代码时,我想将其分享给可能使用它的人。
此版本还需要一个(可选)参数,该参数指定要为管道缓冲的块的数量。我的示例调用现在如下所示:
导致在丢弃发生之前缓冲 16384 个块。这会多使用大约 32 MB 的内存,但是...谁在乎呢?
当然,在真实环境中,我使用命名管道,以便我可以根据需要附加和分离。看起来像这样:
此外,该进程对信号的反应如下:
SIGUSR1->将计数器打印到 STDERR
SIGTERM、SIGINT ->第一个退出主循环并将缓冲区刷新到管道,第二个立即终止程序。
也许这对将来的人有帮助......
享受
This is a (very) old thread, but I've run into a similar problem of late. In fact, what I needed is a cloning of stdin to stdout with a copy to a pipe that is non blocking. the proposed ftee in the first answer really helped there, but was (for my use case) too volatile. Meaning I lost data I could have processed if I had gotten to it in time.
The scenario I was faced with is that I have a process (some_process) that aggregates some data and writes its results every three seconds to stdout. The (simplified) setup looked like this (in the real setup I am using a named pipe):
Now, raw_data.gz has to be compressed and has to be complete. ftee does this job very well. But the pipe I am using in the middle was too slow to grab the data flushed out - but it was fast enough to process everything if it could get to it, which was tested with a normal tee. However, a normal tee blocks if anything happens to the unnamed pipe, and as I want to be able to hook in on demand, tee is not an option. Back to the topic: It got better when I put a buffer in between, resulting in:
But that was still losing data I could have processed. So I went ahead and extended the ftee proposed before to a buffered version (bftee). It still has all the same properties, but uses an (inefficient ?) internal buffer in case a write fails. It still loses data if the buffer runs full, but it works beautifully for my case. As always there is a lot of room for improvement, but as I copied the code off of here I'd like to share it back to people that might have a use for it.
This version takes one more (optional) argument which specifies the number of the blocks that are to buffered for the pipe. My sample call now looks like this:
resulting in 16384 blocks to be buffered before discards happen. this uses about 32 Mbyte more memory, but... who cares ?
Of course, in the real environment I am using a named pipe so that I can attach and detach as needed. There is looks like this:
Also, the process reacts on signals as follows:
SIGUSR1 -> print counters to STDERR
SIGTERM, SIGINT -> first exits the main loop and flushed the buffer to the pipe, the second terminated the program immediatly.
Maybe this helps someone in the future...
Enjoy
为什么不定期轮换日志呢?甚至还有一个程序可以为您执行此操作
logrotate
。还有一个用于生成日志消息并根据类型对其执行不同操作的系统。它称为
syslog
。您甚至可以将两者结合起来。让您的程序生成 syslog 消息,配置 syslog 将它们放入文件中,并使用 logrotate 确保它们不会填满磁盘。
如果事实证明您正在为小型嵌入式系统编写并且程序的输出量很大,那么您可能会考虑多种技术。
Why not periodically rotate the logs? There's even a program to do it for you
logrotate
.There's also a system for generating log messages and doing different things with them according to type. It's called
syslog
.You could even combine the two. Have your program generate syslog messages, configure syslog to place them in a file and use logrotate to ensure they don't fill the disk.
If it turned out that you were writing for a small embedded system and the program's output is heavy there are a variety of techniques you might consider.
看起来像 bash
<>
重定向运算符(3.6.10 打开文件描述符进行读取和写入参见) 使得写入用它打开的文件/fifo 是非阻塞的。这应该可行:
#bash IRC 频道上 gniourf_gniourf 给出的解决方案。
It seems like bash
<>
redirection operator (3.6.10 Opening File Descriptors for Reading and WritingSee) makes writing to file/fifo opened with it non-blocking.This should work:
Solution given by gniourf_gniourf on #bash IRC channel.
嵌入式设备上经常使用的 BusyBox 可以创建一个内存缓冲日志
填充
和读取该日志,但只提供一个全局日志。
,可以很好地
BusyBox often used on embedded devices can create a ram buffered log by
which can be filled by
and read by
Works quite well, but only provides one global log.
日志记录可以定向到 UDP 套接字。由于UDP是无连接的,因此不会阻塞发送程序。当然,如果接收器或网络无法跟上,日志将会丢失。
然后,当您想要观察日志记录时:
还有一些其他很酷的好处,例如能够同时附加多个侦听器或广播到多个设备。
The logging could be directed to a UDP socket. Since UDP is connection-less, it won't block the sending program. Of course logs will be lost if the receiver or network can't keep up.
Then when you want to observe the logging:
There are some other cool benefits like being able to attach multiple listeners at the same time or broadcast to multiple devices.
如果您可以在嵌入式设备上安装屏幕,那么您可以在其中运行“myprogram”并将其分离,然后在您想查看日志时随时重新连接。例如:
每当您想查看输出时,重新附加它:
这样您就不必担心程序输出使用磁盘空间。
If you can install screen on the embedded device then you can run 'myprogram' in it and detach it, and reattach it anytime you want to see the log. Something like:
Whenever you want to see the output, reattach it:
This way you won't have to worry about the program output using disk space at all.
给定的
fifo
方法的问题是,当管道缓冲区被填满并且没有读取过程发生时,整个事情将会挂起。对于
fifo
方法的工作,我认为您必须实现一个类似于 BASH:从两个输入流读取的最佳架构(请参阅下面稍微修改的代码,示例代码 2)。对于解决方法,您还可以使用
while ... read
构造,而不是通过在while .. .read
循环将定期覆盖日志文件指定的行数。这将防止日志文件不断增长(示例代码 1)。The problem with the given
fifo
approach is that the whole thing will hang when the pipe buffer is getting filled up and no reading process is taking place.For the
fifo
approach to work I think you would have to implement a named pipe client-server model similar to the one mentioned in BASH: Best architecture for reading from two input streams (see slightly modified code below, sample code 2).For a workaround you could also use a
while ... read
construct instead oftee
ing stdout to a named pipe by implementing a counting mechanism inside thewhile ... read
loop that will overwrite the log file periodically by a specified number of lines. This would prevent an ever growing log file (sample code 1).如果您的进程写入任何日志文件,然后擦除该文件并时不时地重新启动,那么它不会变得太大,或者使用
logrotate
。这就是您所需要的。您将获得与终端一样多的向后滚动。
不需要任何非标准的东西。我没有尝试过使用小日志文件,但我们所有的日志都像这样旋转,而且我从未注意到丢失的行。
If your process writes to any log file and then wipes the file and starts again every now and again, so it doesn't get too big, or uses
logrotate
.Is all you need. You will get as much scroll-back as your terminal.
Nothing non standard is needed. I've not tried it with small log files but all our logs rotate like this and I have never noticed loosing lines.
为了追随 Fabraxias 的脚步,我将分享我对 racic 代码的小修改。在我的一个用例中,我需要抑制对
STDOUT
的写入,因此我添加了另一个参数:swallow_stdout
。如果不是0
,则将关闭到STDOUT
的输出。由于我不是
C
编码员,因此我在阅读代码时添加了注释,也许它们对其他人有用。To follow in Fabraxias foot steps I'm going to share my small modification of racic's code. In one of my use cases I needed to suppress the writes to
STDOUT
, so I've added another parameter:swallow_stdout
. If that is not0
, then output toSTDOUT
will be turned off.Since I'm no
C
coder I've added comments while reading the code, maybe they are useful for others.