如何“grep”连续的流?

发布于 2024-11-30 17:56:05 字数 255 浏览 1 评论 0原文

是否可以在连续流上使用 grep

我的意思是有点像 tail -f 命令,但在输出上使用 grep 以便仅保留我感兴趣的行。

我已经尝试过 tail -f; | grep 模式 但似乎 grep 只能在 tail 完成后执行,也就是说永远不会执行。

Is that possible to use grep on a continuous stream?

What I mean is sort of a tail -f <file> command, but with grep on the output in order to keep only the lines that interest me.

I've tried tail -f <file> | grep pattern but it seems that grep can only be executed once tail finishes, that is to say never.

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(13

孤独岁月 2024-12-07 17:56:05

使用BSD grep(FreeBSD,Mac OS X等)时打开grep的行缓冲模式

tail -f file | grep --line-buffered my_pattern

不久前看起来--line-buffered并不重要对于 GNU grep(几乎在任何 Linux 上使用),因为它默认刷新(YMMV 对于其他类 Unix 系统,如 SmartOS、AIX 或 QNX)。然而,截至 2020 年 11 月,需要 --line-buffered (至少在 openSUSE 中的 GNU grep 3.5 中是这样,但根据下面的评论,似乎通常需要)。

Turn on grep's line buffering mode when using BSD grep (FreeBSD, Mac OS X etc.)

tail -f file | grep --line-buffered my_pattern

It looks like a while ago --line-buffered didn't matter for GNU grep (used on pretty much any Linux) as it flushed by default (YMMV for other Unix-likes such as SmartOS, AIX or QNX). However, as of November 2020, --line-buffered is needed (at least with GNU grep 3.5 in openSUSE, but it seems generally needed based on comments below).

你如我软肋 2024-12-07 17:56:05

我使用 tail -f; | grep 始终。

它将等到 grep 刷新,而不是完成(我使用的是 Ubuntu)。

I use the tail -f <file> | grep <pattern> all the time.

It will wait till grep flushes, not till it finishes (I'm using Ubuntu).

故人如初 2024-12-07 17:56:05

我认为你的问题是 grep 使用一些输出缓冲。尝试

tail -f file | stdbuf -o0 grep my_pattern

将 grep 的输出缓冲模式设置为无缓冲。

I think that your problem is that grep uses some output buffering. Try

tail -f file | stdbuf -o0 grep my_pattern

it will set output buffering mode of grep to unbuffered.

中二柚 2024-12-07 17:56:05

如果您想在整个文件(而不仅仅是尾部)中查找匹配项,并且希望它坐下来等待任何新匹配项,那么这很有效:

tail -c +0 -f <file> | grep --line-buffered <pattern>

-c +0< /code> 标志表示输出应从文件开头 (+) 开始 0 字节 (-c)。

If you want to find matches in the entire file (not just the tail), and you want it to sit and wait for any new matches, this works nicely:

tail -c +0 -f <file> | grep --line-buffered <pattern>

The -c +0 flag says that the output should start 0 bytes (-c) from the beginning (+) of the file.

终难遇 2024-12-07 17:56:05

在大多数情况下,您可以 tail -f /var/log/some.log |grep foo ,它会正常工作。

如果您需要在正在运行的日志文件上使用多个 grep 并且发现没有输出,则可能需要将 --line-buffered 开关粘贴到中间 grep(s),像这样:

tail -f /var/log/some.log | grep --line-buffered foo | grep bar

In most cases, you can tail -f /var/log/some.log |grep foo and it will work just fine.

If you need to use multiple greps on a running log file and you find that you get no output, you may need to stick the --line-buffered switch into your middle grep(s), like so:

tail -f /var/log/some.log | grep --line-buffered foo | grep bar
冷弦 2024-12-07 17:56:05

你可能会认为这个答案是增强..通常我使用

tail -F <fileName> | grep --line-buffered  <pattern> -A 3 -B 5

-F 在文件旋转的情况下更好(如果文件旋转,-f 将无法正常工作)

-A 和 -B 对于获取模式出现之前和之后的行很有用..这些块将出现在虚线分隔符之间,

但对我来说,我更喜欢执行以下操作

tail -F <file> | less

,如果您想在流式日志中搜索,这非常有用。我的意思是回顾过去并深入观察

you may consider this answer as enhancement .. usually I am using

tail -F <fileName> | grep --line-buffered  <pattern> -A 3 -B 5

-F is better in case of file rotate (-f will not work properly if file rotated)

-A and -B is useful to get lines just before and after the pattern occurrence .. these blocks will appeared between dashed line separators

But For me I prefer doing the following

tail -F <file> | less

this is very useful if you want to search inside streamed logs. I mean go back and forward and look deeply

诗笺 2024-12-07 17:56:05

没有看到有人提供我通常的做法:

less +F <file>
ctrl + c
/<search term>
<enter>
shift + f

我更喜欢这个,因为您可以使用 ctrl + c 随时停止并浏览文件,然后只需按 shift + f 返回实时流式搜索。

Didn't see anyone offer my usual go-to for this:

less +F <file>
ctrl + c
/<search term>
<enter>
shift + f

I prefer this, because you can use ctrl + c to stop and navigate through the file whenever, and then just hit shift + f to return to the live, streaming search.

风筝有风,海豚有海 2024-12-07 17:56:05

sed 将是更好的选择(编辑器)

tail -n0 -f; | sed -n '/search string/p'

然后如果您希望 tail 命令在找到特定字符串后退出:

tail --pid=$(($BASHPID+1)) -n0 -f <文件> | sed -n '/搜索字符串/{p; q}'

显然是一个 bashism:$BASHPID 将是 tail 命令的进程 ID。 sed 命令位于管道中 tail 之后的下一个命令,因此 sed 进程 ID 将为 $BASHPID+1。

sed would be a better choice (stream editor)

tail -n0 -f <file> | sed -n '/search string/p'

and then if you wanted the tail command to exit once you found a particular string:

tail --pid=$(($BASHPID+1)) -n0 -f <file> | sed -n '/search string/{p; q}'

Obviously a bashism: $BASHPID will be the process id of the tail command. The sed command is next after tail in the pipe, so the sed process id will be $BASHPID+1.

爱的故事 2024-12-07 17:56:05

关于这个问题有点晚了,考虑到这种工作是监控工作的重要组成部分,这是我的答案(不是那么短)...

使用

1. 命令 tail

这个命令稍微详细一点比阅读已经发布的答案

  1. 之间的区别遵循选项tail -ftail -F,来自手册页

    <块引用>

     -f, --follow[={名称|描述符}]
              随着文件的增长输出附加数据;
    ...
       -F 与 --follow=name --retry 相同
    ...
       --重试
              如果文件无法访问,请继续尝试打开该文件
    

    这意味着:通过使用 -F 而不是 -ftail 将在删除时重新打开文件(在日志上)旋转,用于示例)。
    这对于观察多天的日志文件很有用。

  2. 能够同时关注多个文件
    我已经用过:

    tail -F /var/www/clients/client*/web*/log/{error,access}.log /var/log/{mail,auth}.log \
               /var/log/apache2/{,ssl_,other_vhosts_}access.log \
               /var/log/pure-ftpd/transfer.log
    

    对于通过数百个文件跟踪事件...(考虑此答案的其余部分以了解如何使其可读...;)

  3. 使用开关-n(Don请勿使用 -c 进行行缓冲!)。
    默认情况下 tail 将显示最后 10 行。可以调整:

    tail -n 0 -F 文件
    

    将跟随文件,但仅打印新行

    tail -n +0 -F 文件
    

    在跟踪他的进度之前将打印整个文件。

2.管道传送时的缓冲问题:

如果您计划过滤输出,请考虑缓冲!请参阅 sed-u 选项、grep--line-bufferedstdbuf 选项code> 命令:(

tail -F /some/files | sed -une '/Regular Expression/p'

比使用 grep 更有效)比在 sed 中不使用 -u 开关更具反应性> 命令。

tail -F /some/files |
    sed -une '/Regular Expression/p' |
    stdbuf -i0 -o0 tee /some/resultfile

3. 最近的日志系统

在最近的系统上,您必须以几乎相同的方式运行 journalctl -xf,而不是 tail -f /var/log/syslog...

journalctl -axf | sed -une '/Regular Expression/p'

但是阅读手册页,该工具是为日志分析而构建的!

4. 将其集成到 脚本中

  1. 两个文件(或更多)的彩色输出

    以下是监视多个文件的脚本示例,第一个文件的输出颜色与其他文件不同:

    <前><代码>#!/bin/bash

    尾-F“$@”|
    sed-une“
    /^==> /{h;};
    //!{
    克;
    s/^\\(.*\\)\\n==>.*${1//\//\\\/}.*<==/\\o33[47m\\1\\ o33[0米/;
    s/^\\(.*\\)\\n==> .* <==/\\o33[47;31m\\1\\o33[0m/;
    p;}”

    它们在我的主机上运行良好,运行:

    sudo ./myColoredTail /var/log/{kern.,sys}log
    
  2. 交互式脚本

    您可能正在查看日志以对事件做出反应?

    这是一个小脚本,当某些 USB 设备出现或消失时播放一些声音,但相同的脚本可以发送邮件或任何其他交互,例如打开咖啡机电源...

    <前><代码>#!/bin/bash

    执行{tailF} < <(tail -F /var/log/kern.log)
    尾部Pid=$!

    同时:;做
    读取-rsn 1 -t .3键盘
    [ "${键盘,}" = "q" ] &&休息
    如果读取 -ru $tailF -t 0 _ ;则
    读取 -ru $tailF 行
    案例 $line in
    *新\ USB\ 设备\ 已找到* ) 播放/some/sound.ogg ;;
    *USB\ 断开连接* ) 播放 /some/othersound.ogg ;;
    埃萨克
    printf "\r%s\e[K" "$line"

    完毕

    回声
    exec {tailF}<&-
    杀死 $tailPid

    您可以按Q退出

Coming some late on this question, considering this kind of work as an important part of monitoring job, here is my (not so short) answer...

Following logs using

1. Command tail

This command is a little more porewfull than read on already published answer

  1. Difference between follow option tail -f and tail -F, from manpage:

       -f, --follow[={name|descriptor}]
              output appended data as the file grows;
    ...
       -F     same as --follow=name --retry
    ...
       --retry
              keep trying to open a file if it is inaccessible
    

    This mean: by using -F instead of -f, tail will re-open file(s) when removed (on log rotation, for sample).
    This is usefull for watching logfile over many days.

  2. Ability of following more than one file simultaneously
    I've already used:

    tail -F /var/www/clients/client*/web*/log/{error,access}.log /var/log/{mail,auth}.log \
               /var/log/apache2/{,ssl_,other_vhosts_}access.log \
               /var/log/pure-ftpd/transfer.log
    

    For following events through hundreds of files... (consider rest of this answer to understand how to make it readable... ;)

  3. Using switches -n (Don't use -c for line buffering!).
    By default tail will show 10 last lines. This can be tunned:

    tail -n 0 -F file
    

    Will follow file, but only new lines will be printed

    tail -n +0 -F file
    

    Will print whole file before following his progression.

2. Buffer issues when piping:

If you plan to filter ouptuts, consider buffering! See -u option for sed, --line-buffered for grep, or stdbuf command:

tail -F /some/files | sed -une '/Regular Expression/p'

Is (a lot more efficient than using grep) a lot more reactive than if you does'nt use -u switch in sed command.

tail -F /some/files |
    sed -une '/Regular Expression/p' |
    stdbuf -i0 -o0 tee /some/resultfile

3. Recent journaling system

On recent system, instead of tail -f /var/log/syslog you have to run journalctl -xf, in near same way...

journalctl -axf | sed -une '/Regular Expression/p'

But read man page, this tool was built for log analyses!

4. Integrating this in a script

  1. Colored output of two files (or more)

    Here is a sample of script watching for many files, coloring ouptut differently for 1st file than others:

    #!/bin/bash
    
    tail -F "$@" |
        sed -une "
            /^==> /{h;};
            //!{
                G;
                s/^\\(.*\\)\\n==>.*${1//\//\\\/}.*<==/\\o33[47m\\1\\o33[0m/;
                s/^\\(.*\\)\\n==> .* <==/\\o33[47;31m\\1\\o33[0m/;
                p;}"
    

    They work fine on my host, running:

    sudo ./myColoredTail /var/log/{kern.,sys}log
    
  2. Interactive script

    You may be watching logs for reacting on events?

    Here is a little script playing some sound when some USB device appear or disappear, but same script could send mail, or any other interaction, like powering on coffe machine...

    #!/bin/bash
    
    exec {tailF}< <(tail -F /var/log/kern.log)
    tailPid=$!
    
    while :;do
        read -rsn 1 -t .3 keyboard
        [ "${keyboard,}" = "q" ] && break
        if read -ru $tailF -t 0 _ ;then
            read -ru $tailF line
            case $line in
                *New\ USB\ device\ found* ) play /some/sound.ogg ;;
                *USB\ disconnect* ) play /some/othersound.ogg ;;
            esac
            printf "\r%s\e[K" "$line"
        fi
    done
    
    echo
    exec {tailF}<&-
    kill $tailPid
    

    You could quit by pressing Q key.

·深蓝 2024-12-07 17:56:05

是的,这实际上效果很好。 Grep 和大多数 Unix 命令在流上一次运行一行。从 tail 出来的每一行都会被分析,如果匹配则继续传递。

Yes, this will actually work just fine. Grep and most Unix commands operate on streams one line at a time. Each line that comes out of tail will be analyzed and passed on if it matches.

站稳脚跟 2024-12-07 17:56:05

这个命令对我有用(Suse):

mail-srv:/var/log # tail -f /var/log/mail.info |grep --line-buffered LOGIN  >> logins_to_mail

收集邮件服务的登录信息

This one command workes for me (Suse):

mail-srv:/var/log # tail -f /var/log/mail.info |grep --line-buffered LOGIN  >> logins_to_mail

collecting logins to mail service

香橙ぽ 2024-12-07 17:56:05

您肯定不会成功

tail -f /var/log/foo.log |grep --line-buffered string2search

当您使用“colortail”作为 tail 的别名时, ,例如。在 bash 中

alias tail='colortail -n 30'

你可以检查
类型别名
如果这输出类似
tail 是colortail -n 30 的别名。
那么你就有了罪魁祸首:)

解决方案:

删除别名并

unalias tail

确保你通过此命令使用“真正的”尾部二进制文件,

type tail

该命令应该输出类似以下内容:

tail is /usr/bin/tail

然后你可以运行你的命令

tail -f foo.log |grep --line-buffered something

祝你好运。

you certainly won't succeed with

tail -f /var/log/foo.log |grep --line-buffered string2search

when you use "colortail" as an alias for tail, eg. in bash

alias tail='colortail -n 30'

you can check by
type alias
if this outputs something like
tail isan alias of colortail -n 30.
then you have your culprit :)

Solution:

remove the alias with

unalias tail

ensure that you're using the 'real' tail binary by this command

type tail

which should output something like:

tail is /usr/bin/tail

and then you can run your command

tail -f foo.log |grep --line-buffered something

Good luck.

黄昏下泛黄的笔记 2024-12-07 17:56:05

如果没有行缓冲选项,请使用 awk(另一个很棒的 bash 实用程序)代替 grep!它将持续从尾部传输您的数据。

这就是你如何使用 grep

tail -f <file> | grep pattern

这就是你如何使用 awk

tail -f <file> | awk '/pattern/{print $0}'

Use awk(another great bash utility) instead of grep where you dont have the line buffered option! It will continuously stream your data from tail.

this is how you use grep

tail -f <file> | grep pattern

This is how you would use awk

tail -f <file> | awk '/pattern/{print $0}'
~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文