通过logstash解析日志

Question

我是 ELK 的新手。我正在尝试使用 Logstash 和 grok 解析日志。

Logstash 配置：

input { stdin { } }

filter {
    grok {
      match => { "message" => "%{SPACE}%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{SYSLOG5424SD}%{SPACE}%{LOGLEVEL:Log_level}%{SPACE}%{GREEDYDATA:Parameters}%{SPACE}\-%{SPACE}\[operation=%{WORD:operation},%{SPACE}duration=%{NUMBER:duration}%{SPACE}sec\]%{SPACE}%{GREEDYDATA:Parameters}%{SPACE}%{GREEDYDATA:Parameters}%{SPACE}%{GREEDYDATA:Parameters}%{SPACE}(?:[^:]+)caller:%{SPACE}%{IPV4:caller},%{SPACE}username:%{SPACE}%{WORD:username}(?:[^:]+)%{SPACE}%{GREEDYDATA:Parameters}%{SPACE}%{GREEDYDATA:Parameters}%{SPACE}Headers:%{GREEDYDATA:Headers}%{SPACE}Payload:%{SPACE}{%{SPACE}%{GREEDYDATA:Payload}%{SPACE}}\n\n(?m)%{GREEDYDATA:java_stack_trace}\n\n" }
    }
}

output {
  file {
   path => ["/tmp/test_log/output/output.log"]
   codec => rubydebug
 }
}

日志：

2022-03-28 01:19:58,178 [default task-5117] INFO  LoggingFeature_LONGOPS - [operation=getClientStatus, duration=14.90 sec] REQ_IN
    ID: 304711
    Address: http://192.168.0.1:8080/test/test/services/getClientStatus [caller: 192.168.0.1, username: TEST_TEST]
    HttpMethod: POST
    Content-Type: application/json
    Headers: {Authorization=********, Accept=application/json, text/plain, */*, User-Agent=axios/0.21.4, connection=close, content-type=application/json, Host=192.168.0.1:8080, Conten$
    Payload:
{"data1":"7777777777","data2":1111}

Cpu usage (90 samples by 100 ms, total 9 sec):
java.lang.Thread.run (90) - 100%
 org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run (90) - 100%
  org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask (90) - 100%

在 grokdebug.herokuapp.com 上它给出了包含所有必需数据的正常响应，并且直接在 Logstash 中第一行后出现错误：

{
     "operation" => "getClientStatus",
          "host" => "test",
     "Log_level" => "INFO",
    "@timestamp" => 2022-03-30T08:49:35.228Z,
    "Parameters" => "LoggingFeature_LONGOPS ",
      "duration" => "14.90",
     "timestamp" => "2022-03-28 01:19:58,178",
       "message" => "2022-03-28 01:19:58,178 [default task-5117] INFO  LoggingFeature_LONGOPS - [operation=getClientStatus, duration=14.90 sec] REQ_IN",
      "@version" => "1"
}
{
    "@timestamp" => 2022-03-30T08:49:35.231Z,
          "tags" => [
        [0] "_grokparsefailure"
    ],
          "host" => "test",
       "message" => "    ID: 304711",
      "@version" => "1"
}

请告诉我，我做错了什么？

我尝试

(?m)%{GREEDYDATA:Parameters:}

在输入中使用指定的多行编解码器，

input {
  file {
    path => "/tmp/test_log/input/*.log"
    codec => multiline {
      pattern => "^%{TIMESTAMP_ISO8601} "
      negate => true
      what => previous
    }
  }
}

选项

config.support_escapes: true

也启用了logstash.yml中的

Answer 1

input {
  file {
    path => "/tmp/test_log/input/*.log"
    codec => multiline {
      pattern => "^%{DATE_EU}"
      negate => true
      what => previous
    }
}