无法参数化输出LogStash索引-AWS S3输入

Question

早上好，

我们正在尝试使用Logstash从AWS S3收集日志。 这是logstash.conf文件：

input {

  tcp {
    port => 5000
    codec => json
  }

  s3 {
    # Keys for user in prod env
    id => "prod" 
    access_key_id => "key1" 
    secret_access_key => "secretkey1"
    region => "eu-central-1"
    bucket => "buket1"
    prefix => "prefix1/"
    include_object_properties => true
  }

  s3 {
    # Keys for user in preprod env
    id => "preprod"
    access_key_id => "key2" 
    secret_access_key => "secretkey2"
    region => "eu-central-1"
    bucket => "bucket2"
    prefix => "prefix2/"
    include_object_properties => true
  }
  
}

filter {

  # Matching prod logs
  if [id] == "prod" {
    mutate {
      add_field => { "env" => "prod" }
    }
  }
  else if [id] == "preprod" {
    mutate {
      add_field => { "env" => "preprod" }
    }
  }

}

output {
  opensearch {
    hosts       => "https://osb-platform"
      user => "username"
      password => "password"
    index       => "4s-%{[env]}-logs-%{+YYYY.MM.dd}"
    ecs_compatibility => disabled
    ssl_certificate_verification => false
  }
}

如您所见，我有两个来自两个不同环境的输入，我想在不同索引上放置来自不同环境的日志。这是通过在ID OD上进行过滤输入来完成的，它可以根据ENV添加一个新字段。

记录已正确收集，但没有区分索引：从OpenSearch中，我可以看到一个名为4S-％{[ENG]} -Logs-2022.05.30的单个索引，就像它无法解析Env Varible一样。

你能帮我吗？

谢谢，

卢卡

Answer 1

任何输入中的id选项用于帮助识别输入时，在监视LogStash时，它不用于过滤，因为文档中不存在。

您需要使用标签或 type 可以做到这一点的选项。

例如，使用type：

  s3 {
    # Keys for user in prod env
    type => "prod" 
    access_key_id => "key1" 
    secret_access_key => "secretkey1"
    region => "eu-central-1"
    bucket => "buket1"
    prefix => "prefix1/"
    include_object_properties => true
  }

然后您的过滤器是：

filter {
  if [type] == "prod" {
    mutate {
      add_field => { "env" => "prod" }
    }
  } else if [type] == "preprod" {
    mutate {
      add_field => { "env" => "preprod" }
    }
  }
}

如果要使用标签，则需要这样的东西：

  s3 {
    # Keys for user in prod env
    tags => ["prod"] 
    access_key_id => "key1" 
    secret_access_key => "secretkey1"
    region => "eu-central-1"
    bucket => "buket1"
    prefix => "prefix1/"
    include_object_properties => true
  }

filter {
  if "prod" in [tags] {
    mutate {
      add_field => { "env" => "prod" }
    }
  } else if "preprod" in [tags] {
    mutate {
      add_field => { "env" => "preprod" }
    }
  }
}