Cisco 模块 (Filebeat) 到 Logstash - 配置问题 - 无法写入现有索引

Question

我能够使用以下配置成功使用 Filebeat 将日志发送到 Elasticsearch。

# ============================== Filebeat inputs ===============================

filebeat.inputs:
- type: log
  enabled: false
  paths:
    - /var/log/*.log

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["http://esnode1.cluster.com:9200","http://esnode2.cluster.com:9200"]


  # Authentication credentials - either API key or username/password.
  username: "elastic"
  password: "XXXXXXXXXXXXX"

  #Index name customization as we do not want 'Filebeat-" prefix for the indices that filbeat creates by default
  index: "network-%{[event.dataset]}-%{+yyyy.MM.dd}"

#Below configuration setting are mandatory when customizing index name
setup.ilm.enabled: false
setup.template:
  name: 'network'
  pattern: 'network-*'
  enabled: false

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: debug
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

# ============================= X-Pack Monitoring ==============================
#monitoring.elasticsearch:
monitoring:
  enabled: true
  cluster_uuid: 9ZIXSpCDBASwK5K7K1hqQA
  elasticsearch:
    hosts: ["http:/esnode1.cluster.com:9200","http://esnode2.cluster.com:9200"]
    username: beats_system
    password: XXXXXXXXXXXXXX

我启用了所有思科模块，它们能够创建如下索引：

network-cisco.ios-YYYY.MM.DD

network-cisco.nexus-YYYY.MM.DD >

network-cisco.asa-YYYY.MM.DD

network-cisco.ftd-YYYY.MM.DD

直到这里没有问题，但一切都停止了什么时候我尝试在 Filebeat 和 Filebeat 之间引入 Logstash；弹性搜索。

下面是network.conf文件的详细信息供您分析。

input {
 beats {
   port => "5046"
 }
}

output {
 if [event.dataset] == "cisco.ios" {
   elasticsearch {
    hosts => ["http://esnode1.cluster.com:9200","http://esnode2.cluster.com:9200"]
    index => "network-%{[event.dataset]}-%{+yyyy.MM.dd}"
    user => "elastic"
    password => "XXXXXXXXXXXX"
    pipeline => "%{[@metadata][pipeline]}"
    manage_template => "false"
    ilm_enabled => "false"
  }

 }

 else if [event.dataset] == "cisco.nexus" {
   elasticsearch {
    hosts => ["http://esnode1.cluster.com:9200","http://esnode2.cluster.com:9200"]
    index => "network-%{[event.dataset]}-%{+yyyy.MM.dd}"
    user => "elastic"
    password => "XXXXXXXXXXXX"
    pipeline => "%{[@metadata][pipeline]}"
    manage_template => "false"
    ilm_enabled => "false"
   }
 }

 else if [event.dataset] == "cisco.asa" {
   elasticsearch {
    hosts => ["http://esnode1.cluster.com:9200","http://esnode2.cluster.com:9200"]
    index => "network-%{[event.dataset]}-%{+yyyy.MM.dd}"
    user => "elastic"
    password => "XXXXXXXXXXXX"
    pipeline => "%{[@metadata][pipeline]}"
    manage_template => "false"
    ilm_enabled => "false"
   }
 }

 else if [event.dataset] == "cisco.ftd" {
   elasticsearch {
    hosts => ["http://esnode1.cluster.com:9200","http://esnode2.cluster.com:9200"]
    index => "network-%{[event.dataset]}-%{+yyyy.MM.dd}"
    user => "elastic"
    password => "XXXXXXXXXXXX"
    pipeline => "%{[@metadata][pipeline]}"
    manage_template => "false"
    ilm_enabled => "false"
   }
 }

 else if [event.dataset] == "cef.log" {
   elasticsearch {
    hosts => ["http://esnode1.cluster.com:9200","http://esnode2.cluster.com:9200"]
    index => "network-%{[event.dataset]}-%{+yyyy.MM.dd}"
    user => "elastic"
    password => "XXXXXXXXXXXX"
    pipeline => "%{[@metadata][pipeline]}"
    manage_template => "false"
    ilm_enabled => "false"
   }
 }
 else if [event.dataset] == "panw.panos" {
   elasticsearch {
    hosts => ["http://esnode1.cluster.com:9200","http://esnode2.cluster.com:9200"]
    index => "network-%{[event.dataset]}-%{+yyyy.MM.dd}"
    user => "elastic"
    password => "XXXXXXXXXXXX"
    pipeline => "%{[@metadata][pipeline]}"
    manage_template => "false"
    ilm_enabled => "false"
   }
 }
   stdout {codec => rubydebug}
}

通过上述配置，我无法连接 Filbeat --> Logstash -->我希望实现的 Elasticsearch 管道。

当我运行logstash时，没有添加数据，stdout能够生成输出，如下所示：

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/network.conf

使用--config_test_and_exit配置文件已成功测试，上面的行也生成stdout json行，但尽管如此，还是有没有文档添加到现有索引（network-cisco.ios-YYYY.MM.DD、network-cisco.nexus-YYYY.MM.DD 等）。

当我尝试通过使用一个elasticsearch输出进行测试将索引名称更改为'test-%{+yyyy.MM.dd}'时，我发现它创建了一个索引与上面相同的执行。

此外，当我将 Logstash 排除在外时，Filebeat 能够继续写入现有索引，但上述 Logstash 配置不会发生这种情况。

任何帮助将不胜感激！

谢谢， 阿伦