Logstash Grok 导入 Apache 访问日志文件 数据丰富

Question

测试数据

apache_access.log

216.244.66.246 - - [30/Apr/2017:04:28:11 +0000] "GET /docs/triton/pages.html HTTP/1.1" 200 5639 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)"
199.21.99.207 - - [30/Apr/2017:04:29:44 +0000] "GET /docs/triton/class_triton_1_1_wind_fetch-members.html HTTP/1.1" 200 1845 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
199.21.99.207 - - [30/Apr/2017:04:29:57 +0000] "GET /docs/triton/class_triton_1_1_breaking_waves_parameters-members.html HTTP/1.1" 200 1967 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
100.43.90.9 - - [30/Apr/2017:04:30:13 +0000] "GET /docs/html/functions_rela.html HTTP/1.1" 200 882 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
217.182.132.36 - - [30/Apr/2017:04:30:31 +0000] "GET /2012/08/sundog-software-featured-in-august-2012-issue-of-develop/ HTTP/1.1" 200 14326 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)"
54.210.20.202 - - [30/Apr/2017:04:30:58 +0000] "POST /wp-cron.php?doing_wp_cron=1493526658.6017329692840576171875 HTTP/1.1" 200 - "http://sundog-soft.com/wp-cron.php?doing_wp_cron=1493526658.6017329692840576171875" "WordPress/4.7.4; http://sundog-soft.com"

Logstash 配置文件

logstash.conf

input {
  file {
    path => "/Users/jasonwu/WorkSpace/learn/Elasticsearch/apache_access.log"
    start_position => "beginning"
    sincedb_path => "/Users/jasonwu/.Trash/sincedb.trash"
  }
}

filter {
  grok {
    match => {
      "message" => ["%{COMBINEDAPACHELOG}"]
    }
  }

  useragent {
    source => "agent"
    target => "agent"
  }

  geoip {
    source => "clientip"
    target => "geoip"
  }

  mutate {
    add_field => {
      "read_timestamp" => "%{@timestamp}"
    }

    remove_field => ["path", "@version", "message", "@timestamp", "host"]
  }
}

output {
  stdout {}

  elasticsearch {
    hosts => ["localhost:9200"]
    index => "logs"
  }
}

执行 Logstash 导入 Elasticsearch

$ bin/logstash -f /Users/jasonwu/WorkSpace/learn/Elasticsearch/logstash.conf