fuentd倒车别名AWS OpenSearch ILM

发布于 2025-01-23 04:42:05 字数 1330 浏览 3 评论 0原文

我使用AWS OpenSearch，Fluent-Bit（EKS上的Daemonset），Fluentd（EKS上的部署）和OpenSearch仪表板进行了日志记录基础架构设置。

我正在制定ILM政策，并面临转盘问题。在达到条件后，我正在尝试实现政策（每1天或索引达到2G尺寸）后，应该发生滚动并移至冷藏。一旦旧的索引总共达到7天，旧索引就应该被删除。

我的流利配置：

<match *.**>   
  @type copy
  <store>
    type elasticsearch  
    include_tag_key true
    host "#{ENV.fetch('ELASTICSEARCH_HOST')}"
    port "#{ENV.fetch('ELASTICSEARCH_PORT')}"
    user "#{ENV.fetch('ELASTICSEARCH_USER')}"
    password "#{ENV.fetch('ELASTICSEARCH_PASSWORD')}"
    log_es_400_reason true
    ca_file /certs/ca.pem
    scheme https
    ssl_verify true 
    logstash_format true
    logstash_prefix rollover-sbx
 </match>

ILM策略

{ “ id”：“ sbx-ism-policy”， “ seqno”：379851， “初等”：2， “政策”： { “ policy_id”：“ sbx-ism-policy”， “描述”：“一个简单的默认策略，它改变了在热状态和冷状态之间的副本计数。”， “ last_updated_time”：1650595677042， “ schema_version”：12， “ error_notification”：null， “ default_state”：“热”， “状态”：[[ { “名称”：“热”， “动作”：[ { “滚下”： { “ min_size”：“ 200MB”， “ min_doc_count”：200， “ min_index_age”：“ 1H” } } ]，， “过渡”：[ { “ state_name”：“冷”， “状况”： { “ min_index_age”：“ 1H” } } 这是给出的 }，， { “名称”：“冷”， “动作”：[ { “关闭”： {} } ]，， “过渡”：[] } ]，， “ ism_template”：[ { “ index_patterns”：[ “ Rollover-SBX*” ]，， “优先”：70， “ last_updated_time”：1650583513796 } 这是给出的 } }

上述最小尺寸和ILM政策的时间是用于测试目的

原文

I have logging infrastructure setup with AWS OpenSearch, Fluent-bit (DaemonSet on EKS), FluentD (Deployment on EKS) and OpenSearch Dashboard.

I am working on the ILM policy and facing issue with the rollover. I am trying to achieve with the policy, after reaching the condition (after every 1 days or when the index reaches size 2G), the rollover should happen and moved to cold storage. Once the old index reaches 7 days in total, the old index should get deleted.

My FluentD config:

<match *.**>   
  @type copy
  <store>
    type elasticsearch  
    include_tag_key true
    host "#{ENV.fetch('ELASTICSEARCH_HOST')}"
    port "#{ENV.fetch('ELASTICSEARCH_PORT')}"
    user "#{ENV.fetch('ELASTICSEARCH_USER')}"
    password "#{ENV.fetch('ELASTICSEARCH_PASSWORD')}"
    log_es_400_reason true
    ca_file /certs/ca.pem
    scheme https
    ssl_verify true 
    logstash_format true
    logstash_prefix rollover-sbx
 </match>

ILM policy

{
"id": "sbx-ism-policy",
"seqNo": 379851,
"primaryTerm": 2,
"policy": {
"policy_id": "sbx-ism-policy",
"description": "A simple default policy that changes the replica count between hot and cold states.",
"last_updated_time": 1650595677042,
"schema_version": 12,
"error_notification": null,
"default_state": "hot",
"states": [
{
"name": "hot",
"actions": [
{
"rollover": {
"min_size": "200mb",
"min_doc_count": 200,
"min_index_age": "1h"
}
}
],
"transitions": [
{
"state_name": "cold",
"conditions": {
"min_index_age": "1h"
}
}
]
},
{
"name": "cold",
"actions": [
{
"close": {}
}
],
"transitions": []
}
],
"ism_template": [
{
"index_patterns": [
"rollover-sbx*"
],
"priority": 70,
"last_updated_time": 1650583513796
}
]
}
}