通过Logstash更新Elasticsearch的现有文档，并插入当前记录

发布于 2025-01-27 08:11:15 字数 1060 浏览 3 评论 0原文

我正在尝试将记录插入Elasticsearch，并更新一个现有文档的字段，该文档将从当前的记录中获得_id。在线搜索后，我发现我们可以将_update_by_query API与logstash中的HTTP插件一起使用。这是以下配置。

output {

    elasticsearch {
            hosts => ["localhost:9200"]
            index => "my_index_*"
            document_id => "%{id_field}"
       }

    http {
           url => "http://localhost:9200/my_index_*/_update_by_query"
           http_method => "post"
           content_type => "application/json"
           format => "message"
           message => '{"query":{"match":{"_id":"%{previous_record_id}"}},"script":{"source":"ctx._source.field_to_be_updated=xyz","lang":"painless"}}'

       }
}

Elasticsearch没有密码保护，因此我没有添加授权标题。但是，当我启动LogStash时，当前记录将被插入，但是我总是在HTTP插件中出现以下错误。

2022-05-05T11:31:51,916][ERROR][logstash.outputs.http    ][logstash_txe] [HTTP Output Failure] Encountered non-2xx HTTP code 400 {:response_code=>400, :url=>"http://localhost:9200/my_index_*/_update_by_query", :event=>#<LogStash::Event:0x192606f8>}

原文

I am trying to insert a record into elasticsearch and also update a field of an existing document whose _id I'll be getting from the current record. After searching online, I found that we can use the _update_by_query api with the http plugin in logstash. This is the below configuration.

output {

    elasticsearch {
            hosts => ["localhost:9200"]
            index => "my_index_*"
            document_id => "%{id_field}"
       }

    http {
           url => "http://localhost:9200/my_index_*/_update_by_query"
           http_method => "post"
           content_type => "application/json"
           format => "message"
           message => '{"query":{"match":{"_id":"%{previous_record_id}"}},"script":{"source":"ctx._source.field_to_be_updated=xyz","lang":"painless"}}'

       }
}

The Elasticsearch has no password protection and so I haven't added an authorization header.
But when I start logstash, the current record gets inserted but I always the below error for the http plugin.

2022-05-05T11:31:51,916][ERROR][logstash.outputs.http    ][logstash_txe] [HTTP Output Failure] Encountered non-2xx HTTP code 400 {:response_code=>400, :url=>"http://localhost:9200/my_index_*/_update_by_query", :event=>#<LogStash::Event:0x192606f8>}

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

々眼睛长脚气 2025-02-03 08:11:15

这不是您应该这样做的方式，您可以简单地使用两种用例的Elasticsearch输出。

第一个用于索引新记录的索引和以下一个用于部分更新另一个记录的记录，其ID为PROSES_RECORD_ID。脚本中可以在params.event中访问事件数据：

elasticsearch {
   hosts => ["localhost:9200"]
   index => "my_index_xyz"
   document_id => "%{previous_record_id}"
   action => "update"

   script => "ctx._source.field_to_be_updated = params.event.xyz"
   script_lang => "painless"
   script_type => "inline"
}

It's not how you're supposed to do it, you can simply use the elasticsearch output for both use cases.

The first one for indexing a new record and the following one for partial updating another record whose id is previous_record_id. The event data can be accessed in params.event within the script:

elasticsearch {
   hosts => ["localhost:9200"]
   index => "my_index_xyz"
   document_id => "%{previous_record_id}"
   action => "update"

   script => "ctx._source.field_to_be_updated = params.event.xyz"
   script_lang => "painless"
   script_type => "inline"
}

回复收藏 0 原文

~没有更多了~