AWS Firehose 交付错误:DynamicPartitioning.MetadataExtractionFailed

发布于 2025-01-11 04:39:30 字数 1350 浏览 0 评论 0原文

我使用 boto3 API 将一些数据发送到预先设计的动态分区的目标 S3 存储桶。这是我设置的: 输入图片这里的描述

这是我发送数据的方法:

import base64
import boto3
import json
def put_batch_data_stream(input, data_stream_name):
    '''
    params:
    * input: dict, must include event_id/pbr_id/pitch_id in the key set
    '''
    client = boto3.client('firehose', region_name = 'somewhere', aws_access_key_id='some-key',
        aws_secret_access_key='some-key')
    
    client.put_record_batch(
        DeliveryStreamName=data_stream_name,
        # the data blob has to be base64-encoded
        Records=[{'Data': base64.b64encode(json.dumps(input).encode('ascii'))}]
    )

    event_id = input['event_id']
    pbr_id = input['pbr_id']
    pitch_id = input['pitch_id']
    
    print(f'Successfully sent pitch {pitch_id} of player {pbr_id} in Event {event_id} to stream {data_stream_name}!')
    return
record = {}
record['event_id'] = 12345
record['pbr_id'] = 54321
record['pitch_id'] = 49876
record['information'] = {'speed': 40, 'height': 76, 'weight': 170, 'age': 34, 'gender': 'male'}

put_batch_data_stream(record, 'my-firehose-data-stream')

这两部分工作正常,但我得到的是错误桶中的错误。它的消息是未提供JSON记录

有什么我想念的吗?

I sent some data using boto3 API to my destination S3 bucket which is predesigned dynamically partitioned. Here is what I set:
enter image description here

And here is what I do to send the data:

import base64
import boto3
import json
def put_batch_data_stream(input, data_stream_name):
    '''
    params:
    * input: dict, must include event_id/pbr_id/pitch_id in the key set
    '''
    client = boto3.client('firehose', region_name = 'somewhere', aws_access_key_id='some-key',
        aws_secret_access_key='some-key')
    
    client.put_record_batch(
        DeliveryStreamName=data_stream_name,
        # the data blob has to be base64-encoded
        Records=[{'Data': base64.b64encode(json.dumps(input).encode('ascii'))}]
    )

    event_id = input['event_id']
    pbr_id = input['pbr_id']
    pitch_id = input['pitch_id']
    
    print(f'Successfully sent pitch {pitch_id} of player {pbr_id} in Event {event_id} to stream {data_stream_name}!')
    return
record = {}
record['event_id'] = 12345
record['pbr_id'] = 54321
record['pitch_id'] = 49876
record['information'] = {'speed': 40, 'height': 76, 'weight': 170, 'age': 34, 'gender': 'male'}

put_batch_data_stream(record, 'my-firehose-data-stream')

These two parts work well but what I got is the error in my error bucket. Its message is Non JSON record provided.

Is there anything I miss?

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。
列表为空,暂无数据
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文