AWS Firehose 交付错误:DynamicPartitioning.MetadataExtractionFailed
我使用 boto3 API 将一些数据发送到预先设计的动态分区的目标 S3 存储桶。这是我设置的:
这是我发送数据的方法:
import base64
import boto3
import json
def put_batch_data_stream(input, data_stream_name):
'''
params:
* input: dict, must include event_id/pbr_id/pitch_id in the key set
'''
client = boto3.client('firehose', region_name = 'somewhere', aws_access_key_id='some-key',
aws_secret_access_key='some-key')
client.put_record_batch(
DeliveryStreamName=data_stream_name,
# the data blob has to be base64-encoded
Records=[{'Data': base64.b64encode(json.dumps(input).encode('ascii'))}]
)
event_id = input['event_id']
pbr_id = input['pbr_id']
pitch_id = input['pitch_id']
print(f'Successfully sent pitch {pitch_id} of player {pbr_id} in Event {event_id} to stream {data_stream_name}!')
return
record = {}
record['event_id'] = 12345
record['pbr_id'] = 54321
record['pitch_id'] = 49876
record['information'] = {'speed': 40, 'height': 76, 'weight': 170, 'age': 34, 'gender': 'male'}
put_batch_data_stream(record, 'my-firehose-data-stream')
这两部分工作正常,但我得到的是错误桶中的错误。它的消息是未提供JSON记录
。
有什么我想念的吗?
I sent some data using boto3 API to my destination S3 bucket which is predesigned dynamically partitioned. Here is what I set:
And here is what I do to send the data:
import base64
import boto3
import json
def put_batch_data_stream(input, data_stream_name):
'''
params:
* input: dict, must include event_id/pbr_id/pitch_id in the key set
'''
client = boto3.client('firehose', region_name = 'somewhere', aws_access_key_id='some-key',
aws_secret_access_key='some-key')
client.put_record_batch(
DeliveryStreamName=data_stream_name,
# the data blob has to be base64-encoded
Records=[{'Data': base64.b64encode(json.dumps(input).encode('ascii'))}]
)
event_id = input['event_id']
pbr_id = input['pbr_id']
pitch_id = input['pitch_id']
print(f'Successfully sent pitch {pitch_id} of player {pbr_id} in Event {event_id} to stream {data_stream_name}!')
return
record = {}
record['event_id'] = 12345
record['pbr_id'] = 54321
record['pitch_id'] = 49876
record['information'] = {'speed': 40, 'height': 76, 'weight': 170, 'age': 34, 'gender': 'male'}
put_batch_data_stream(record, 'my-firehose-data-stream')
These two parts work well but what I got is the error in my error bucket. Its message is Non JSON record provided
.
Is there anything I miss?
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论