当消费缓慢时扩展我的 Kinesis 消费者
假设我有一个包含 2 个分片的 AWS Kinesis 流。 因此，我有两个消费者从每个分片进行消费。
队列中有大量条目，我的消费者正在缓慢地消耗它。 为了解决这个问题，我可以使用 Kafka 消费者组之类的方法，创建两个消费者应用程序并消费记录。
但我想知道是否可以重新分片我的队列（这将跨分片分配记录）并为该分片添加消费者。 即重新分片后，我的流将有 4 个分片，因此有 4 个消费者。 这样也能增加消耗，解决我的问题。
当队列出现摄取问题时，通常建议使用第二种方法，第二种方法的优缺点是什么？
Assume I have a AWS Kinesis stream with 2 shards.
Therefore I have two consumers consuming from each shard.
There are large number of entries in the queues and my consumer is consuming it slowly.
To solve this I can go with Kafka consumer group like approach ,create two consumer applications and consume records.
But I want to know whether I can Reshard my queue (which will distribute the records across shards) and add consumers for that shards.
i.e After resharding my stream will have 4 shards and hence 4 consumers.
This will also increase the consumption and solve my problem.
Whats the pros and cons of the second approach as the second approach is generally suggested when the queue has ingestion issue?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
添加分片以增加消耗能力正是 kinesis 的目的。它将增加“消费者应用程序”的并行性，并且应该非常无缝。
请注意，AWS 最近引入了无服务器运动模型，您不必再关心分片计数。这几乎相当于让
aws自己关心分片的数量，因此您不必再担心这个问题。Adding shards to increase consuming capacity is exactly what kinesis is about. It will increase the parallelism of your "consumer application" and should be pretty seamless.
Note that aws recently introduced a serverless kinesis model where you don't have to care about the shard count anymore. It's pretty much equivalent to letting
awscare about the number of shards by itself so you don't have to worry about that anymore.