SQS到ECS（Fargate）或SQS到Lambda到ECS
这个问题更多是关于如何最好地建立ETL管道的建筑问题。目前，我有一个通过SQ pings pings pings pings lambda。但是，处理数据可能需要超过15分钟（AWS的运行时限制），并使用SAM构建进行部署会导致.zip大于250MB，因此会引发错误。因此，需要替代AWS lambda。到目前为止，我看到的替代方案是：
SQS-＆gt; ECS（Fargate） sqs-＆gt; lambda-＆gt; ECS（fargate）
我没有找到两个选项的支持/缺点的提示，以及通常的首选选项。关于如何解决这个问题的任何提示？
This question is more of an architectural question about how best to build an ETL pipeline. At present, I have an AWS Lambda that is pinged via SQS. But processing data can take a little more than 15 minutes (runtime limit of AWS) and using the sam build for deployment results in a .zip larger than 250MB, therefore it throws an error. Therefore an alternative to AWS Lambda is needed. The alternatives what I'm seeing so far are:
SQS -> ECS (Fargate)
SQS -> Lambda -> ECS (Fargate)
I don't find any hints to the pro/cons of both options, and what is generally a preferred option. Any tips on how to approach this?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
一旦您开始按照15分钟的限制或lambda的部署文件大小限制，那么可能是时候搬到ECS Fargate了。
请注意，当您将SQS配置为Lambda的事件源时，AWS实际上正在运行一个过程，该过程将对消息进行SQS队列进行轮询，然后用这些消息调用您的Lambda功能。 SQS永远不会在任何地方“推”消息，这是“拉”服务。
考虑到这一点，迁移到ECS时，您需要更改代码，以便将SQ进行轮询以进行处理。通常，配置这种情况的方法是拥有一个docker映像，当作为容器中产生时，它只会永远对SQS队列进行轮询，并处理其接收到的任何消息。然后，根据SQS队列中的消息数配置ECS自动缩放。当队列中有0个消息时，ECS可以扩展到0个运行任务。当队列中有消息时，EC可以扩展到您希望运行的许多任务以处理消息。此设置中无需Lambda。
As soon as you start running up against the 15 minute limits, or the deployment file size limits of Lambda, then it's probably time to move to ECS Fargate.
Please note that when you configured SQS as an event source for Lambda, behind the scenes AWS is actually running a process that polls the SQS queue for messages, and then calls your Lambda function with those messages. SQS never "pushes" messages anywhere, it is a "pull" service.
With that in mind, when moving to ECS you would need to change your code so that it polls SQS for messages to process. In general the way to configure this is to have a docker image that, when spawned as a container simply polls the SQS queue forever, processing any messages it receives. Then have ECS auto-scaling configured based on the number of messages in the SQS queue. When there are 0 messages in the queue, ECS can scale down to 0 running tasks. When there are messages in the queue, ECS can scale up to however many number of tasks you wish to run in order to process the messages. There's no need for Lambda in this setup.