如何使用GO同时下载多个S3对象?
我正在尝试从S3下载4个对象。他们每个人几乎都是1 MB。如果我连续下载它们,我将获得每个对象的延迟约为40-50ms,总延迟为200ms。但是,如果我尝试同时下载它们,总延迟将增加到300ms。我不确定为什么并发下载较慢。有人可以帮助我同时下载对象的正确方法吗?谢谢!
这是我使用的GO代码:
downloader := s3manager.NewDownloaderWithClient(client)
out := make([][]byte, len(keys))
wg := sync.WaitGroup{}
for i := 0; i < len(keys); i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
wb := &aws.WriteAtBuffer{}
_, err := downloader.Download(wb, &s3.GetObjectInput{
Bucket: aws.String(s3client.bucketName),
Key: aws.String(keys[i]),
})
out[i] = wb.Bytes()
}(i)
}
wg.Wait()
return out
I am trying to download 4 objects from S3. Each of them is almost of size 1 MB. If I download them serially, I get a latency of around 40-50ms per object with a total latency of appx 200ms. However, If I try to download them concurrently, the total latency increases to 300ms. I am not sure why concurrent downloads are slower. Can someone help me with what is the correct way to download objects concurrently? Thanks!
Here is the GO code that I am using:
downloader := s3manager.NewDownloaderWithClient(client)
out := make([][]byte, len(keys))
wg := sync.WaitGroup{}
for i := 0; i < len(keys); i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
wb := &aws.WriteAtBuffer{}
_, err := downloader.Download(wb, &s3.GetObjectInput{
Bucket: aws.String(s3client.bucketName),
Key: aws.String(keys[i]),
})
out[i] = wb.Bytes()
}(i)
}
wg.Wait()
return out
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论