S3 中每个目录的最大文件数
如果我有一百万张图像，将它们存储在某个文件夹/子文件夹层次结构中还是直接将它们全部转储到存储桶中（没有任何文件夹）会更好吗？
将所有图像转储到无层次结构存储桶中会减慢 LIST 操作吗？
动态创建文件夹和子文件夹并设置其 ACL（从编程角度来说）是否会产生大量开销？
If I had a million images, would it be better to store them in some folder/sub-folder hierarchy or just dump them all straight into a bucket (without any folders)?
Would dumping all the images into a hierarchy-less bucket slow down LIST operations?
Is there a significant overhead in creating folders and sub folders on the fly and setting up their ACLs (programatically speaking)?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（3）
S3 不尊重分层命名空间。 每个存储桶仅包含许多从键到对象的映射（以及关联的元数据、ACL 等）。
即使对象的键可能包含“/”，S3 也会将路径视为纯字符串并将所有对象放入平面命名空间中。
根据我的经验，随着对象数量的增加，LIST 操作确实需要（线性）更长的时间，但这可能是 Amazon 服务器上以及到客户端的 I/O 所需增加的症状。
然而，查找时间似乎并没有随着对象数量的增加而增加 - 它很可能是某种 O(1) 哈希表实现 - 因此在同一个存储桶中拥有许多对象应该与正常使用的小存储桶一样具有性能（即不是列表）。
对于 ACL，可以在存储桶和每个单独的对象上设置授权。 由于没有层次结构，它们是您唯一的两个选择。 显然，如果您有数百万个文件，设置尽可能多的存储桶范围的授权将大大减少您的管理麻烦，但请记住，您只能授予权限，而不能撤销它们，因此存储桶范围的授权应该是ACL 的所有内容的最大子集。
我建议分成单独的存储桶，用于：
S3 doesn't respect hierarchical namespaces. Each bucket simply contains a number of mappings from key to object (along with associated metadata, ACLs and so on).
Even though your object's key might contain a '/', S3 treats the path as a plain string and puts all objects in a flat namespace.
In my experience, LIST operations do take (linearly) longer as object count increases, but this is probably a symptom of the increased I/O required on the Amazon servers, and down the wire to your client.
However, lookup times do not seem to increase with object count - it's most probably some sort of O(1) hashtable implementation on their end - so having many objects in the same bucket should be just as performant as small buckets for normal usage (i.e. not LISTs).
As for the ACL, grants can be set on the bucket and on each individual object. As there is no hierarchy, they're your only two options. Obviously, setting as many bucket-wide grants will massively reduce your admin headaches if you have millions of files, but remember you can only grant permissions, not revoke them, so the bucket-wide grants should be the maximal subset of the ACL for all its contents.
I'd recommend splitting into separate buckets for:
原始问题“S3 中每个目录的最大文件数”的答案是：无限制。 另请参阅S3 对存储桶中对象的限制。
Answer to the original question "Max files per directory in S3" is: UNLIMITED. See also S3 limit to objects in a bucket.
我使用一种目录结构，其中有一个根目录，然后至少有一个子目录。 我经常使用“文档导入日期”作为根下的目录。 这可以使管理备份变得更加容易。 无论您使用什么文件系统，最终都必然会达到文件计数限制（即使不是物理限制，也是实用的限制）。 您可能也会考虑支持多个根。
I use a directory structure with a root then at least one sub directory. I often use "document import date" as the directory under the root. This can make managing backups a little easier. Whatever file system you are using you're bound to hit a file count limit (a practical if not a physycal limit) eventually. You might think about supporting multiple roots as well.