HadoopFS (HDFS) 作为分布式文件存储
我正在考虑使用 HDFS 作为我们的客户端视频托管服务的水平扩展文件存储系统。我主要担心 HDFS 并不是为了这种需求而开发的，这更像是“目前在需要处理大量数据的情况下使用的开源系统”。 我们不想处理数据，只是存储它们，在 HDFS 的基础上创建类似小型内部 Amazon S3 模拟的东西。
可能重要的一点是，存储的文件大小将从 100Mb 增加到 10Gb。
有人用 HDFS 来达到这样的目的吗？
I'm consider to use HDFS as horizontal scaling file storage system for our client video hosting service. My main concern that HDFS wasn't developed for this needs this is more "an open source system currently being used in situations where massive amounts of data need to be processed".
We don't want to process data just store them, create on a base of HDFS something like small internal Amazon S3 analog.
Probably important moment is that stored file size will be quite git from 100Mb to 10Gb.
Did anyone use HDFS in such purposes?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（3）
如果您使用的是 S3 等效项，那么它应该已经提供了一个分布式、可安装的文件系统，不是吗？也许您可以访问 http://openstack.org/projects/storage/ 查看 OpenStack。
If you are using an S3 equivalient then it should already provide a distributed, mountable file-system no? Perhaps you can check out OpenStack at http://openstack.org/projects/storage/.
主要缺点是缺乏 POSIX 语义。您无法安装该驱动器，并且需要特殊的 API 来读取和写入该驱动器。 Java API是主要的。有一个名为 libhdfs 的项目，它通过 JNI 生成 C API ，但我从来没有用过它。 Thriftfs 是另一种选择。
我也不确定与其他替代方案相比的读取性能。也许其他人知道。您是否检查过其他分布式文件系统，例如Lustre？
The main disadvantage would be the lack of POSIX semantics. You can't mount the drive, and you need special APIs to read and write from it. The Java API is the main one. There is a project called libhdfs that makes a C API over JNI, but I've never used it. Thriftfs is another option.
I'm also not sure about the read performance compared to other alternatives. Maybe someone else knows. Have you checked out other distributed filesystems like Lustre?
为此，您可能需要考虑 MongoDB。他们有 GridFS，允许您将其用作存储。然后，您可以通过分片水平扩展存储并通过复制提供容错能力。
You may want to consider MongoDB for this. They have GridFS which will allow you to use it as a storage. You can then horizontally scale your storage through shards and provide fault tolerance with replication.