如何将文件从 S3 复制到 Amazon EMR HDFS？
我正在 EMR 上运行 Hive， 并且需要将一些文件复制到所有EMR实例中。
据我了解，一种方法是将文件复制到每个节点上的本地文件系统，另一种方法是将文件复制到 HDFS，但是我还没有找到一种直接从 S3 复制到 HDFS 的简单方法。
解决这个问题的最佳方法是什么？
I'm running hive over EMR,
and need to copy some files to all EMR instances.
One way as I understand is just to copy files to the local file system on each node the other is to copy the files to the HDFS however I haven't found a simple way to copy stright from S3 to HDFS.
What is the best way to go about this?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（3）
最好的方法是使用 Hadoop 的 distcp 命令。示例（在集群节点之一上）：
% ${HADOOP_HOME}/bin/hadoop distcp s3n://mybucket/myfile /root/myfile这将从名为 myfile 的 S3 存储桶复制一个名为 myfile 的文件mybucket 到 HDFS 中的
/root/myfile。请注意，此示例假设您在“本机”模式下使用 S3 文件系统；这意味着 Hadoop 将 S3 中的每个对象视为一个文件。如果您在块模式下使用 S3，则可以将上例中的 s3n 替换为 s3。有关本机 S3 和块模式之间的差异的更多信息以及对上面示例的详细说明，请参阅 http ://wiki.apache.org/hadoop/AmazonS3。我发现distcp是一个非常强大的工具。除了能够使用它在 S3 中复制大量文件之外，您还可以对大型数据集执行快速的集群到集群复制。 distcp 不是通过单个节点推送所有数据，而是并行使用多个节点来执行传输。与将所有内容复制到本地文件系统作为中介的替代方案相比，这使得 distcp 在传输大量数据时速度显着加快。
the best way to do this is to use Hadoop's distcp command. Example (on one of the cluster nodes):
% ${HADOOP_HOME}/bin/hadoop distcp s3n://mybucket/myfile /root/myfileThis would copy a file called myfile from an S3 bucket named mybucket to
/root/myfilein HDFS. Note that this example assumes you are using the S3 file system in "native" mode; this means that Hadoop sees each object in S3 as a file. If you use S3 in block mode instead, you would replace s3n with s3 in the example above. For more info about the differences between native S3 and block mode, as well as an elaboration on the example above, see http://wiki.apache.org/hadoop/AmazonS3.I found that distcp is a very powerful tool. In addition to being able to use it to copy a large amount of files in and out of S3, you can also perform fast cluster-to-cluster copies with large data sets. Instead of pushing all the data through a single node, distcp uses multiple nodes in parallel to perform the transfer. This makes distcp considerably faster when transferring large amounts of data, compared to the alternative of copying everything to the local file system as an intermediary.
现在 Amazon 本身有一个通过 distcp 实现的包装器，即： s3distcp。
示例将日志文件从 Amazon S3 复制到 HDFS
以下示例说明如何将存储在 Amazon S3 存储桶中的日志文件复制到 HDFS。在此示例中，--srcPattern 选项用于限制复制到守护程序日志的数据。
Now Amazon itself has a wrapper implemented over distcp, namely : s3distcp .
Example Copy log files from Amazon S3 to HDFS
This following example illustrates how to copy log files stored in an Amazon S3 bucket into HDFS. In this example the --srcPattern option is used to limit the data copied to the daemon logs.
请注意，根据亚马逊的说法，在 http://docs.amazonwebservices.com/ElasticMapReduce/ latest/DeveloperGuide/FileSystemConfig.html“Amazon Elastic MapReduce - 文件系统配置”，S3 块文件系统已弃用，其 URI 前缀现在为s3bfs:// 并且他们特别不鼓励使用它，因为“它可能会触发竞争条件，从而可能导致您的作业流程失败”。
根据同一页面，HDFS 现在是 S3 下的“一流”文件系统，尽管它是短暂的（当 Hadoop 作业结束时就会消失）。
Note that according to Amazon, at http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/FileSystemConfig.html "Amazon Elastic MapReduce - File System Configuration", the S3 Block FileSystem is deprecated and its URI prefix is now s3bfs:// and they specifically discourage using it since "it can trigger a race condition that might cause your job flow to fail".
According to the same page, HDFS is now 'first-class' file system under S3 although it is ephemeral (goes away when the Hadoop jobs ends).