将Hadoop纱与Mesos集成
我创建了一个HDFS群集。我必须配置纱线，以便允许YARN应用程序Master能够按需在Mesos群集上创建用于工作处理的容器。
如何将HDFS群集与Mesos Infra集成，以便可以在Mesos上创建容器？
除了纱线群集（客户端节点，边缘节点或通过Mesos Infra旋转的资源），我需要找出一种运行应用程序主体创建的容器的方法。基本上，我必须创建一个按需，仅计算群集，该群集一旦用用纱线就可以运行纱线应用
I have created a hdfs cluster . I have to configure yarn so as to allow yarn application master to be able to create containers for job processing on the mesos cluster on demand .
how can i integrate the hdfs cluster with the mesos infra so that it can create containers on mesos ?
i need to figure out a way to run the containers created by the application master on another resources apart from the YARN cluster ( a client node, or edge node, or the resources spun through mesos infra ) . basically , i have to create an on-demand ,compute only cluster which can run the yarn apps once yarn is used up
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
Mesos是作为纱线更通用的版本而创建的，并不是真正打算一起使用的（纱线应用程序不能部署到Mesos）。 Spark应用程序是整个Hadoop生态系统中可以（独立）两者都可以部署的唯一过程。
值得指出的是，Mesos被移至Apache Attic（编辑：似乎很快就搬出了，但此后没有发行版）。换句话说，它被视为弃用。有了一点配置，纱线可以运行普通的Docker容器，如果您使用的是使用Mesos的话。 Apache Twill是用于在纱线顶部创建分布式应用程序的库，但这也是Apache Attic（并留在）中。
您也不需要特殊的配置即可与Mesos应用程序中的HDF进行通信，仅
hadoop-client依赖项和配置的core> core> core-site.xml和hdfs--- site.xml文件Mesos was created as a more generic version of YARN, they're not really intended to be used together (YARN apps cannot be deployed to Mesos). Spark apps are about the only process in the whole Hadoop ecosystem that can be deployed (independently) to both.
Worth pointing out that Mesos was moved to Apache Attic (edit: and quickly moved out, it seems, but there's been no releases since then). In other words, it's seen as deprecated. With a bit of configuration, YARN can run plain Docker containers, if that's what you're using Mesos for. Apache Twill was a library for creating distributed applications on top of YARN, but that's also in the Apache Attic (and stayed).
You also don't need special configurations to communicate with HDFS from Mesos applications, only the
hadoop-clientdependency and a configuredcore-site.xmlandhdfs-site.xmlfile