Spark 2集群资源动态分配给正在运行的作业
我们有一个 Spark 2 HDInsight 集群，具有 650 GB 和 195 个 Vcore。这是一个 9 个工作节点和 2 个头节点的集群。问题在于作业没有充分利用集群。例如，当我运行一项作业时，它仅使用 164 GB 内存，而我们有 650 GB 内存。我通过将 spark.executor.memory 从 10 GB 增加到 40 GB 解决了这个问题。 Spark.executor.instances 是 16 。但当我运行多个作业时，问题又出现了。首先发生的作业将使用整个集群直到完成。其他作业将仅处于运行模式，内存仅有 3 GB。要求是，当只有一个作业运行时，集群必须得到充分利用。如果有多个作业，则必须在作业之间共享资源（Ram 和 Vcore）。
We have a spark 2 HDInsight cluster which has 650 GB and 195 Vcores. This is a 9 worker nodes and 2 head nodes cluster. The problem is that the jobs are not fully utilizing the cluster. For example when I run one job, Its using only 164 GB of memory when we have 650 GB. I have solved this problem by increasing the spark.executor.memory to 40 GB from 10 GB. The spark.executor.instances is 16 . But again the problem comes when I run multiple jobs. The job which came first will use the entire cluster till it finishes. The other jobs will just be in running mode with only 3 GB of memory. The requirement is that , the cluster has to be fully utilized when only one job is running. If there are multiple jobs, the resources ( Ram and Vcores ) have to be shared among the jobs.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
我建议您将纱线调度程序更改为容量调度程序。这样更利于资源共享。这将帮助您确保更好地共享资源。默认情况下，hadoop 是“先进先出”的。我非常不同意 100% 使用 HDInsight。您将执行程序增加到 40GB 的修复正是 3GB 的新作业无法进入集群的原因。 （仅仅因为您分配了 40GB 并不意味着您的作业将使用它。）如果您想增加集群使用率，您可以考虑添加线程较少的执行程序以充分利用集群。这与容量调度程序和容量调度程序相结合。先发制人可能是获得更高性能和灵活性的答案。
I suggest that you change your yarn scheduler to Capacity scheduler. This is better at sharing resources. This will help you to ensure the resources are better shared. By default hadoop is 'First in First out'. I respectfully disagree with 100% utilization of HDInsight. Your fix of increase executors to 40GB is exactly why a new job of 3GB can't get into your cluster. (And just because you allocate 40GB does not mean your job will use it.) If you want to increase cluster usage you might consider adding executors with less threads to fully utilize the cluster. This in combination with Capacity Scheduler & Pre-emption might be the answer to getting more performance and flexibility.