YARN 队列

开启队列

  <property>
    <name>yarn.resourcemanager.scheduler.class</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
  </property>

yarn.scheduler.capacity.root.default.capacity：一个百分比的值，表示占用整个集群的百分之多少比例的资源，这个queue-path下所有的capacity之和是100

yarn.scheduler.capacity.root.default.user-limit-factor：每个用户的低保百分比，比如设置为1，则表示无论有多少用户在跑任务，每个用户占用资源最低不会少于1%的资源

yarn.scheduler.capacity.root.default.maximum-capacity：弹性设置，最大时占用多少比例资源

yarn.scheduler.capacity.root.default.state：队列状态，可以是RUNNING或STOPPED

yarn.scheduler.capacity.root.default.acl_submit_applications：哪些用户或用户组可以提交任务

yarn.scheduler.capacity.root.default.acl_administer_queue：哪些用户或用户组可以管理队列

队列属性配置

资源分配比例: default(40%), batch(60%)

队列管理权限: default与batch队列管理员为admin

队列提交任务: detault队列为所有用户均可提交任务，batch队列只有devops, admin用户可以提交任务。

default队列里单个任务最大占用资源比例为60%

下面配置为两个队列

default队列 资源占比22.68% 弹性32.68%

cdp-model 资源占比77.32% 弹性87.32%

yarn.scheduler.capacity.root.default.user-limit-factor参数设置为2

<?xml version="1.0"?>
<configuration>
  <property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>default,cdp-model</value>
  </property>
  <property>         
    <name>yarn.scheduler.capacity.resource-calculator</name>        
    <value>org.apache.hadoop.yarn.util.resource.DominantResourceCalculator</value>          
  </property>
  <property>
    <name>yarn.scheduler.capacity.maximum-applications</name>
    <value>10000</value>
    <description>Maximum number of applications that can be pending and running.</description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.node-locality-delay</name>
    <value>-1</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.queue-mappings</name>
    <value></value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.queue-mappings-override.enable</name>
    <value>false</value>
  </property>
    <!-- default queue -->
  <property>
    <name>yarn.scheduler.capacity.root.default.capacity</name>
    <value>22.68</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.default.user-limit-factor</name>
    <value>2</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
    <value>32.68</value>
  </property>  
  <property>
    <name>yarn.scheduler.capacity.root.default.state</name>
    <value>RUNNING</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
    <value>*</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
    <value>*</value>
  </property>


  <!-- cdp-model queue -->
  <property>
    <name>yarn.scheduler.capacity.root.cdp-model.capacity</name>
    <value>77.32</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.cdp-model.user-limit-factor</name>
    <value>2</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.cdp-model.maximum-capacity</name>
    <value>87.32</value>
  </property>    
  <property>
    <name>yarn.scheduler.capacity.root.cdp-model.state</name>
    <value>RUNNING</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.cdp-model.acl_submit_applications</name>
    <value>*</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.cdp-model.acl_administer_queue</name>
    <value>*</value>
  </property>
</configuration>

刷新队列配置

yarn rmadmin -refreshQueues

原文链接：https://blog.csdn.net/ganzheyu/article/details/80356554