Alauda AI

Configuring quota management for distributed workloads

As an administrator, you can configure quota management for distributed workloads by creating the required Kueue resources: ResourceFlavor, ClusterQueue, and LocalQueue. These resources control how distributed workloads (such as Ray jobs and PyTorch training jobs) consume cluster resources.

TOC

PrerequisitesProcedure1. Create a ResourceFlavor2. Create a ClusterQueue3. Create a LocalQueue4. Verify the configurationUsing HAMi virtual GPU resourcesNext steps

Prerequisites

  • You have cluster administrator permissions.
  • The Alauda Build of Kueue cluster plugin is installed.
  • The KubeRay operator is installed (for Ray-based distributed workloads).
  • The Alauda Container Platform Web CLI has communication with your cluster.

Procedure

1. Create a ResourceFlavor

A ResourceFlavor represents a set of node resources. For distributed workloads that require GPUs, create a ResourceFlavor with node labels that match your GPU nodes.

Example: ResourceFlavor for GPU nodes

apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: gpu-flavor
spec:
  nodeLabels:
    nvidia.com/gpu.product: Tesla-T4
  tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule
  1. nodeLabels: Targets nodes with a specific GPU model. Workloads admitted with this flavor are automatically scheduled to matching nodes.
  2. tolerations: Allows workloads to be scheduled on GPU-tainted nodes. Add tolerations that match the taints on your GPU nodes.

Apply the ResourceFlavor:

kubectl apply -f gpu-flavor.yaml

Check existing ResourceFlavors:

kubectl get resourceflavors

2. Create a ClusterQueue

A ClusterQueue defines the total resource quota available for distributed workloads.

Example: ClusterQueue for distributed workloads with GPU resources

apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: distributed-workloads-queue
spec:
  namespaceSelector: {} 
  resourceGroups:
  - coveredResources: ["cpu", "memory", "pods"] 
    flavors:
    - name: "default-flavor"
      resources:
      - name: "cpu"
        nominalQuota: 32
      - name: "memory"
        nominalQuota: 128Gi
      - name: "pods"
        nominalQuota: 20
  - coveredResources: ["nvidia.com/gpu"] 
    flavors:
    - name: "gpu-flavor"
      resources:
      - name: "nvidia.com/gpu"
        nominalQuota: 8
  1. namespaceSelector: {}: An empty selector allows all namespaces to use this ClusterQueue. To restrict access to specific namespaces, use matchLabels.
  2. General compute resources: CPU, memory, and pod count quotas for the distributed workload infrastructure (head nodes, driver pods, etc.).
  3. GPU resources: The total number of GPUs available for distributed workloads. Adjust based on your cluster capacity.
INFO

Note: Every resource that a distributed workload might request must be listed in coveredResources with a nominalQuota value (even if 0). If a workload requests a resource that is not covered, it will not be admitted.

Apply the ClusterQueue:

kubectl apply -f distributed-workloads-queue.yaml

3. Create a LocalQueue

A LocalQueue allocates resources from the ClusterQueue to a specific namespace where distributed workloads will run.

Example: LocalQueue for a team namespace

apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  namespace: team-ml
  name: team-ml-queue
spec:
  clusterQueue: distributed-workloads-queue

Apply the LocalQueue:

kubectl apply -f team-ml-queue.yaml

Verify the LocalQueue:

kubectl get localqueues -n team-ml

4. Verify the configuration

  1. Check that the ClusterQueue is active:

    kubectl get clusterqueues

    The ClusterQueue should show as Active.

  2. Check that the LocalQueue is connected:

    kubectl get localqueues -n team-ml -o wide

Using HAMi virtual GPU resources

If you use Alauda Build of HAMi for GPU virtualization and sharing, configure the ClusterQueue with HAMi-specific resource names:

apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: distributed-workloads-queue
spec:
  namespaceSelector: {}
  resourceGroups:
  - coveredResources: ["cpu", "memory", "pods"]
    flavors:
    - name: "default-flavor"
      resources:
      - name: "cpu"
        nominalQuota: 32
      - name: "memory"
        nominalQuota: 128Gi
      - name: "pods"
        nominalQuota: 20
  - coveredResources: ["nvidia.com/gpualloc", "nvidia.com/total-gpucores", "nvidia.com/total-gpumem"]
    flavors:
    - name: "gpu-flavor"
      resources:
      - name: "nvidia.com/gpualloc"
        nominalQuota: "8"
      - name: "nvidia.com/total-gpucores"
        nominalQuota: "800"
      - name: "nvidia.com/total-gpumem"
        nominalQuota: "81920"
INFO

Note: If you use Alauda Build of HAMi, use nvidia.com/gpualloc, nvidia.com/total-gpucores, and nvidia.com/total-gpumem as the GPU resource names. If you use the Alauda Build of NVIDIA GPU Device Plugin (physical GPUs), use nvidia.com/gpu instead.

Next steps