Gang scheduling and topology-aware placement for AI training in Kubernetes

Distributed AI training workloads impose complex scheduling requirements that Kubernetes’s built-in scheduler can’t meet. Kubernetes schedules pods individually and independently, but distributed training introduces two requirements that break this model: Pods must land on hardware with the right inter-GPU bandwidth, and all pods must be scheduled simultaneously. If either requirement goes unmet, training stalls or runs far below the hardware’s potential.

Topology-aware scheduling (TAS) and gang scheduling address these two requirements and help organizations extract full value from GPU capacity, which represents a growing share of infrastructure spend. TAS shapes the relative placement of pods to enable fast communication between GPUs. Gang scheduling helps ensure that all pods in a distributed job start running simultaneously, preventing the partial starts that leave GPUs allocated but idle. Together, they give Kubernetes the placement precision and coordination that distributed workloads require.

In this post, we’ll look at the scheduling problems that AI training presents for Kubernetes and how TAS and gang scheduling, implemented by Kueue and the Coscheduling plugin, solve them. We’ll explore:

• How the Kubernetes scheduler is unsuited for AI training
• How to validate gang scheduling and TAS in Kubernetes

We’ll also cover the evolving capabilities of Kubernetes native scheduler to support AI training workloads.

How the Kubernetes scheduler is unsuited for AI training

In most batch workloads, pods operate independently. A pod can fail, be rescheduled, or be replaced with no effect on the rest of the workload. The Kubernetes scheduler handles this model well by placing pods one at a time on any eligible node without coordinating across pods.

But the pods in an AI training workload are interdependent. They need to be placed on hardware with appropriate inter-GPU bandwidth, and they all need to start simultaneously. The following sections explain these requirements and how Kueue and Coscheduling address them by augmenting Kubernetes’ scheduling capabilities.

How topology placement affects distributed AI training performance

The physical interconnections between nodes are a key factor in the performance of AI training workloads. In each step of the training process, workers exchange data through collective communication operations, and the type and frequency of those operations depends on the job’s parallelization strategy. In pure data-parallel (DDP) training, one AllReduce operation per step aggregates, averages, and distributes gradients—the computed signals that indicate how the model’s parameters should be adjusted to improve its predictions. In large-scale training strategies that shard the model across workers, those workers instead exchange data continuously throughout a step, by using operations like gather, reduce, and all-to-all. These large-scale strategies include tensor, pipeline, and expert parallelism, which are often combined in LLM pretraining. Whichever pattern a job uses, the underlying constraint is the same: This communication is latency-sensitive, and where those workers land in the cluster determines how much of that latency the job absorbs.

The bandwidth available between workers depends on where they sit relative to each other in the cluster. The following topology domains are common in GPU clusters:

• Same node: This is generally the lowest-latency option. When the GPUs are connected by NVLink and NVSwitch, workers can use NVIDIA’s high-bandwidth GPU interconnect. Otherwise, same-node GPU communication may traverse Peripheral Component Interconnect Express (PCIe) connections.
• Different nodes in the same rack: Workers communicate via high-speed network fabric. Cross-node traffic travels through PCIe to the network interface and back, with intermediate latency.
• Different racks: This is the highest-latency option. Workers communicate across the cluster’s InfiniBand fabric, which is the lowest-bandwidth hop in the path and adds further latency on top of the cross-node PCIe path.

Different workloads have different tolerances for latency. In a pretraining job that comprises a large number of workers, each step blocks until the slowest worker completes its share of that step’s collective communication. Any cross-rack latency here compounds quickly, reducing effective GPU utilization. A small fine-tuning job involves fewer workers exchanging smaller gradients, so the same cross-rack penalty represents a much smaller fraction of each step’s total time.

Without topology awareness, the Kubernetes scheduler places workers on any eligible nodes, regardless of their physical relationships in the cluster. The gang assembles and the training job runs, but communication overhead—rather than computation—consumes a substantial portion of each step. The job completes more slowly and at greater cost than the hardware should require. Crucially, infrastructure metrics don’t reflect this inefficiency because they measure activity, not productivity. Nothing looks wrong from an infrastructure perspective, but training throughput remains low relative to what the hardware should deliver.

Kueue closes part of this gap through topology-aware placement. It groups nodes into domains—block, rack, and host—based on topology labels applied to those nodes (for example, topology.kubernetes.io/rack). Kueue places a job’s pods within a single domain that satisfies its bandwidth requirements. But Kueue doesn’t discover cluster topology on its own. The labels that define each domain have to be applied to nodes first, typically by a cluster administrator or a cloud provider’s topology labeler.

Kueue enforces this placement decision at admission time. It’s an open source Kubernetes-native job queuing system that sits above the kube-scheduler, controlling which jobs reach the scheduler and when. During that admission check, it also verifies that a job’s pods fit within a topology domain. It evaluates each job against quota, priority, and preemption criteria and decides whether to allow the job to proceed. It tracks that decision in a Kueue Workload object, which represents the resource requirements, admission status, and pod sets for a job. Kueue also provides optional life cycle management for admitted jobs, including evicting and requeueing workloads whose pods don’t all reach a ready state within a configured timeout.

When Kueue admits a job to be scheduled (by setting spec.suspend: false, allowing the Job controller to create pods), it writes scheduling constraints into the training job’s pod templates based on the placement mode and topology assignment it selected. Those constraints restrict which nodes the Kubernetes scheduler may assign so that pods stay within the selected topology domain.

The following diagram shows the path that a pod takes from job submission to binding. Kueue acts first, writing topology constraints at admission time and clearing the job for scheduling. The Job controller creates the pods, but they exist only as objects in etcd, without assigned nodes or running containers. Coscheduling acts second, holding each pod in a waiting state until the full gang is ready to bind simultaneously.

How to validate gang scheduling and TAS in Kubernetes

Kueue and Coscheduling address the unique scheduling requirements of AI training workloads. To operate your training jobs effectively and efficiently, you need to know that Kueue and Coscheduling are working as intended. Because tracking down scheduling failures requires piecing together signals from multiple layers of your stack, this section breaks down how to validate each step of the process:

• Monitor activity in the admission queue to ensure jobs are being accepted.
• Validate placement effectiveness to confirm jobs land in the right physical topology.
• Monitor Coscheduling gang assembly to catch partial starts.
• Confirm scheduling effectiveness with training throughput signals to tie infrastructure metrics directly to application performance.

Monitor activity in the admission queue

Kueue exposes OpenMetrics-formatted metrics that show whether jobs are clearing the admission queue or stalling there.

kueue_pending_workloads shows queue depth broken down by status. The active status counts jobs currently waiting in the queue for capacity to open up. The inadmissible status counts jobs that Kueue has evaluated and cannot currently admit, typically because quota in the assigned ClusterQueue is exhausted, no valid topology domain exists, or a constraint can’t be satisfied. Jobs remain inadmissible until cluster conditions change. A growing inadmissible count that doesn’t clear is a sign of a configuration or capacity problem rather than normal queuing.

kueue_admission_wait_time_seconds measures how long jobs wait between creation or requeue and admission. A rising p99 indicates that jobs are arriving faster than Kueue can admit them, which can be due to quota exhaustion, priority contention among many jobs, or preemption processing taking time to complete.

kueue_evicted_workloads_total tracks evictions with a reason label that distinguishes the cause of each eviction. When a node fails under TAS and Kueue cannot find a replacement, it evicts the affected workload with reason="NodeFailures". A rise in this signal points to non-recoverable node loss in the assigned topology domain. A related reason, reason="PodsReadyTimeout", fires when a gang’s pods do not reach a ready state within the configured timeout. That timeout expires for many reasons, including slow scheduling, image pulls, resource pressure, or a node failure whose recovery timed out. Treat an increase as a prompt to investigate rather than proof of a single cause. Both are post-admission stability signals, separate from confirming that a workload landed in its intended topology domain.

Validate placement effectiveness

In addition to tracking Kueue metrics to understand admission health and eviction behavior, you can inspect the Kueue Workload object to confirm a workload’s topology assignment:

kubectl -n <namespace> get workloads.kueue.x-k8s.io <workload-name> -o yaml

Look for status.admission.podSetAssignments[].topologyAssignment. This field lists the topology levels and domain values into which each PodSet was placed, and is the authoritative signal that TAS was applied.

After you’ve confirmed the topology assignment, use GPU and network telemetry data to validate whether the hardware is delivering the performance that placement implies. Same-node GPU communication should show NVLink or other local interconnect activity when that hardware path is available. Evaluate cross-node communication by looking at PCIe activity together with network or fabric telemetry data: PCIe throughput reflects the GPU-to-host transfers required for cross-node communication, but it does not measure the network hop itself.

If throughput is low despite a valid topology assignment, look for a physical infrastructure explanation, such as driver issues, hardware faults, or fabric degradation, rather than treating it as a Kueue placement failure. Hardware bottlenecks like these can surface as lower-than-expected bandwidth and slower collective communication times, not necessarily as idle GPUs, which is why infrastructure metrics can look healthy while training throughput degrades.

Monitor Coscheduling gang assembly in Kubernetes

The Coscheduling plugin uses a custom resource called a PodGroup to track each training gang. A PodGroup records the gang’s minimum worker requirement and current scheduling state. Pods belong to the same PodGroup when they use the same scheduling.x-k8s.io/pod-group label.

You can query PodGroups with kubectl:

kubectl get podgroups.scheduling.x-k8s.io --all-namespaces

For more detail about a specific gang, describe the PodGroup or inspect its status:

kubectl -n <namespace> describe podgroup.scheduling.x-k8s.io <podgroup-name>

kubectl -n <namespace> get podgroup.scheduling.x-k8s.io <podgroup-name> -o yaml

The exact columns shown by kubectl can vary by version and CRD configuration, so focus on the PodGroup’s spec and status fields. spec.minMember shows the minimum number of pods required for the gang to run. spec.scheduleTimeoutSeconds, when configured, controls how long the group can wait. status.phase shows the gang’s current state, which is one of the following values:

• Pending: The PodGroup has been accepted but the scheduler has not allocated enough resources.
• Scheduling: More pods have been scheduled than spec.minMember, but the number of running pods hasn’t reached spec.minMember yet.
• Running: At least spec.minMember pods are running.
• Unknown: Some, but not all, of the required pods were scheduled. The rest can’t be scheduled (for example, due to insufficient resources), and the scheduler waits for controllers to recover them.
• Finished: The required pods completed successfully.
• Failed: At least one of the required pods has failed.

You should expect a healthy gang to move from Pending or Scheduling to Running within the scheduling window you expect for that workload. If a gang remains Pending or Scheduling longer than expected, check the PodGroup events, associated pod events, and scheduler logs. A delay like this can indicate that the cluster cannot satisfy the gang’s resource requirements within the topology domain Kueue assigned, or that no available node satisfies scheduling requirements such as node selectors, taints, affinity, or resource availability.

A gang stuck between Scheduling and Running with only partial placement may surface as Unknown rather than remaining in Scheduling. The Unknown phase differs from Failed, because the scheduler is still waiting on controllers to recover the unplaced pods, not reporting an outright failure.

To distinguish an admission problem from a scheduler-level assembly problem, correlate PodGroup state with Kueue’s Workload status and admission metrics. If the Workload is still inadmissible, the issue is upstream of Coscheduling: Kueue has not admitted the job, so the fix is likely quota, priority, admission checks, or topology configuration. If the Workload is admitted and has a topology assignment, but the PodGroup remains Pending or Scheduling, the issue is later in the pipeline: Coscheduling and the kube-scheduler cannot assemble enough pods within the assigned placement and current cluster state.

If a PodGroup reaches Failed, investigate the failed pods directly and look for common issues such as application failure, image or startup errors, or node disruption.

Confirm scheduling effectiveness with training throughput signals

Training frameworks are the software libraries that orchestrate distributed training, such as PyTorch, JAX, and Ray Train. Only by monitoring signals from these frameworks can you determine whether GPU cycles are producing completed training progress or being consumed by stalled communication.

Ray is unique among these frameworks in its placement group model, which maps directly onto gang scheduling and gives it scheduling-aware observability. ray_placement_groups tracks the current number of placement groups by State, such as PENDING, CREATED, or REMOVED. A sustained increase in PENDING placement groups indicates that Ray is waiting for placement group resources to become available. ray_tasks tracks tasks by State, which can help show whether application work is accumulating while placement groups remain pending.

When these signals rise together, Ray cannot satisfy its placement requirements. This can be a downstream effect of a Coscheduling assembly issue or a Kueue topology constraint problem rather than a simple application-layer slowdown.

Other distributed training frameworks (including PyTorch DDP, Kubeflow operators, and Horovod) don’t expose metrics that directly reflect the Kubernetes scheduler’s effectiveness. These frameworks do provide indirect indicators in the form of throughput metrics, such as steps per second, samples per second, and tokens per second. A throughput drop in these metrics alongside decreasing GPU interconnect bandwidth and PodGroup timeouts suggests a scheduling root cause. PodGroup timeouts confirm that gang assembly did not complete within the expected scheduling window. Correlating those timeouts with the Kueue Workload’s topology assignment and GPU, PCIe, and network telemetry data can help determine whether placement constraints contributed to the slowdown.

Native Kubernetes scheduling: How v1.35 through v1.37 address AI training requirements

Kueue and Coscheduling are out-of-tree components, installed and managed separately from Kubernetes. Kubernetes is beginning to add native scheduler capabilities that address the complex scheduling requirements this post describes, but those features are still early.

Kubernetes v1.35 introduced native support for gang scheduling as an alpha feature. Kubernetes v1.36 added several related alpha features for AI training workloads, including topology-aware scheduling, workload-aware preemption, and initial Job controller integration via a feature gate. Kubernetes v1.36 also added a native PodGroup API, distinct from the out-of-tree PodGroup custom resource definition (CRD) used by the Coscheduling plugin. Kubernetes v1.37 graduated gang scheduling and workload-aware preemption to beta under the GenericWorkload feature gate, which remains disabled by default.

The native TAS implementation enables the kube-scheduler to understand physical cluster topology and place pods within an appropriate topology domain, such as a rack or zone, without requiring Kueue to write node affinity rules into pod templates.

These native capabilities are not yet a drop-in replacement for the Kueue and Coscheduling path described in this post. Gang scheduling and workload-aware preemption reached beta in v1.37 but ship disabled by default and still lack feature parity with Kueue and Coscheduling. Topology-aware workload scheduling remains alpha.

Kueue and Coscheduling remain the production path for distributed training scheduling. Their monitoring signals, including admission queue depth, gang assembly state, topology placement, and GPU throughput, map directly to the concepts the native APIs expose. Teams that build observability into the current stack will have a working baseline when they evaluate or migrate to these native capabilities.

Get started with gang scheduling and topology-aware placement

Monitoring distributed AI training scheduling in Kubernetes requires visibility across components that standard tooling doesn’t naturally connect: Kueue’s admission queue, Coscheduling’s gang assembly, the GPU interconnect, and the training framework. Applying a common topology tag across all of these signals helps you monitor your clusters’ gang scheduling and TAS activity and ensure that your training jobs make efficient progress.

To get started, read the Kueue documentation and the Coscheduling plugin documentation.

This post originally appeared on the Datadog blog, where you can read the full version, including how to monitor and troubleshoot this stack in production.