Volcano DRA report puts GPU gang admission under scrutiny

A reported admission bypass makes partial GPU jobs the test case

Volcano issue #6183, opened October 28, 2026, reports a failure mode with a direct operational consequence: a distributed GPU job can start only some of its workers. According to the report, Pods using Dynamic Resource Allocation, or DRA, without ordinary resource requests can bypass gang scheduling through BestEffort backfill. Workers that start may then hold GPUs while waiting for teammates that remain unscheduled.

This is an open bug report, not evidence of a released fix or a universally reproduced failure. Its scope should remain attached to the reported configuration until operators establish their own reproduction. The immediate change is a validation priority: test whether gang admission still holds when GPU requirements are expressed through ResourceClaims rather than familiar resource requests.

For infrastructure teams, the question is not simply whether Kubernetes assigned a GPU. It is whether the scheduler admitted a viable distributed workload, whether its claims resolved as intended, and whether useful computation began. A green Pod status alone cannot answer those questions.

Gang admission and device allocation are different contracts

Gang scheduling groups work that needs a minimum set of members before execution makes sense. An eight-worker training run might require all eight workers; another application might support a smaller elastic membership. The scheduler must apply the declared minimum, not infer it from a job name or assume every distributed application requires identical treatment.

DRA addresses resource discovery, selection and allocation through objects including ResourceClaims. It does not, by itself, establish an application-level promise that every member of a distributed job will become ready together. The scheduler, DRA components, device driver and workload controller have related but distinct responsibilities.

That distinction matters in this report. An admission path that considers ordinary requests differently from DRA-backed device requirements can make a workload appear eligible for backfill even when its useful execution depends on other workers. The practical failure is a mismatch between resource allocation and group admission.

Do not collapse every startup problem into this suspected bug. Image pulls, device preparation, application rendezvous, networking and storage access can also leave a job partially running. Diagnose the point where the expected group-level decision diverges from observed behavior before assigning a cause.

Establish the exact environment before testing

Record the Kubernetes version, Volcano version, scheduler configuration, DRA driver version and relevant feature settings. Keep the rendered Pod specifications, group definition, ResourceClaims and claim templates. Include admission policies that mutate requests or inject containers: the submitted manifest may not be the manifest the scheduler evaluates.

The supplied Kubernetes DRA update reference is background for examining the API and behavior of the release under evaluation. It is not evidence that a particular Volcano configuration has been tested. Check documentation against the actual cluster version rather than assuming that newer API documentation describes an older installation.

Use an isolated test namespace and a bounded device pool. Reproducing stranded GPUs on a production queue can delay unrelated jobs, particularly when allocations persist until a controller explicitly cancels the workload. Agree on a maximum test duration and an owner authorized to stop it.

Preserve evidence before cleanup. Scheduler events, allocation state and workload-controller decisions can disappear or become harder to interpret after retries replace Pods. A useful reproduction includes both the failing case and a closely matched control, not just a screenshot of pending workers.

Build a matrix around requests and available capacity

Start with a fixed-size job whose minimum membership equals its worker count. Give each worker a clearly defined device requirement and make the application expose whether it has completed rendezvous. Then create a capacity shortage that prevents the complete group from fitting while leaving enough devices for a subset.

Compare DRA-only workers with an otherwise equivalent case that includes realistic CPU and memory requests. Repeat under sufficient capacity. Keep scheduler settings and device selection constraints unchanged so that the experiment isolates the admission path rather than changing several variables at once.

Run each case more than once and retain timestamps. Scheduling is asynchronous; a brief startup spread is not equivalent to a permanently stranded partial gang. Define the difference before testing, using the application's startup expectations and an explicit observation window.

Ordinary requests are necessary accounting, not a proven fix

CPU and memory requests describe real work around GPU execution: data loading, tokenization, checkpoint handling, communication and host-side runtime processes. Omitting them can distort placement even when device allocation works correctly. Set requests from measured requirements and expected concurrency rather than adding arbitrary values solely to influence scheduling classification.

However, adding requests must not be presented as a verified remedy for issue #6183 without a successful reproduction and control test. It changes scheduler inputs and may change which path is exercised. That is a useful diagnostic result, but it does not prove the underlying problem is resolved across workloads or versions.

Also distinguish the report's BestEffort backfill terminology from Kubernetes Pod quality-of-service classification. Inspect the actual Pod QoS class and scheduler decisions separately. A label that sounds familiar is not sufficient evidence that two subsystems use the same classification rules.

Requests are not generally hard consumption ceilings. Limits and device isolation mechanisms address different questions. A CPU request does not constrain GPU memory, and a ResourceClaim should not be interpreted as an application-level guarantee of exclusive device use without checking the driver and allocation configuration.

Follow ResourceClaims through the whole startup sequence

For each worker, establish which ResourceClaim it references, whether that claim is allocated, which device selection constraints apply, and whether allocation is compatible with the node selected for the Pod. Distinguish claim creation from allocation and allocation from successful device preparation inside the runtime.

A shared-looking claim reference is not proof that multiple workers can safely use the same GPU. Access semantics depend on the resource configuration and driver support. Inspect the intended sharing model explicitly, including whether the workload requires separate devices, partitioned devices or supported concurrent access.

A small inspection sequence can anchor the investigation:

NS=gpu-admission-test
kubectl get pods -n "$NS" -o wide
kubectl get resourceclaims -n "$NS" -o yaml
kubectl get pods -n "$NS" \
  -o custom-columns='NAME:.metadata.name,SCHEDULER:.spec.schedulerName,QOS:.status.qosClass,PHASE:.status.phase'
kubectl get events -n "$NS" --sort-by=.metadata.creationTimestamp

These commands expose state; they do not prove gang correctness. Save the complete Pod specifications and inspect scheduler logs using your deployment's procedures. Correlate group identity, Pod identity and claim identity so that retries do not accidentally combine evidence from different attempts.

Measure application progress, not just running Pods

Build a per-job timeline with submission, claim allocation, Pod assignment, container start, application rendezvous and first useful computation. Record cancellation and confirmed resource release as well. This separates admission delay from device preparation delay and application startup delay.

A useful partial-start detector compares the required membership with workers that have actually entered execution. Alert when some workers have started but the minimum application membership remains unavailable beyond an agreed grace period. Suppress expected startup transients, and account for applications that intentionally permit elastic membership.

Do not use low GPU utilization as the only signal. A waiting process may reserve device memory while performing little compute, while a healthy workload can have legitimate communication or input stalls. Combine job membership, allocation duration and application progress with device telemetry.

The response should normally act on the job, not randomly kill individual workers. Define whether the controller cancels and retries the entire group, how it prevents retry storms, and how operators verify that claims and devices are reusable. Test that recovery path as carefully as initial admission.

An illustrative eight-GPU calculation

Consider an illustrative pool of eight GPUs and a fixed-size job requiring eight workers, each using one GPU. Four GPUs are occupied by another workload for the next 20 minutes. If four workers from the new job start immediately but cannot compute until all eight join, they reserve four GPUs without advancing the job during that interval.

The stranded allocation is four GPUs multiplied by 20 minutes: 80 GPU-minutes, or 1.33 GPU-hours. At an illustrative internal allocation rate of $3 per GPU-hour, that is $4 of capacity charged to workers that have not begun useful computation. These are hypothetical accounting values, not product benchmarks or necessarily avoidable cash expenditure.

Suppose the complete job needs 30 minutes once all workers join. Partial admission produces 80 waiting GPU-minutes plus 240 useful GPU-minutes, totaling 320 allocated GPU-minutes. All-or-nothing admission at minute 20 produces 240 allocated GPU-minutes. Both jobs finish at minute 50 under these assumptions; the admission policy changes stranded allocation, not completion time.

If independent five-minute tasks can use the four free GPUs and finish before minute 20, that window could accommodate 16 one-GPU tasks. This opportunity depends on compatible placement, predictable durations and no interference with the gang's eventual admission. Backfill is valuable; the requirement is to keep it from admitting unusable fractions of an indivisible job.

GPU sharing does not replace admission control

GPU time-slicing, MIG and MPS address how workloads use devices, not whether a distributed job has enough workers to proceed. Time-slicing allows multiplexed access but should not be treated as a hard GPU-memory or performance boundary. Advertising more sharing slots cannot make a blocked collective operation useful.

MIG, on supported hardware and configurations, creates hardware-backed GPU partitions with defined resources. MPS supports concurrent CUDA execution and has configuration-dependent controls, but it should not be assumed equivalent to MIG's hardware isolation. Neither mechanism automatically supplies gang semantics for a multiworker application.

For Kubernetes GPU sharing with hard limits, ask which dimension must be bounded: device memory, compute access, tenant visibility or allocation count. Then verify what the hardware, driver and orchestration stack actually enforce. A quota on requested resources is not necessarily a cap on runtime interference.

For deeper estate-planning context, use Clastiq's guide to open models and GPU estate operations. Keep the admission test focused: validate group feasibility first, then evaluate sharing and isolation separately. Combining them prematurely makes both failures and apparent successes harder to explain.

Connect coordinated infrastructure to accountable outcomes

Cognition-AI, based in Muscat, Oman, owns the rFactorial product family. Within that family, Clastiq Orchestrator provides private-cloud orchestration across compute, GPU, storage and networks. The relevance here is coordinated infrastructure planning: a distributed GPU job depends on more than device availability alone.

A practical operating workflow starts with a declared worker minimum, measured CPU and memory requirements, device requirements, storage access and network needs. Operators then validate admission, observe application startup and confirm cleanup. Clastiq's stated scope connects to those infrastructure domains; it does not establish that Clastiq implements Volcano, supports a particular DRA driver or resolves this reported bug.

Clastiq Lite brings self-service AI and GPU sharing to teams. In that setting, publish admission expectations alongside sharing choices. Teams should know whether a request represents an exclusive device or shared access, what isolation they can expect, and how a partially started job is handled. These are governance requirements to establish, not undocumented product features to assume.

Measure outcomes using stranded allocation minutes, time to full membership, cancellation-to-release time and completed useful jobs. For on-premises GPUs, distinguish internal chargeback from actual cash savings: ownership and operating costs continue while hardware waits. Better admission can recover usable capacity without automatically reducing the infrastructure bill.

What to do this week

• Preserve the issue's reported configuration and compare it with your Kubernetes, Volcano and DRA environment before declaring exposure.
• Run the four-case admission matrix in an isolated pool, recording both scheduler decisions and application-level membership.
• Audit rendered CPU and memory requests, ResourceClaim references and device constraints; avoid treating request changes as a confirmed fix.
• Add a bounded partial-start alert and test job-level cancellation, retry behavior and confirmed device release.
• Report stranded GPU-minutes separately from useful computation, then review sharing boundaries and chargeback rules with workload owners.

Explore Clastiq Orchestrator for private-cloud orchestration across compute, GPU, storage and networks, and the rFactorial product family for the wider product context. Bring the admission matrix and recovery requirements to that evaluation rather than assuming a product label proves scheduling correctness.

Sources

• Volcano issue #6183
• Kubernetes DRA updates reference, September 3, 2026
• Clastiq guide to open models and GPU estate operations
• rFactorial product family