Kueue’s patch makes GPU reservations an acceptance-test issue
Kueue v0.20.4, released October 29, 2026, addresses two defects that matter to operators sharing accelerator infrastructure: availability checks involving stale or incomplete Dynamic Resource Allocation (DRA) ResourceSlice pools, and fair-sharing quota reclamation. The operational question is whether a reservation represents capacity that a workload can actually use—and whether another team can recover its allocation when it needs it.
The release notes provide the upgrade peg. They should also prompt a test-plan change. A healthy controller and an admitted workload are insufficient evidence of a healthy GPU service. Operators need to follow the chain from quota reservation through device allocation, pod scheduling and application startup, then prove that borrowing and reclamation behave according to policy.
For rFactorial, the product group owned by Muscat-based Cognition-AI, this is a private-cloud orchestration issue. Clastiq Orchestrator spans compute, GPU, storage and networks; Clastiq Lite brings self-service AI and GPU sharing to teams. Those verified capabilities make admission, availability and team governance relevant evaluation criteria. They do not establish a Kueue or DRA integration for either product.
Separate admission from allocation and isolation
GPU sharing is an overloaded term. At the cluster level, it can mean that teams borrow unused quota. At the device level, it can mean multiple processes use one accelerator. These mechanisms solve different problems and offer different protections.
Kueue manages workload admission against configured quotas and policies. A quota reservation is an accounting and admission decision, not a guarantee that an application has acquired usable device memory. DRA adds a resource-claim and device-allocation path; drivers publish device information through ResourceSlices. Scheduling, node health and application initialization still have to succeed.
| Layer | Operator question | Evidence to inspect |
|---|---|---|
| Admission | May this workload consume team quota? | Workload conditions and queue accounting |
| Device allocation | Does an eligible device exist? | ResourceClaims, ResourceSlices and driver status |
| Execution | Did the pod obtain and initialize it? | Pod events, node health and application logs |
| Isolation | What can another tenant consume or disrupt? | Sharing mode, enforceable limits and contention tests |
The patch targets admission-related correctness, not all four layers. Even accurate device discovery cannot prevent an image-pull failure, storage delay or incompatible application runtime. Keep those failures distinct so a successful upgrade is neither credited for unrelated improvements nor blamed for every pending pod.
Why ResourceSlice freshness matters
DRA’s device inventory is published through ResourceSlices grouped into pools. Operators must interpret the pool as a coherent inventory rather than treating any individual slice as complete truth. Generation and completeness information matter when drivers update their published resources.
A stale view can describe devices that are no longer available. An incomplete pool can leave a consumer without enough information to make a sound availability decision. The relevant Kueue 0.20.4 fix concerns these availability checks. Do not generalize that into a promise that every reservation now guarantees immediate execution.
The Kubernetes DRA update is useful architectural context, but Kubernetes, Kueue and the installed device driver have separate versions and requirements. Record the actual combination you run. Confirm that the availability-check path described by the fix applies to your configuration before designing a regression test around it.
Avoid manually corrupting production ResourceSlices to reproduce the defect. Use an isolated cluster, a driver-supported test mechanism or a controlled fixture. Preserve the before-and-after inventory and workload conditions. The test needs to demonstrate behavior under known stale or incomplete state, not merely create an unexplained scheduling outage.
Reclamation is a service commitment
Borrowing improves utilisation when one team has idle entitlement and another has queued work. But a borrowing policy is credible only if the entitled team can recover capacity under the configured rules. The fair-sharing quota-reclamation defect fixed in this release therefore deserves its own acceptance test.
Distinguish reclaiming quota accounting from obtaining a running GPU workload. Depending on the queue and preemption configuration, returning borrowed capacity can involve terminating work, waiting for pods to stop, releasing device claims and admitting replacement work. Checkpointing and termination grace periods affect the disruption and elapsed time.
Write the policy in operational language: which queues may borrow, which workloads can be displaced, how priority interacts with entitlement, and what happens to interrupted jobs. Then map those statements to the settings supported by your deployed Kueue version. Do not assume that enabling fair sharing automatically delivers the reclamation behavior your organisation expects.
Test with checkpoint-capable batch jobs before exposing long, non-restartable workloads to reclamation. A technically successful quota return can still waste expensive computation. Report both time to recover usable capacity and work lost during interruption; neither metric substitutes for the other.
Build an upgrade baseline before changing controllers
Inventory the Kubernetes version, Kueue version, DRA driver, device classes, queue configuration and workload integrations. Identify which workloads actually use DRA; do not assume a fix for that allocation path changes traditional GPU resource requests in the same way.
Capture pending-workload counts, quota-reservation age, admission-to-running latency and reclamation timing before the upgrade. Segment by queue and requested device type. Cluster-wide averages can conceal one team holding reservations while its workloads remain unable to start.
A short read-only collection sequence helps establish the evidence set:
kubectl get clusterqueues.kueue.x-k8s.io -o yaml
kubectl get localqueues.kueue.x-k8s.io -A -o yaml
kubectl get workloads.kueue.x-k8s.io -A -o yaml
kubectl get resourceslices.resource.k8s.io -o yaml
kubectl get resourceclaims.resource.k8s.io -A -o yaml
kubectl get events -A --sort-by=.metadata.creationTimestamp
Run these with appropriately scoped credentials. Available APIs depend on the cluster configuration, and events have limited retention. Save timestamped snapshots in an approved location, removing sensitive workload metadata before sharing outside the operations team.
Review the patch’s installation and compatibility guidance against your deployment method. Keep manifests, values and configuration changes under version control. Define a rollback trigger, but check whether any accompanying API or configuration changes constrain rollback. Do not bundle unrelated driver, firmware and queue-policy changes into the same experiment.
Acceptance tests should follow the complete workload lifecycle
Start with a positive control: a healthy, eligible device and a workload that fits the intended queue. Confirm reservation, allocation, scheduling and application initialization. Use a small application that performs real accelerator work, rather than treating container startup as proof of GPU access.
Next, exercise stale and incomplete pool conditions in the isolated environment. Record when the driver publishes the state, when Kueue observes it and what happens to the workload. Compare the outcome with the release’s regression expectations and your configured availability-check behavior. The acceptance criterion is not simply that every affected workload remains pending; it is that known-unusable inventory does not produce the erroneous reservation behavior covered by the fix.
Restore a complete, current inventory and verify recovery. A controller that avoids an incorrect reservation but never notices restored devices still fails the service objective. Measure how long the workload takes to progress after the inventory becomes valid.
For reclamation, let one queue borrow unused entitlement, then submit eligible work from the entitled queue. Observe the quota decision, any preemption, pod termination, claim release and the replacement workload’s first successful GPU operation. Repeat with different priorities and with borrowing disabled as a control.
Finally, introduce an unrelated execution failure, such as a deliberately invalid image reference. Confirm that monitoring distinguishes admission success from execution failure. This prevents the platform from presenting all reserved resources as productive GPU consumption.
An illustrative capacity and reclamation calculation
Consider an illustrative cluster with 16 equivalent GPUs and two teams, A and B, each assigned nominal quota for eight. Assume one GPU per job, no device-level sharing, compatible devices, and queue policies that permit borrowing and reclamation. These are hypothetical operating assumptions, not product benchmarks.
Initially, A runs four jobs and B runs 12. B therefore borrows four GPUs from A’s unused entitlement. When A submits four additional eligible jobs, successful reclamation should allow A to reach eight running jobs and B to return to eight, subject to the configured rules and workload shutdown behavior.
Suppose a pre-upgrade test takes 12 minutes from A’s submission until all four replacement jobs complete their first GPU operation. After the patch, the same controlled test takes three minutes. The improvement is nine minutes for the four-job group, or 36 GPU-minutes of earlier access: four GPUs multiplied by nine minutes. That is 0.6 GPU-hours of access shifted toward A—not necessarily additional cluster throughput.
At an illustrative internal allocation rate of $3 per GPU-hour, that interval represents $1.80 of capacity accounting. It is not an electricity saving or a reduction in hardware ownership cost. If B’s interrupted jobs lose more useful computation than A gains, total productive work can fall despite faster reclamation.
Run the availability case separately. If four of the 16 GPUs become unavailable, physical usable capacity is 12. Nominal quotas totalling 16 do not create four replacement devices. Report the capacity deficit separately from any incorrect reservations, and verify recovery when the inventory returns to 16 usable devices.
Time-slicing, MIG and MPS answer a different question
Operators searching for Kubernetes GPU sharing with hard limits need to distinguish admission quota from device enforcement. Giving a namespace four schedulable GPU shares does not, by itself, limit each share to one-quarter of GPU memory or guarantee one-quarter of execution time.
Time-slicing allows workloads to take turns using a GPU through the supported sharing implementation. It can broaden access for intermittent tasks, but an advertised share count should not be treated as a hard memory partition. Validate the implementation’s behavior under memory pressure, contention and faults.
Multi-Instance GPU, or MIG, partitions supported hardware into instances with defined memory and compute resources. Available profiles and operational constraints depend on the GPU and software stack. This is closer to hardware-backed partitioning, but it does not remove the need for tenant security controls, compatible workloads and capacity planning around instance sizes.
Multi-Process Service, or MPS, supports concurrent CUDA process execution and offers controls whose applicability depends on the configuration. It is not interchangeable with MIG or a general tenant-security boundary. Verify precisely which resource controls are enforced rather than inferring isolation from a concurrency setting.
Kueue quotas can govern admission to resources exposed by such arrangements, but quotas do not manufacture stronger device isolation. For complementary workload sizing, use Clastiq’s DeepSeek context-cost guide and the technical material at Clastiq. Context size and memory demand belong in capacity planning alongside queue policy; this release does not change that relationship.
Connect the tests to private-cloud operations
Clastiq Orchestrator’s verified scope—private-cloud orchestration across compute, GPU, storage and networks—puts GPU scheduling in the context of a complete service. A workload may have an available accelerator yet remain unproductive because its data path, host capacity or network access is not ready.
Clastiq Lite’s self-service AI and team GPU-sharing role makes clear allocation policies especially important. A useful operating workflow starts with an approved workload request, identifies its device and memory needs, applies team entitlement and borrowing rules, and measures whether the resulting execution meets the team’s expectations. Treat this as an operating model to evaluate, not a claim about undocumented product interfaces.
Ask how requests, approvals, runtime observations and interrupted work will be recorded in your deployment. Establish who may change sharing modes, who may authorize borrowing, and who resolves disputed chargeback. Keep the quota owner, workload owner and infrastructure operator identifiable even when one team performs several roles.
No supplied product information establishes native Kueue support, DRA support, MIG management or MPS configuration in Clastiq. Validate any proposed architecture separately. The defensible connection is the need to coordinate infrastructure and self-service access with explicit availability, isolation and governance requirements.
Measure idle ownership cost without hiding contention
On-premises GPUs incur ownership costs while idle. Queue sharing can improve access to that sunk capacity, but reservation counts are a poor utilisation proxy. Track reserved GPU-hours, allocated GPU-hours and application-productive GPU-hours separately, documenting how productive work is measured.
Add reservation-to-first-work latency, reclaim-to-first-work latency, interrupted compute time and unavailable-device duration. Review percentiles and failed attempts, not only averages. A lower queue wait can coexist with worse completion times if workloads contend heavily on shared devices.
For financial reporting, divide the agreed ownership-cost pool by productive GPU-hours to calculate an effective cost per productive hour. Keep that allocation metric separate from cash savings. Better scheduling does not erase depreciation, and aggressive reclamation may increase wasted work. The upgrade succeeds when correctness improves and teams receive predictable access without concealing those trade-offs.
What to do this week
- Inventory Kueue, Kubernetes, driver and queue versions; identify which workloads exercise the affected DRA path.
- Capture baseline reservation age, admission latency and reclamation timing, including application-level GPU readiness.
- Test stale and incomplete ResourceSlice pools safely, then prove recovery after valid inventory returns.
- Reproduce borrowing and reclamation with checkpoint-capable jobs; record both returned capacity and interrupted computation.
- Document whether each GPU service offers time-slicing, hardware partitioning or another sharing mode, and state its actual memory protections.
Evaluate Clastiq Orchestrator against these private-cloud operating requirements, with Clastiq Lite’s self-service team GPU sharing in scope. Explore the rFactorial product family for the verified product context, keeping integration and isolation requirements explicit during evaluation.