A failed-call credit does not return GPU time
Inferock’s October 14, 2026 Product Hunt launch introduced serverless inference with credits for eligible failed calls, according to the supplied launch information. The product page and week 42 leaderboard are the reference points for that announcement. The change worth examining is commercial: failure handling becomes part of the inference offer, rather than something customers discover only through support tickets.
That is a product claim, not independently demonstrated reliability. The launch information does not establish eligibility definitions, exclusions, credit settlement timing or observed failure rates. Operators should not infer service quality from a listing. Its value here is as a low-weight signal that failed requests belong explicitly in service economics.
For an operator running a quarter rack to a few racks, the question is concrete. A failed call may generate no customer revenue while consuming accelerator time, memory bandwidth, network capacity and electricity. A retry can consume those resources again. A credit changes the invoice; it does not reverse the work.
The practical response is a request-level ledger connected to admission control, tenancy and physical capacity accounting.
The bottleneck is admitted work, not just utilisation
Small GPU estates usually have little spare capacity behind the advertised service. Several tenants may share an inference pool while other accelerators serve virtual machines, Kubernetes workloads or reserved jobs. A model reload, slow storage path or client retry burst can displace useful work without making accelerator utilisation look low.
High utilisation therefore does not necessarily mean healthy economics. Devices can remain busy repeating work that should have completed once. Meanwhile, customer-visible throughput falls, queue age rises and service credits accumulate.
Clastiq Orchestrator provides the infrastructure scope needed to address this problem: bare-metal provisioning, Kubernetes and VM tenancy, GPU partitioning, Ceph storage, metering, chargeback and quota engineering. The request ledger proposed here is an application-layer design to integrate with those capabilities, not a claim that a particular inference billing integration ships ready-made.
Keep three questions separate: what the customer owes, what resources the workload consumed, and whether another request should enter the system. One counter cannot safely answer all three.
Give logical requests and execution attempts separate identities
Assign a logical request ID when the gateway receives a call. Assign a distinct attempt ID whenever the system performs an execution attempt. Carry both through admission, routing, scheduling, inference and settlement. Tenant identity must come from authenticated credentials, not an editable request field.
An idempotency key should identify the same customer operation within a documented scope and retention window. It can prevent duplicate charging or execution where supported, but it is not a universal replay mechanism. Streaming responses, expired records and changed request parameters need explicit handling.
Record transitions rather than overwriting a single status column. Accepted means the gateway admitted the operation under policy. Queued means it is waiting for execution resources. Started means backend execution began. Completed requires a defined terminal success condition. Cancellation, expiry and failure need distinct terminal outcomes.
Retry is a relationship between attempts, not a replacement for the original outcome. A logical request can complete successfully after an earlier attempt failed, leaving both useful output and wasted work to account for.
| Ledger event | Required evidence | Economic consequence |
|---|---|---|
| Accepted | Tenant, request ID, policy version | Admission recorded, not automatically billed |
| Queued | Enqueue time, deadline, queue | Capacity obligation becomes visible |
| Cancelled | Initiator, timestamp, execution state | Work already consumed remains attributable |
| Retried | Parent attempt, reason, cost owner | Additional work stays linked |
| Completed | Usage counters, terminal response | Billing becomes eligible for settlement |
Also capture model revision, backend, partition, pricing version and error class. Store enough evidence to explain a disputed invoice without retaining prompts by default.
Make completion and cancellation operationally precise
A successful backend result is not always a successful customer interaction. The model may finish after the gateway times out. A streaming client may disconnect after receiving some output. A transport failure may leave the operator uncertain whether the final response arrived.
Define the billing boundary before implementation. Options include generated tokens, gateway-emitted tokens or an explicitly acknowledged application result. These are different promises. An ordinary HTTP service generally cannot prove that a person consumed the response, and gateway emission is not proof of client receipt.
For cancellation, record when the client asked to stop and when execution actually stopped. If a backend cannot interrupt a batch immediately, the interval between those timestamps still consumes capacity. Do not mark that interval as idle simply because the customer connection closed.
For each ambiguous outcome, publish a settlement rule. A conservative service might withhold charges pending reconciliation; another might bill verifiable partial output. Either approach needs documentation, consistent implementation and a dispute path. Do not silently retry a partially streamed answer as though nothing reached the client.
Use an append-only ledger with independent settlement
The gateway should emit durable events, while inference workers report execution and token usage independently. Reconcile the two streams using request and attempt IDs. A missing terminal event should enter an exception queue, not become an assumed success or a free request by accident.
An illustrative event envelope is small enough to inspect:
{
"event_id": "evt-9281",
"request_id": "req-1042",
"attempt_id": "req-1042-a2",
"parent_attempt_id": "req-1042-a1",
"tenant_id": "tenant-7",
"event": "completed",
"model_revision": "model-r18",
"billing_policy": "policy-v3",
"retry_cost_owner": "operator",
"input_tokens": 800,
"output_tokens": 240,
"attributed_gpu_seconds": 1.75
}
This is a proposed schema, not a Clastiq API example. Production records also need timestamps, schema versions and attribution-method versions. Deduplicate by event ID and enforce uniqueness for settlement entries. Otherwise, replaying telemetry after an outage can replay charges.
Keep immutable usage facts separate from financial adjustments. A credit references the original charge and its policy basis; it should not erase consumed GPU seconds. Corrections should append compensating entries with an audit trail.
Reconcile token revenue with allocated GPU time
Token billing measures an output-related quantity. GPU accounting measures resource occupation. They will not move in lockstep: prompt lengths, output lengths, model architectures, batching, cache hits and memory pressure all affect the relationship.
Start with measured wall-clock device occupation for each serving pool. Separate productive execution, failed attempts, model loading, reservation idle time and unallocated idle time where instrumentation permits. For shared batches, allocate device-seconds using a documented weighting method, and label that allocation as an estimate rather than a direct per-request measurement.
Partition accounting needs equal care. A partition occupied for an hour is not automatically one full-device GPU-hour. Report partition-hours directly or use a declared normalisation method. Memory fractions alone may not represent compute entitlement or throughput.
The reconciliation invariant is that allocated work must not exceed measured pool consumption for the same interval. Preserve an unattributed bucket when evidence is missing. A visible discrepancy is more useful than invented precision.
Finally, reconcile financial entries separately: gross charges, credits, adjustments and net revenue. Never subtract financial credits from physical consumption.
A worked hour shows why retries need a cost owner
Consider an illustrative pool of eight whole GPUs observed for one hour. It offers eight GPU-hours of capacity. Assume occupied execution totals six GPU-hours: five serving billable successful work and one spent on failed attempts and recovery work that the operator absorbs. The remaining two GPU-hours are idle. Execution occupancy is therefore 6 ÷ 8 = 75%.
Assume a fully loaded capacity cost of $2 per provisioned GPU-hour, including an allocated share of equipment, power, cooling and operations. The pool costs 8 × $2 = $16 for the hour, whether occupied or idle. This is an accounting assumption, not a market price or benchmark.
During the hour, billable successful requests produce 12 million input tokens and 4 million output tokens. At illustrative prices of $0.50 per million input tokens and $3 per million output tokens, gross charges are $6 + $12 = $18. Suppose eligible failed-call adjustments total $2 against provisional charges on earlier invoices. Net revenue recognised in this settlement period becomes $16.
The service covers its assumed capacity cost but leaves no contribution beyond that cost model. The absorbed failure and recovery work carries $2 of allocated capacity cost. Idle capacity carries another $4. Together, those categories explain why apparently busy hardware produces a thin result.
Do not double-count the $2 credit as additional GPU expenditure. It reduces revenue. The failure work already sits inside the $16 capacity cost. Nor should the earlier credits be presented as a current-hour failure rate: operational and settlement cohorts differ.
Allocate retry costs by cause and contractual control
A retry policy should name both the trigger and the cost owner. Backend crashes, operator maintenance errors and internal routing failures normally belong in the operator’s service-cost bucket. Tenant-requested regeneration is new work. Invalid inputs should usually be rejected before expensive execution, with any applicable charge disclosed.
Timeouts require investigation rather than automatic blame. The client may set an unrealistic deadline, or the operator may admit work it cannot start promptly. Record the requested deadline, queue delay and execution duration so the allocation follows evidence.
Set a maximum attempt count and an end-to-end deadline for each logical request. Pass remaining time downstream instead of resetting the clock at every layer. Use backoff and jitter, but remember that delayed retries still consume future capacity.
A platform retry budget and a tenant retry budget should both apply. Internal retries must remain attributable to the originating tenant for fairness, even when their financial cost belongs to the operator. Attribution is not the same as charging.
Stop one tenant’s failures consuming the shared pool
Rate limits alone are insufficient. Ten requests can mean ten short completions or ten long-context generations that occupy memory and execution slots for much longer. Admission control should consider concurrent requests, queued work, context limits, maximum output and estimated resource demand.
Give each tenant bounded queue depth and bounded execution concurrency. Separate normal traffic from recovery traffic so retry bursts cannot monopolise the scheduler. Reserve only a defined share of capacity for retries, and stop admitting recovery work when that budget is exhausted.
Use circuit breakers at the appropriate scope. A broken model revision should not disable unrelated models. A tenant submitting pathological workloads should not exhaust everybody’s queue. Conversely, a pool-wide hardware fault should not be misclassified as simultaneous tenant misconduct.
Clastiq’s tenancy, quota and GPU partitioning functions provide enforcement points beneath these request policies. Depending on hardware support and workload behaviour, use partitions, dedicated devices or separate pools. Kubernetes namespaces and VM boundaries also require explicit network, storage and resource policies; labels alone do not guarantee isolation.
Storage, power and sovereignty belong in the reconciliation
Model loading and storage contention can initiate the same failure cycle as insufficient compute. A cold model pulled from Ceph may delay execution beyond a request deadline. Repeated restarts can then trigger repeated model reads and retries. Track loading separately from inference, and test storage under concurrent recovery conditions.
Store ledger events durably, with bounded local buffering during collector outages. Decide whether billing uncertainty should stop admission or allow limited operation. Silent telemetry loss followed by estimated invoices is not an acceptable default.
Power remains a facility constraint, not merely an accounting line. A recovery burst can increase concurrent activity across devices and approach rack power or cooling limits. Base admission headroom on measured electrical behaviour and facility limits, rather than treating every idle GPU as immediately usable capacity.
For sovereign or air-gapped deployments, keep usage records, identity mappings and billing exports within the approved boundary. Prompts and responses are generally unnecessary for chargeback. Define retention, access controls and deletion policies for metadata as well as content. Human-led local support should use scoped, audited access rather than unrestricted ledger exports.
What to do this week
- Write the outcome contract. Define accepted, queued, completed, cancelled, failed and expired states, including partial streaming and eligibility for credits. Version the policy before changing invoices.
- Trace one request across the stack. Carry logical request and attempt IDs from gateway to worker and settlement. Test event duplication, missing terminal events and collector recovery.
- Reconcile a full operating day. Compare token charges with measured pool GPU-hours, separating retries, loading, idle capacity and unattributed consumption. Investigate discrepancies before automating settlement.
- Contain a controlled retry storm. In a test environment, inject backend failures for one tenant and verify that queue bounds, retry budgets and circuit breakers protect another tenant’s service.
- Review physical and data boundaries. Check recovery behaviour against rack power, Ceph throughput and approved metadata locations. Assign an owner to each exception and credit decision.
Clastiq Orchestrator runs this infrastructure on the operator’s own hardware; request a demo.