An October showcase, not a release milestone
Janas appeared in Hacker News’s October 13, 2026 archive as an inference engine that streams model experts from SSD. The public showcase brought a specific infrastructure trade-off into view: reduce the requirement to keep expert weights in GPU memory by fetching them through the storage and host-transfer path. October 13 is the dated community appearance used here, not a verified first-release date.
The evidence deserves proportionate weight. Hacker News is a discovery signal, not a performance validation service. The linked repository is the place to examine implementation details and developer-reported comparisons, but those comparisons require reproduction before they support a capacity purchase or tenant commitment. This article does not present an independently verified Janas benchmark.
For operators with a quarter rack to a few racks, the question is whether that trade creates useful capacity without undermining existing services. A model becoming loadable is only the first gate. Useful throughput, predictable completion time and isolation from neighbouring workloads determine whether it belongs in production.
The bottleneck moves beyond GPU memory
Mixture-of-experts models select subsets of expert computation for each token. That creates an opportunity to retain frequently used experts in faster memory while fetching others when needed. The attraction is straightforward: the complete expert set may exceed available accelerator memory even when the active computation for a token is comparatively limited.
The operational consequence is less straightforward. A cache miss can introduce an SSD read, movement through host memory, a CPU-to-GPU transfer and scheduling delay before useful accelerator work resumes. Implementations may overlap some of these stages, but operators must establish how much overlap survives their workload. Available SSD bandwidth alone does not establish inference performance.
On a small estate, these paths often share resources with ordinary work. The same host may run VM disks, Kubernetes image pulls, storage recovery and another inference tenant. A benchmark conducted on an idle server can conceal the contention that appears when those services operate together.
GPU partitioning can help constrain accelerator resources where supported. It does not automatically partition host memory bandwidth, CPU scheduling, PCIe capacity or SSD queues. An offloaded tenant can remain inside its assigned GPU allocation while causing another tenant to miss its response-time objective through a shared host bottleneck.
The placement unit therefore needs to include a resource path, not just a GPU count: accelerator allocation, host memory, CPU locality, SSD service capacity and transfer headroom. Treat expert offloading as a distinct workload class until measurements justify combining it with existing tenants.
Build a benchmark that exposes misses
Start by freezing the test inputs. Record the Janas commit, model revision, weight format, quantisation, driver stack, runtime configuration and storage layout. Capture host memory capacity, accelerator memory capacity and relevant device topology. Reproducibility matters more than choosing the most flattering prompt collection.
Use three operating conditions: a deliberately cold start, a warmed expert cache and a mixed workload that changes routing behaviour. Document exactly which caches are cold. Restarting a process does not necessarily evict the operating system’s page cache; a supposed SSD experiment may actually measure host RAM.
For every condition, record time to first token, inter-token latency, completed output tokens per second and total job duration. Report tail behaviour as well as medians. Interactive tenants experience pauses, while batch operators care about completion deadlines and the number of successful jobs delivered during the scheduling window.
Measure expert-cache hits and misses directly where instrumentation permits. If the engine does not expose them, say so and use correlated storage reads, transfer traces and latency as indirect evidence. Do not label inferred events as measured misses. Distinguish bytes requested by the engine from bytes actually read from physical storage.
| Measurement | What it tests | Warning sign |
|---|---|---|
| Expert-cache miss rate | Whether reuse limits fetching | Misses rise with request diversity |
| SSD bandwidth and read latency | Whether storage serves fetches promptly | Queues grow under concurrency |
| Host-to-device traffic | Whether transfers delay execution | Transfer bursts align with stalls |
| Neighbour p99 latency | Whether tenancy remains isolated | Existing service objective is breached |
| Energy per completed job | Whether slower execution remains economical | Longer runtime outweighs memory savings |
Then increase concurrent requests in controlled steps. Keep prompt lengths and output limits documented, and test realistic variation rather than only identical requests. Repeated prompts may overstate cache locality. Include the long-context jobs and changing request mixes that your own users actually submit.
A short host-side capture can establish the first correlation between latency and storage pressure:
# Run alongside a fixed, documented inference workload.
mkdir -p measurements
iostat -x -y 1 180 > measurements/block-io.txt &
vmstat -w 1 180 > measurements/host-memory.txt &
wait
These commands are Linux starting points, not a complete benchmark. Add engine timing, accelerator telemetry and hardware-appropriate transfer counters. Check device latency, queue depth, swap activity and CPU pressure. A single device-utilisation percentage is not a reliable saturation verdict, especially for parallel SSD devices.
Test the neighbours, not just the engine
Run the same benchmark with a representative interactive tenant on the host. Establish that tenant’s baseline before starting offloading, then compare its tail latency and throughput during cold loading, steady inference and request bursts. The relevant result is the change experienced by the neighbour, not just Janas’s own output rate.
Repeat with storage activity that the estate cannot avoid: VM reads, image distribution or a controlled recovery exercise. Do not launch destructive stress tests on production storage. Use a maintenance window or a representative test pool, and preserve a stop condition if service latency crosses its agreed limit.
Watch for feedback loops. A slower storage path can lengthen requests, increasing the number of requests resident simultaneously. That can enlarge working sets and raise cache pressure further. An admission limit based only on available GPU memory may admit more work precisely when the shared path needs relief.
Place the ceiling where the entire system remains stable. If eight simultaneous requests meet the deadline but sixteen cause queue growth that does not recover between bursts, sixteen is not saleable capacity. Test cancellation and recovery too: aborted requests must release allocations and avoid leaving a transfer backlog that penalises the next tenant.
A worked capacity and chargeback example
Consider an illustrative estate with four GPUs and a twelve-hour overnight batch window. It has 48 GPU-hours of reservation capacity each night. These are hypothetical planning numbers, not Janas results or Cognition-AI benchmarks. Assume a private batch job occupies one whole GPU allocation throughout its run.
Suppose local testing finds that offloading lets the chosen model run on that allocation and complete in six hours. The isolated host sustains two jobs sequentially per GPU, yielding eight completed jobs across the estate. Each night therefore consumes 48 reserved GPU-hours. This describes allocation occupancy, not measured GPU compute utilisation.
Now assume the operator’s internal allocation rate is $1.50 per GPU-hour, excluding incremental offload-storage charges. Each job incurs $9.00 for its six GPU-hours. If the measured incremental SSD and host-service allowance is priced at $0.25 per job-hour, another $1.50 brings the internal charge to $10.50 per completed job, or $84 for eight jobs.
For a storage-path check, suppose each job reads an average of 1.2 GB/s from SSD during its six-hour run. That is 25,920 GB, or 25.92 TB in decimal units, per job. Four concurrent jobs demand an average 4.8 GB/s across their storage paths; eight jobs account for 207.36 TB of nightly reads. Bursts may be higher than these averages.
Those read volumes are not equivalent to SSD write endurance consumption. They do, however, expose how much service bandwidth the deployment depends on. Model staging, cache rewriting and unrelated writes require separate accounting. If the storage layout cannot supply the measured concurrent demand with latency headroom, the eight-job plan is invalid.
Finally, suppose shared-host contention extends each job to eight hours. Only four complete within the twelve-hour window without carrying work beyond it. At the same rates, each completed job costs $12.00 in GPU allocation plus $2.00 in incremental service allowance: $14.00. Lower memory requirements have not guaranteed lower delivered-job cost or preserved overnight capacity.
Make acceptance a service-class decision
Use separate acceptance gates for private batch and interactive service. Offloading is acceptable for a batch class when the model produces acceptable outputs, tested worst-case completion fits its deadline, delivered-job cost meets budget, and the workload does not breach neighbouring service objectives. A single failed gate is enough to reject that placement.
For example, an operator might require the measured high-percentile batch completion time to remain below ten hours inside a twelve-hour window, leaving recovery margin. The threshold is a local policy, not a Janas performance claim. Test it under cold-cache conditions and the concurrent workload permitted by the admission policy.
For a latency-sensitive tenant, start with its actual response-time contract. If expert fetches cause time-to-first-token or inter-token latency to exceed that contract under an allowed workload, reject offloading for that class. A good average cannot compensate for failed tail-latency commitments. Test the tenant itself and its neighbours against their respective objectives.
Prefer a conservative operating rule: reserve offloaded inference for isolated, deadline-tolerant private batch pools until concurrency and interference tests pass. Where isolation is unavailable, schedule exclusive host windows. Revisit the decision after model, runtime, firmware or storage changes; the accepted configuration is a measured combination, not a permanent property of the engine.
Turn the result into enforceable infrastructure policy
A system such as Clastiq Orchestrator provides the operational framework around this decision: bare-metal provisioning, Kubernetes and VM tenancy, GPU partitioning, Ceph storage, per-tenant metering and chargeback, and quota and policy engineering. The scheduler and policies still need workload-specific engineering. None of those capabilities should be mistaken for a pre-existing Janas integration.
Provision a reproducible host configuration and define a batch placement class with explicit eligibility. Labels or equivalent inventory attributes should describe tested storage locality and resource availability. Request CPU and host-memory allocations alongside GPUs. Configure placement so an apparently free accelerator does not attract offloaded work onto a host whose storage path is already committed.
Keep authoritative model storage separate from the inference hot path where that improves predictability. Ceph can hold model artefacts and tenant data; validated model copies can be staged onto local SSDs before the batch window. Alternatively, test a shared-storage design directly. Neither arrangement is automatically superior: replication, network traffic, recovery behaviour and local capacity determine the result.
Meter the resources that explain costs. GPU allocation hours remain useful, but offloading also warrants storage-read bytes, job duration, host-memory reservation and completed-job counts. Where available, record energy and shared-path pressure. Define which measurements are billable and which are diagnostic; otherwise tenants may receive a chargeback invoice they cannot reconcile with their submitted work.
Quota engineering should cover concurrent offloaded jobs and model-staging capacity, not merely GPU count. Device-level I/O controls may help, but their effectiveness depends on the runtime and storage stack. Verify enforcement under contention. When isolation cannot be demonstrated, dedicate hosts or storage paths rather than promising protection that partitioning alone cannot deliver.
Power policy also needs a whole-node view. GPU power may fall during transfer stalls while CPU, memory and SSD activity continue. Longer runtimes can increase energy per completed job even when instantaneous accelerator power is lower. Measure at an appropriate boundary and leave capacity for normal cooling and facility constraints; memory fit is not a power budget.
Keep private inference private through the entire path
Open-source availability does not by itself establish sovereign operation. Review model licences, dependencies, outbound telemetry and artefact provenance before admitting the engine. Pin the tested software and model versions, retain hashes, and make the deployment reproducible without assuming continued access to a public repository or model endpoint.
An air-gapped deployment requires more than moving weights onto a server. Stage runtime packages, validate their provenance, define an update process and test startup without network access. Keep prompts, generated outputs and diagnostic traces within the intended trust boundary. Performance logs can contain tenant identifiers or prompt fragments and need the same deliberate handling as other operational data.
Clastiq Orchestrator’s air-gap deployment scope and human-led local support are relevant to maintaining that boundary on owned hardware. They do not replace an engine security review or prove a particular configuration compliant. Cognition-AI, based in Muscat, Oman, should be understood here as the infrastructure platform provider, not as the source of an independently validated Janas speed claim.
What to do this week
- Pin a Janas commit and one permitted model revision. Record the October 13, 2026 showcase as a discovery reference, not a release milestone or benchmark endorsement.
- Run cold-cache, warm-cache and mixed-request tests on one representative host. Capture completion time, tail latency, physical reads and transfer activity.
- Add one interactive neighbour and one realistic storage workload. Set stop conditions and identify a concurrency limit that preserves both services.
- Calculate cost per completed job and nightly completion capacity, including GPU reservations, incremental storage service and measured energy where available.
- Create an isolated batch placement policy with staging quotas, admission limits and a rollback path before offering the workload to tenants.
Clastiq Orchestrator runs this infrastructure on the operator’s own hardware — request a demo.