Mistral Large 4’s Active Parameters Are Not a VRAM Budget

October 6 changed the planning question

Mistral announced the public API preview of Mistral Large 4 on October 6, 2026, describing approximately one trillion total parameters and 49 billion active parameters. Downloadable weights were promised for later October; they were not available at announcement. For operators considering local deployment, that distinction separates an evaluation opportunity from a deployable service.

An API preview lets teams investigate application behaviour, subject to their data policies. It does not establish local memory requirements, supported inference engines, quantisation quality, redistribution rights or production throughput. Those decisions require the released checkpoint, licence, architecture configuration and a tested serving implementation. An open-weight deployment plan should therefore remain conditional, not become a capacity commitment.

The immediate infrastructure trap is treating 49 billion active parameters as the model’s VRAM requirement. In a mixture-of-experts model, active parameters describe the subset engaged during computation, not the complete set of weights the serving system must make accessible. Conditional computation can reduce work per token without reducing resident weights to the active subset.

For a quarter-rack operator, the question is not simply whether the model can start. It is whether serving it leaves a functioning estate for other tenants, maintenance and failure recovery.

Start with total weights, not active compute

Using one trillion parameters as a planning approximation, raw weight storage is roughly 2,000 GB at 16 bits, 1,000 GB at eight bits, and 500 GB at four bits. These figures use decimal GB and exclude everything except parameter values. Inventory actual device capacity in bytes before comparing it with these estimates.

Quantised checkpoints also need scales, metadata and sometimes higher-precision tensors. Engines may unpack weights, create temporary buffers or retain additional representations. A four-bit label therefore does not guarantee a 500 GB resident footprint. Nor does a checkpoint’s size on disk prove its GPU memory use after loading.

Expert offload changes the calculation but does not erase it. Moving weights through host memory or storage adds bandwidth demand and latency exposure. Whether that trade works depends on routing, locality, batching, engine support and service objectives. Treat offload as a separate deployment design requiring measurement, not a convenient subtraction from VRAM.

The initial worksheet should capture the exact checkpoint revision, precision, quantisation format and supported engine version. If any is unknown, mark the entry provisional. An operator should be able to distinguish a measured number from a vendor statement or an internal assumption without reading the whole planning document.

Build a memory ledger before booking GPUs

Separate four categories: resident weights, runtime overhead, KV cache and unallocated safety reserve. Keep runtime overhead distinct from reserve: overhead is memory the service expects to consume; reserve protects against uncertainty and allocation spikes.

The following example is illustrative, not a Mistral Large 4 measurement. Assume a proposed serving group with sixteen GPUs, each offering 80 GB of usable capacity under the worksheet’s decimal convention. Aggregate capacity is 1,280 GB.

The allowance is deliberately explicit: 20% of raw weight storage, pending measurement. It is not a claim about this model or an accepted universal sizing rule. The safety reserve is 10% of device capacity. Under these assumptions, four-bit serving has 392 GB of remaining aggregate capacity; eight-bit serving exceeds capacity by 208 GB.

A conditional fit is not approval. Aggregate memory hides uneven shards, replicated tensors and device-local workspaces. Every GPU must pass its own peak-memory check. Sixteen devices with adequate total memory can still fail because one rank receives an oversized shard or a temporary allocation lands on the busiest device.

Use a small calculation to make assumptions auditable:

# Decimal GB; replace assumptions with measured values.
parameters = 1_000_000_000_000
bits = 4
gpus, gb_per_gpu = 16, 80
weights_gb = parameters * bits / 8 / 1e9
runtime_gb = weights_gb * 0.20
kv_gb = 160
reserve_gb = gpus * gb_per_gpu * 0.10
required_gb = weights_gb + runtime_gb + kv_gb + reserve_gb
print(f"Required: {required_gb:.0f} GB")
print(f"Capacity: {gpus * gb_per_gpu} GB")

Record both aggregate and worst-rank results after loading the actual checkpoint. Reject configurations that require spending the safety reserve just to reach idle readiness.

Allocate KV cache from admitted demand

Weights are only the starting point. KV cache grows with retained tokens and concurrent sequences. A service that loads successfully at idle can fail when several tenants submit long documents together.

For a conventional attention implementation, a useful approximation is: two tensors multiplied by layer count, KV-head count, head dimension, bytes per cache element and retained token count. Actual architectures may use different cache structures, attention windows or compression. Do not substitute attention-query heads for KV heads, and do not assume the formula describes Large 4 before inspecting its configuration.

For an arithmetic example only, assume 80 layers, eight KV heads, head dimension 128 and two-byte cache elements. Each retained token then consumes 327,680 bytes: 2 × 80 × 8 × 128 × 2. A sequence retaining 32,768 tokens consumes approximately 10.74 GB. Twelve such sequences consume approximately 128.85 GB; adding 20% for allocation effects gives approximately 154.62 GB.

That hypothetical workload fits the worksheet’s 160 GB cache allocation narrowly. It does not prove that twelve Large 4 requests fit. It demonstrates why concurrency needs a token budget rather than a request-count limit alone. Retained context includes generated tokens, not merely the original prompt.

Set limits for input length, output length, active sequences and aggregate retained tokens. Account for cache replication and sharding in the chosen engine. Test mixed short and long requests, cancellation, cache reclamation and prefix-sharing behaviour. A promised context window is a model capability; the context capacity sold to tenants is an infrastructure allocation.

Make interconnect a pass-or-fail gate

Memory capacity distributed across racks is not automatically usable serving capacity. Tensor, pipeline and expert parallelism impose different communication patterns. Expert routing can create token exchanges between devices, while other parallel layouts introduce collective operations or pipeline transfers.

Before selecting a layout, draw the physical topology: devices within each host, local links, network adapters, switch paths and oversubscribed uplinks. Record which resources other tenants share. A scheduler that counts free GPUs without considering placement can assemble a group that fits in memory but misses latency targets.

There is no defensible universal network-speed requirement from parameter count alone. Test the supported parallel strategy on the proposed hardware. Measure time to first token, inter-token latency and tail latency under realistic concurrency, not only aggregate tokens per second. Include long-prefill bursts, which can interfere with ongoing decoding.

Require a placement rule that keeps the serving group on an approved topology. If the model needs multiple hosts, verify the communication transport, collective libraries and failure behaviour before admission. More GPUs can introduce additional communication and replication, so expanding the group is not guaranteed to improve useful throughput.

For a small estate, losing one host may remove the entire distributed replica. If the remaining inventory cannot reconstruct the serving group, document the service as having no equivalent local failover. Do not describe spare individual GPUs as redundancy when the model requires a complete coordinated set.

Decide whether one model deserves the estate

Consider an operator with 24 GPUs. Reserving sixteen for this model consumes two-thirds of the estate, leaving eight for other inference, development and VM tenants. That reservation may be reasonable for a committed workload. It is harder to justify for occasional experimentation.

Over a 30-day month, the sixteen-GPU reservation represents 11,520 GPU-hours: 16 × 24 × 30. At an illustrative fully loaded internal cost of $2.50 per reserved GPU-hour, the monthly allocation costs $28,800. This is an accounting assumption, not a market price or a Cognition-AI benchmark.

If only 35% of reserved time is classified as productive serving time, that yields 4,032 productive GPU-hours. Recovering the full reservation cost requires approximately $7.14 per productive GPU-hour: $28,800 divided by 4,032. At 70% productive occupancy, the corresponding figure falls to approximately $3.57.

These figures describe reservation economics, not hardware-counter utilisation or cost per token. A latency-sensitive service may need idle headroom; that headroom still has a cost. Define productive occupancy consistently, then measure accepted tokens and service-level compliance separately.

A per-tenant ledger should distinguish a capacity retainer from variable consumption. Reserved capacity can be charged by allocation time; shared inference can be attributed using input tokens, output tokens, cache occupancy or another documented tariff. Token counts alone do not capture every cost difference between workloads.

The rejection test is operational: if the model displaces services with stronger demand, prevents maintenance or leaves no viable recovery path, reject the standing reservation. Alternatives include a smaller local model, scheduled evaluation windows or an external API for workloads permitted to leave the environment. Sovereignty requirements may rule out that last option.

Turn the worksheet into enforceable tenancy

A spreadsheet does not stop a tenant from exhausting cache or occupying a required host. The approved plan must become resource assignments, admission limits and billing records.

Clastiq Orchestrator covers bare-metal provisioning, Kubernetes and VM tenancy, GPU partitioning, Ceph storage, per-tenant metering and chargeback, and quota and policy engineering. Those functions provide the infrastructure controls around a serving stack; they do not establish compatibility with an unreleased checkpoint or replace inference-engine validation.

Use bare-metal provisioning to establish repeatable host configurations. Place the distributed serving group in a dedicated resource pool, with tenancy policies preventing unrelated allocations from breaking its topology. Kubernetes or VM tenancy should reflect the isolation boundary the operator actually needs, rather than treating all model access as identical.

GPU partitioning can preserve capacity for smaller workloads where hardware and runtime support it. It cannot turn an oversized model into a small one, and partition boundaries may restrict memory capacity or communication options. Validate the partition layout rather than assuming slices are interchangeable with full devices.

For shared inference, connect infrastructure quotas to serving-layer controls. Per-tenant GPU allocation does not automatically enforce per-tenant retained-token limits. The gateway and inference service need admission policies, queue limits and request attribution, with their records reconciled to the operator’s metering system.

Protect both fairness and availability: one tenant’s long prompts should not consume the entire cache allocation or starve short interactive requests. Apply separate service classes where necessary, and make rejection or queueing behaviour explicit in tenant agreements.

Include storage, power and sovereignty

A 500 GB raw four-bit estimate is also a storage and startup problem. The estate may need original checkpoints, converted artifacts, rollback versions and temporary conversion space. Ceph capacity planning must include its protection overhead and recovery headroom, not just the apparent size of one model directory.

Measure cold-start time from artifact retrieval through service readiness. Concurrent starts can stress storage and network links already used by tenant volumes. Consider controlled staging to local storage where appropriate, while preserving artifact checksums, access controls and revision tracking. GPU memory fit is irrelevant if recovery takes longer than the promised outage window.

Power planning should use measured server draw during loading, prefill and sustained decoding, including CPU, memory, storage and networking. Check rack feeds, facility limits and cooling capacity before approving the reservation. Allocating GPUs changes workload intensity; it does not create electrical headroom.

For GCC operators, including estates in Oman, sovereignty means documenting where prompts, outputs, logs, backups and support artifacts travel. A public API preview is not a sovereign local deployment. Local weights alone also do not guarantee sovereignty if telemetry or support bundles cross the approved boundary.

Cognition-AI is based in Muscat, Oman, and Clastiq Orchestrator includes air-gap deployments and human-led local support. An air-gap plan still needs controlled artifact import, dependency availability, licence review and an update process. Support procedures should specify permitted diagnostics and data handling rather than assume that local support automatically resolves every governance requirement.

What to do this week

1. Freeze the evidence sheet. Record the October 6 API-preview status, pending weights, source URL and all unresolved licence, architecture and runtime questions. Do not book production capacity against an assumed release artifact.
2. Complete the memory ledger. Calculate weight precision, measured format overhead, per-rank runtime peaks, cache allocation and reserve. Label every unmeasured value as an assumption.
3. Define an admission envelope. Specify tenant concurrency, retained-token budgets, output caps and latency targets. Prepare tests that combine long prefills with ongoing interactive decoding.
4. Validate the physical estate. Map interconnect topology, storage startup paths, rack power and cooling. Identify whether a host failure leaves enough compatible capacity to restart the whole replica.
5. Approve or reject the reservation. Price the complete GPU group, displaced workloads and recovery limitations. Choose a standing service only when committed demand justifies it; otherwise schedule a bounded evaluation.

Clastiq Orchestrator runs this infrastructure on the operator’s own hardware: request a demo.

Sources

• Mistral Large 4 announcement