Inferra v1.2.0

This release brings speculative decoding to the connector — MTP on Qwen3.8 validated, and DSpark and DFlash parallel drafting supported, DeepSeek-V4 and V4.1 included — makes warm reads of hybrid models such as Qwen3.8 faster, adds a one-command support bundle, and introduces a preview of serving a single prompt larger than GPU memory. It also fixes several cache-reuse gaps that v1.1.0 shipped with, so multi-turn conversations and sliding-window models reuse more of what is already cached.

The comparison throughout this page is against v1.1.0, the previous official release. Everything below — highlights, performance and the Resolved list — is measured or counted against that release, not against an intermediate release candidate.

Highlights — what's new since v1.1.0

  • Speculative decoding with the connector: MTP on Qwen3.8 validated. Qwen3.8-27B-FP8 with MTP speculative decoding, at one and three draft tokens, runs with the connector and is validated against the engine without it, warm hits included. The connector reads the target's and the drafter's KV layout at startup and caches the drafter's KV together with the target's.

  • Parallel drafting: DSpark and DFlash, including DeepSeek-V4 and V4.1. DSpark on DeepSeek-V4-Flash and DeepSeek-V4.1-Flash at TP8, and DSpark and DFlash on Qwen3.8-27B-FP8, now run with the connector, which restores the drafter's KV exactly as the engine produced it. EAGLE3, n-gram drafting and MTP on further models are supported too; see the Speculative-Decoding Support Matrix in Supported vLLM Versions and Models.

  • Several vLLM replicas share one cache. Replicas of the same model bound to the same logical namespace reuse each other's cached KV, so you can scale serving out and keep the reuse. Earlier notes advised one replica per logical namespace; that advice is withdrawn.

  • One-command support bundle. lcf-cli diag collect gathers the Kubernetes, host, daemon, vLLM, lcf-api and observability state support needs into one redacted archive, and reaches the observability service by itself. See Collecting Diagnostics for Support in Troubleshooting.

  • Faster warm reads on hybrid models. A warm Gated DeltaNet hybrid now fetches the recurrent state its first forward waits on before the attention KV, in a few grouped reads instead of one per layer. On Qwen3.8-27B-FP8 a 128,000-token prompt is served from the storage tier in 230 ms (276 ms in v1.1.0), a 500,000-token prompt in 634 ms (747 ms), and the 3 TB working-set run at concurrency 8 in 613 ms (864 ms) — see Performance.

  • Multi-turn conversations reuse the whole cached prefix. In v1.1.0 the first forward of a request that reused a cached prefix was never stored, so a later turn of the same conversation could reuse only the first match and recomputed the rest on every turn. Each turn now extends what the next one can reuse.

  • Sliding-window models reuse cache behind a GPU hit. gpt-oss and Llama-4-Scout reloaded nothing from Inferra whenever part of the prompt was still in GPU memory — with the gpt-oss chat template, that was every chat request. They now reload the rest of the prefix behind the GPU-resident part.

  • Preview: prompts larger than GPU memory. VP streaming lets one request whose KV cache is larger than the engine's GPU KV capacity be served: its early KV is paged out to Inferra and streamed back, window by window, into attention. It is off by default and enabled together with Lightbits Support. Validated so far on Llama-4-Scout-17B-16E at TP8 with a 128,000-token prompt about 3.3 times the GPU KV capacity. See Prompts Larger Than GPU Memory — VP Streaming (Preview) in The Fetch → Install → Run Flow.

Performance — long-context prompts on vLLM 0.30

Measured on a server with 8x NVIDIA H100 (Hopper) GPUs running vLLM 0.30.0, the default engine. Each model runs at the tensor-parallel width shown. Every figure is engine time-to-first-token — the queue-plus-prefill time inside the serving engine, not end-to-end client latency.

Which tier serves the read

Inferra holds KV in two places: a host-RAM tier on the inference node, and the NVMe-backed storage tier behind it. Both are cache hits and both skip the prefill; the RAM tier also skips the fabric round-trip. Cold prefills the prompt in full.

At concurrency 1, working set small enough to sit in the RAM tier:

Model

Prompt

Cold

From the storage tier

From the RAM tier

Speed-up (storage / RAM)

gpt-oss-120b (TP2)

120K

3,701 ms

215 ms

164 ms

17.2x / 22.5x

Qwen3.8-27B-FP8 (TP4)

128K

4,858 ms

230 ms

207 ms

21.2x / 23.5x

DeepSeek-V4-Flash (TP8)

128K

5,267 ms

447 ms

448 ms

11.8x / 11.8x

Qwen3.8-27B-FP8 (TP4)

500K

40,411 ms

634 ms

471 ms

63.8x / 85.9x

DeepSeek-V4-Flash (TP8)

500K

33,908 ms

786 ms

771 ms

43.1x / 44.0x

In both cache columns the engine's own GPU cache is empty, so what you see is Inferra doing the work, not the GPU. The RAM tier adds most on Qwen3.8 at 500K and on gpt-oss-120b, and little on DeepSeek-V4-Flash, whose reads are nearly as fast from either tier; the storage-tier column is the figure to plan with, because it is what you get once the working set outgrows host memory.

When the working set is far beyond RAM

The case above fits in a RAM tier. The case below deliberately does not: roughly 3 TB of KV cache at concurrency 8, far past anything host memory could hold, so every read is an NVMe-backed storage-tier read.

Model

Prompt

Cold

From the storage tier

Speed-up

gpt-oss-120b (TP2)

120K

16,077 ms

679 ms

23.7x

Qwen3.8-27B-FP8 (TP4)

120K

15,251 ms

613 ms

24.9x

Two baselines

The Qwen3.8 rows of the concurrency-1 table compare against a connector-free engine; the gpt-oss-120b and DeepSeek-V4-Flash rows and the 3 TB table compare against the same engine's own cold pass, with the connector attached. Where both were measured, attaching the connector moves a cold prefill by under 1%, so the gap between the two baselines is small.

How to read these numbers. They are engine-side TTFT, specific to the models, tensor-parallel widths, prompt sizes and concurrency shown. Inferra replaces prefill work; it does not speed up decode, so output throughput moves far less than TTFT does — see Known issues for the configurations where the connector still costs some decode throughput. Benefit scales with context length: at 12,000-token prompts gpt-oss-120b gains 4.2x from the storage tier, against 17.2x at 120,000. Capture your own cold and warm figures at install time using Post-Deployment Validation; those are the baseline that makes a later regression obvious.

Upgrade notes

Upgrading from v1.0.0 or v1.1.0 needs a clean install

This release changes the format lcft keeps the cluster metadata in, and upgrading a v1.0.0 or v1.1.0 cluster in place is not supported. Uninstall the old release, remove the data lcft wrote on the hosts (the cluster metadata, cluster.redb, and the cached data under the selected mount points), then install this release. The cache starts empty, so every model runs cold until its working set has been stored again, and the tenants and namespaces you created must be created again. Install the vLLM connector wheel from this release too. See Step 9: Uninstall in LCF-CLI Deployment.

Speculative decoding

Before enabling --speculative-config, check your method and model against the Speculative-Decoding Support Matrix in Supported vLLM Versions and Models. Engines without speculative decoding are not affected.

vLLM's GPU KV cache is about 4 GiB per GPU smaller

The connector now reserves its own GPU working memory — 4 GiB per GPU worker by default — inside vLLM's --gpu-memory-utilization budget instead of on top of it, which is what prevents the out-of-memory failure listed under Resolved. At the same setting, vLLM therefore holds slightly fewer tokens in GPU memory than with v1.1.0; each worker logs the before and after figures at startup. Re-check the maximum concurrency you plan for on models whose GPU KV cache is small.

lcfc gains a privileged init container

mount-thp-tier mounts the RAM-tier directory on the host and exits. If your admission policy restricts privileged containers in the Inferra namespace, allow it; the parent directory of lcfc.ramTier.shmemHostPath (default /dev/shm) must exist on every lcfc node.

The data path no longer depends on reported link speed

Every physical interface with an IPv4 address is now a data-path candidate. If a node has addressed interfaces that must not carry Inferra traffic, list the ones that should in dataPlane.ucxNetDevices — see Choosing the data-plane NICs in LCF-CLI Deployment.

The connector's plaintext telemetry port is open to the whole cluster by default

In v1.1.0 the observability NetworkPolicy admitted plaintext OTLP (port 24317) only from the namespaces in lcf-observability.networkPolicy.otlpPlaintextFromNamespaces, which dropped the telemetry of vLLM ranks running with hostNetwork on other nodes. It now admits any in-cluster source; the port stays reachable only inside the cluster. To narrow it again, set lcf-observability.networkPolicy.otlpFromCidrs to your node CIDRs — the namespace list then applies on top of it. See Observability.

lcf-cli auto mode now uses an untainted control-plane node

In v1.1.0 a node with the control-plane label was always excluded from classification. It is now excluded only when a control-plane or master taint with a NoSchedule or NoExecute effect keeps pods off it; an untainted control-plane node is classified by its GPUs and storage like any other node. If such a node must not run Inferra, taint or cordon it, or label the nodes you want explicitly, before re-running lcf-cli deploy.

Version compatibility

Which components of this release may run against which. Check it before a rolling upgrade; the Upgrade notes above say what happens when a pair is mismatched.

Component pair

Compatible versions

If mismatched

lcfc ↔ lcft

Same release only. Both must be v1.2.0; the data path between them changed in this release.

They refuse each other's connection, and the node serves without the storage tier until both match. Upgrade them in the same rollout.

vLLM connector wheel (pylcf + connector) ↔ lcfc

Same release. Both should be v1.2.0.

Not qualified. Upgrade the connector in the same window as lcfc.

Availability

Published on Cloudsmith as v1.2.0 — container images, the Helm chart, the lcf-cli installer and the Python wheels (1.2.0). Contact Lightbits support for access.

Known issues in v1.2.0

Each item describes what you may observe, how it affects you, and the recommended action while a permanent fix is in progress. Every item listed is either reproduced on this release by our automated validation suite or backed by an open defect; each entry's Status says which. Issues that were neither are not carried forward.

1. Decode throughput is lower on two configurations on vLLM 0.30

  • What you may see — With the connector attached, decode is about 1% to 6% slower on Qwen3.8-27B-FP8, depending on prompt length, and about 11% to 13% slower on GLM-5.2-FP8 with pipeline parallelism. gpt-oss-120b, DeepSeek-V4-Flash and DeepSeek-V4.1-Flash decode within 2% of the engine without the connector.

  • Impact — Decode throughput only. Time-to-first-token and output are unaffected. The GLM case comes from vLLM 0.30 itself, which slows every pipeline rank once any KV connector is attached.

  • Recommended action — None for most workloads: the time-to-first-token gain on long prompts is far larger. If decode throughput is your binding constraint, measure with your own output lengths and talk to Lightbits support.

  • Status — Confirmed by CI Reproduced on this release by the automated validation suite.

2. cache_salt does not separate requests in the Inferra cache

  • What you may see — vLLM's per-request cache_salt keeps differently salted requests from sharing the engine's GPU prefix cache, but the connector does not carry the salt into its keys, so a salted request can be served KV from Inferra that another request stored for the same prompt tokens.

  • Impact — Output is correct; what is lost is isolation between salts.

  • Recommended action — Serve workloads that must not share cached KV from engines bound to separate logical namespaces.

  • Status — Open defect Not covered by the automated suite that validates each release.

Resolved since v1.1.0

Scoped to the previous official release: everything below was open in v1.1.0 and is closed in this one. Items that were opened and closed entirely between the two — within an intermediate release candidate — are not listed, because no customer ran a release that had them.

  • Hybrid models with MTP speculative decoding returned wrong output after a cache hit — on Gated DeltaNet hybrids such as Qwen3.6 with MTP on, a request that reused a cached prefix could answer with empty, wrong or runaway output. The restored recurrent state was placed in a scratch slot that speculative decoding uses, instead of the slot the engine reads; with MTP off, output was correct. Fixed, and the connector now checks speculative decoding at startup — see Highlights.

  • A later conversation turn reused less of a cached prefix than it should — the first forward of a request that reused a cached prefix was never stored, so every later lookup stopped at the first match and the rest was recomputed on each turn. Performance only; output was correct. Fixed.

  • Sliding-window models reloaded nothing when part of a prompt was in GPU memory — gpt-oss and Llama-4-Scout declined to reload from Inferra whenever the engine held part of the prompt locally, and a declined lookup was left running, so later requests for the same prompt inherited its "not found". With the gpt-oss chat template, chat requests reused nothing. Performance only. Fixed.

  • The engine could run out of GPU memory under a wave of concurrent long prompts — the connector's device staging buffers were allocated after vLLM had sized its KV cache to fill the GPU memory budget, so they competed with the activations of large prefills. They are now reserved inside that budget; see Upgrade notes. Fixed.

  • A RAM tier could take a node's memory twice — on a node with a huge-page reservation that lcfc could not use, the fallback tier was sized as if the reserved memory were still free, and could push the node into an out-of-memory condition. lcfc now leaves the whole reservation out, and refuses to start with an error naming both figures when nothing is left. Fixed.

  • Core dumps carried the RAM tier — a core dump of lcfc, or of a vLLM process with the connector attached, could include the RAM tier's KV payload, many GiB of it. The tier is now excluded from core dumps. Fixed.

  • A network interface that reported no link speed left the storage tier without devices — v1.1.0 Known issues item 3. On a node without RDMA whose interface reported no link speed, typical of virtual machines with a virtio network interface, lcft registered no devices and the node silently had no storage tier. Every physical interface with an IPv4 address is now used for the data path, whatever its reported speed, and the ethtool workaround is no longer needed. Fixed.