Supported vLLM Versions and Models

The connector is built for vLLM 0.23, 0.27.1, 0.28.0, and 0.30.0. The deployment tools default to 0.30.0; the older versions are an explicit opt-in.

Version

Status

0.30.0

Default New in 1.1.0

0.28.0

Opt-in

0.27.1

Opt-in

0.23

Opt-in

vLLM 0.30.0 support, and 0.30.0 as the default, are new in 1.1.0 — a 1.0.x build does not support it. The attention-op decorator stack, preempted-output bookkeeping, and KV layout differ across these releases, and the connector detects the running engine rather than requiring a different wheel for each one.

Confirm with support

Confirm with Lightbits Support which versions are validated for your Inferra release.

Changing the vLLM version on a running cluster cold-starts the KV fabric

A fabric key identifies the engine's physical KV byte format, so cached prefixes written by one engine version are not readable by another and are not served to it: they miss and are recomputed. The first pass after an engine change therefore runs at cold-prefill latency until the working set is re-stored, and the old objects age out under the normal eviction policy. This is deliberate — serving another engine's blocks produces no error, no failed integrity check and normal hit rates, and only the content the model receives is wrong.

Validated Models

Validated with the connector on 8x NVIDIA H100 (Hopper) servers:

Model

Parallelism

vLLM

Validated from

gpt-oss-120b

TP2

0.30.0

1.0.0 (on 0.27.1); 0.30.0 from 1.1.0

Qwen3.8-27B-FP8

TP4

0.30.0

1.1.0

DeepSeek-V4-Flash

TP8

0.30.0

1.1.0

DeepSeek-V4.1-Flash

TP8

0.30.0

1.1.0

GLM-5.2-FP8

TP2 x PP8, two 8-GPU nodes

0.30.0

1.1.0

gpt-oss-20b, Qwen3.5-35B-A3B, Llama-4-Scout-17B-16E, Llama-3.1-8B, Llama-3.1-70B-FP8

TP1 to TP8

0.27.1

1.0.0

The last row has not been re-measured on vLLM 0.30.0

It was validated on vLLM 0.27.1 at 1.0.0; select v0.27.1 for those models unless Lightbits Support confirms 0.30.0 for your deployment. For speculative decoding with any of these models, see the Speculative-Decoding Support Matrix below. GLM-5.2-FP8 spans two nodes with pipeline parallelism — contact Lightbits Support for its launch profile, and see the Release Notes' Known issues for its decode throughput on vLLM 0.30.0.

Each model otherwise uses its own standard vLLM launch flags (tokenizer, parsers, trust settings); the connector adds only the --kv-transfer-config shown in The Fetch → Install → Run Flow.

Keep the Hybrid KV-cache Manager On

Do not pass --disable-hybrid-kv-cache-manager, for any model

Leave vLLM's hybrid KV-cache manager (HMA) at its default, on. Every validated configuration above runs with it on, and the connector handles each model's KV-cache groups itself.

Model family

What HMA off would do

Gated DeltaNet hybrids (Qwen 3.5, Qwen 3.8)

They interleave linear-attention layers with full attention. With HMA on, the connector checkpoints and restores their recurrent state at a cache boundary. With HMA off it cannot, so it refuses cache reuse for the model entirely — every request is a cold prefill. Output stays correct; the acceleration is simply gone.

Sliding-window and chunked-local models (gpt-oss, Llama-4-Scout)

They keep their windowed layers in their own groups. Turning HMA off makes vLLM allocate those layers at full length, which cuts the KV the GPU can hold.

DeepSeek-V4 / V4.1

They need HMA on for the engine to start at all.

This reverses the advice for earlier releases

Documentation for earlier releases told you to disable HMA for every model except Qwen 3. On this release that advice is wrong: it disables reuse on Qwen 3.8 and stops DeepSeek-V4 from starting.

CUDA-graph mode

New in 1.1.0 The connector retains vLLM's native FULL_AND_PIECEWISE CUDA-graph mode — a 1.0.x connector forced the whole engine to PIECEWISE, which cost decode throughput.

Prefill queries longer than decode's scheduled width run layerwise so KV copies can overlap model compute, while decode-shaped steps can replay FULL graphs. A prompt continuation with only one token left can also have that decode shape, so a cache hit that leaves only a decode step of prompt tokens is loaded before the request is scheduled rather than alongside its first forward. Before any FULL replay, the connector holds a KV load still in flight from an earlier step until its layers are ready, so no required per-layer wait is skipped.

No connector CUDA-graph setting is needed. A log saying the connector forced the entire engine to PIECEWISE is not expected for this release. Benchmark long-output workloads with and without the connector for your model and concurrency, but do not assume the old decode-throughput cost of a forced PIECEWISE downgrade applies.

Speculative-Decoding Support Matrix

Speculative decoding can add drafter KV layers next to the target model's own. At startup the connector reads the actual KV layout of both the target and the drafter, and from then on stores and restores the drafter's KV together with the target's.

Validated in this release

Drafting method (method)

Model

Draft tokens validated

mtp

Qwen3.8-27B-FP8

1 and 3

This combination runs on vLLM 0.30.0 with the connector, in Lightbits' continuous validation, against the same engine without it on the same prompts — with speculative decoding off, and with one and three draft tokens — including warm hits that restore cached KV after the GPU's own prefix cache was reset.

Supported in this release

Drafting method (method)

Model

dspark

DeepSeek-V4-Flash (deepseek-ai/DeepSeek-V4-Flash-DSpark), TP8

dspark

DeepSeek-V4.1-Flash, TP8

dspark

Qwen3.8-27B-FP8, with RedHatAI/Qwen3.8-27B-speculator.dspark

dflash

Qwen3.8-27B-FP8, with incoai/Qwen3.8-27B-DFlash2

mtp

Qwen3.6-35B-A3B

mtp

GLM-4.7-Flash

eagle3

Llama-3.1-8B, gpt-oss-20b, gpt-oss-120b

ngram

Llama-3.1-8B

For each of these, Lightbits verified on GPUs, on vLLM 0.30.0, that the KV the connector restores — the drafter's included — is identical to what the engine produced. DSpark and DFlash were verified with seven draft tokens (five on DeepSeek-V4.1-Flash). ngram proposes tokens from the prompt itself and adds no drafter layer.

A configuration in neither table

Run the cold-versus-warm check in Prove It's Actually Using LCF before serving it, and talk to Lightbits Support.

On a running engine, the connector logs one line per role describing the drafter layers and groups it found:

kubectl -n <vllm-namespace> logs deploy/<vllm-deployment> -c vllm | grep 'LCF spec-decode geometry'