Supported vLLM Versions and Models
The connector is built for vLLM 0.23, 0.27.1, 0.28.0, and 0.30.0. The deployment tools default to 0.30.0; the older versions are an explicit opt-in.
Version | Status |
|---|---|
0.30.0 |
|
0.28.0 |
|
0.27.1 |
|
0.23 |
|
vLLM 0.30.0 support, and 0.30.0 as the default, are new in 1.1.0 — a 1.0.x build does not support it. The attention-op decorator stack, preempted-output bookkeeping, and KV layout differ across these releases, and the connector detects the running engine rather than requiring a different wheel for each one.
Confirm with Lightbits Support which versions are validated for your Inferra release.
A fabric key identifies the engine's physical KV byte format, so cached prefixes written by one engine version are not readable by another and are not served to it: they miss and are recomputed. The first pass after an engine change therefore runs at cold-prefill latency until the working set is re-stored, and the old objects age out under the normal eviction policy. This is deliberate — serving another engine's blocks produces no error, no failed integrity check and normal hit rates, and only the content the model receives is wrong.
Validated Models
Validated with the connector on 8x NVIDIA H100 (Hopper) servers:
Model | Parallelism | vLLM | Validated from |
|---|---|---|---|
gpt-oss-120b | TP2 | 0.30.0 | 1.0.0 (on 0.27.1); 0.30.0 from 1.1.0 |
Qwen3.8-27B-FP8 | TP4 | 0.30.0 | 1.1.0 |
DeepSeek-V4-Flash | TP8 | 0.30.0 | 1.1.0 |
DeepSeek-V4.1-Flash | TP8 | 0.30.0 | 1.1.0 |
GLM-5.2-FP8 | TP2 x PP8, two 8-GPU nodes | 0.30.0 | 1.1.0 |
gpt-oss-20b, Qwen3.5-35B-A3B, Llama-4-Scout-17B-16E, Llama-3.1-8B, Llama-3.1-70B-FP8 | TP1 to TP8 | 0.27.1 | 1.0.0 |
It was validated on vLLM 0.27.1 at 1.0.0; select v0.27.1 for those models unless Lightbits Support confirms 0.30.0 for your deployment. For speculative decoding with any of these models, see the Speculative-Decoding Support Matrix below. GLM-5.2-FP8 spans two nodes with pipeline parallelism — contact Lightbits Support for its launch profile, and see the Release Notes' Known issues for its decode throughput on vLLM 0.30.0.
Each model otherwise uses its own standard vLLM launch flags (tokenizer, parsers, trust settings); the connector adds only the --kv-transfer-config shown in The Fetch → Install → Run Flow.
Keep the Hybrid KV-cache Manager On
Leave vLLM's hybrid KV-cache manager (HMA) at its default, on. Every validated configuration above runs with it on, and the connector handles each model's KV-cache groups itself.
Model family | What HMA off would do |
|---|---|
Gated DeltaNet hybrids (Qwen 3.5, Qwen 3.8) | They interleave linear-attention layers with full attention. With HMA on, the connector checkpoints and restores their recurrent state at a cache boundary. With HMA off it cannot, so it refuses cache reuse for the model entirely — every request is a cold prefill. Output stays correct; the acceleration is simply gone. |
Sliding-window and chunked-local models (gpt-oss, Llama-4-Scout) | They keep their windowed layers in their own groups. Turning HMA off makes vLLM allocate those layers at full length, which cuts the KV the GPU can hold. |
DeepSeek-V4 / V4.1 | They need HMA on for the engine to start at all. |
Documentation for earlier releases told you to disable HMA for every model except Qwen 3. On this release that advice is wrong: it disables reuse on Qwen 3.8 and stops DeepSeek-V4 from starting.
CUDA-graph mode
New in 1.1.0 The connector retains vLLM's native FULL_AND_PIECEWISE CUDA-graph mode — a 1.0.x connector forced the whole engine to
PIECEWISE, which cost decode throughput.
Prefill queries longer than decode's scheduled width run layerwise so KV copies can overlap model compute, while decode-shaped steps can replay FULL graphs. A prompt continuation with only one token left can also have that decode shape, so a cache hit that leaves only a decode step of prompt tokens is loaded before the request is scheduled rather than alongside its first forward. Before any FULL replay, the connector holds a KV load still in flight from an earlier step until its layers are ready, so no required per-layer wait is skipped.
No connector CUDA-graph setting is needed. A log saying the connector forced the entire engine to PIECEWISE is not expected for this release. Benchmark long-output workloads with and without the connector for your model and concurrency, but do not assume the old decode-throughput cost of a forced PIECEWISE downgrade applies.
Speculative-Decoding Support Matrix
Speculative decoding can add drafter KV layers next to the target model's own. At startup the connector reads the actual KV layout of both the target and the drafter, and from then on stores and restores the drafter's KV together with the target's.
Validated in this release
Drafting method ( | Model | Draft tokens validated |
|---|---|---|
| Qwen3.8-27B-FP8 | 1 and 3 |
This combination runs on vLLM 0.30.0 with the connector, in Lightbits' continuous validation, against the same engine without it on the same prompts — with speculative decoding off, and with one and three draft tokens — including warm hits that restore cached KV after the GPU's own prefix cache was reset.
Supported in this release
Drafting method ( | Model |
|---|---|
| DeepSeek-V4-Flash ( |
| DeepSeek-V4.1-Flash, TP8 |
| Qwen3.8-27B-FP8, with |
| Qwen3.8-27B-FP8, with |
| Qwen3.6-35B-A3B |
| GLM-4.7-Flash |
| Llama-3.1-8B, gpt-oss-20b, gpt-oss-120b |
| Llama-3.1-8B |
For each of these, Lightbits verified on GPUs, on vLLM 0.30.0, that the KV the connector restores — the drafter's included — is identical to what the engine produced. DSpark and DFlash were verified with seven draft tokens (five on DeepSeek-V4.1-Flash). ngram proposes tokens from the prompt itself and adds no drafter layer.
Run the cold-versus-warm check in Prove It's Actually Using LCF before serving it, and talk to Lightbits Support.
On a running engine, the connector logs one line per role describing the drafter layers and groups it found: