The Fetch → Install → Run Flow

The engine runs the stock vllm/vllm-openai:v0.30.0 image by default. Select v0.23.0, v0.27.1 or v0.28.0 explicitly when your validated deployment requires it.

There is no custom Lightbits vLLM image and no LCF entrypoint wrapper. The connector is installed into that stock image at pod startup by a shell shim that runs as the container's command, then execs the engine:

command: ["/bin/sh", "-c", "<install the two wheels> && exec \"$@\"", "--"]
args: ["vllm", "serve", "<model-path>", "…"]

The install step is whichever option below you chose:

pip install --index-url "$LCF_PIP_INDEX_URL" \
"lcf-vllm-connector==x.y.z" "pylcf==x.y.z" && exec "$@"
pip install --no-deps --force-reinstall \
/lcf-wheels/pylcf-*.whl /lcf-wheels/lcf_vllm_connector-*.whl && exec "$@"
There is no separate import step

vLLM resolves the connector class from kv_connector_module_path in --kv-transfer-config, so installing the wheels is all that is required to register LcfVllmConnectorV1.

To deploy a specific version, change which wheel version the install step pins and restart the pod.

Get the Wheels into the Pod

The connector is not a separate deployment

It is the two wheels injected into the one vLLM serving pod. So "deploying the connector" = deploying (or updating) the single LCF vLLM Deployment — the vLLM engine and the connector always ship and restart together. There is no second workload to install.

Which path applies to you

On a released build, use Option B (Cloudsmith pip index) or Option C (bake the wheels into your own image). Those are the two paths the release ships wheels for, and Option B is what your Cloudsmith entitlement token grants you.

Option A is not available on a released build. It needs an lcf-vllm-plugin-wheels OCI image, and the release does not publish one to Cloudsmith — that image is built only for internal development builds, in a registry your entitlement token cannot reach. It is documented below for readers working with an internal build; skip it otherwise.

The supported path for a released build. The release publishes lcf-vllm-connector and pylcf as wheels on the Cloudsmith Python index, and your entitlement token grants access to it. Inside the pod:

pip install \
--index-url "https://dl.lightbitslabs.com/${CLOUDSMITH_TOKEN}/lightbits-inferra/python/simple/" \
"lcf-vllm-connector==x.y.z" "pylcf==x.y.z"

The index URL uses the same Cloudsmith host/path as the downloads in the Deployment and Usage Guide (dl.lightbitslabs.com/${CLOUDSMITH_TOKEN}/lightbits-inferra/...). Confirm the exact Python index URL with Lightbits Support if pip cannot find the packages.

That URL is a credential

The entitlement token is inside the path, so the index URL is as sensitive as the token itself. Put it in a Secret and reference it — never inline it in the Deployment, and never commit a manifest that contains it:

kubectl -n <vllm-namespace> create secret generic lcf-pip-index \
--from-literal=url="https://dl.lightbitslabs.com/${CLOUDSMITH_TOKEN}/lightbits-inferra/python/simple/"
env:
- name: LCF_PIP_INDEX_URL
valueFrom:
secretKeyRef: { name: lcf-pip-index, key: url }

The launch shim then uses $LCF_PIP_INDEX_URL and the token never appears in the pod spec, in kubectl describe, or in your version control. Note that pip may still echo the index URL in its output on an error, so treat the pod's logs as sensitive too. Option C avoids the problem entirely: the wheels are already in the image and the pod needs no index access at all.

A manual pip install does not survive a restart

A manual pip install inside a running pod does not survive a restart — the launch shim reinstalls on every start, overwriting it. To make Option B persist, put the pip install in the pod's launch shim so it runs on every start, or stage the wheels into /lcf-wheels from an init container (pip-download from the index) and have the shim install from there.

Bake the wheels + config into a custom image. Instead of injecting at runtime, build a supported vLLM image that already has lcf-vllm-connector/pylcf installed and the connector env + --kv-transfer-config baked in.



Gains

An immutable, reproducible, fast-starting image. Needs no registry or index access at pod start.

Cost

You rebuild the image to change versions rather than just retagging a bundle.

Options B and C are both fully supported on a released build — pick whichever fits your release flow. Option C is the usual choice for locked-down or air-gapped sites.

Not published for a released build

The connector wheel bundle is published as an OCI image only for internal development builds. No lcf-vllm-plugin-wheels image is published for a release, so on 1.2.0 there is nothing for this init container to pull. The rest of this option applies only if Lightbits Support has given you an internal build and access to the registry that carries it.

With such a build, the vLLM Deployment carries an init container whose image is the connector wheel bundle, tagged x.y.z (for example …/lcf/lcf-vllm-plugin-wheels:x.y.z); on startup it copies the *.whl files into the shared /lcf-wheels volume, and the launch shim then installs them from there. Two images are therefore in that one pod: the vLLM 0.30.0 runtime image by default (the engine) and this wheel-bundle image (the connector).

"Set the bundle to x.y.z" means: point that init container's wheel-bundle image tag at the version you want, then roll the pod. Two equivalent ways:

  • Via the LCF deployment tooling / Helm values (recommended): set the connector version value (for example x.y.z = 1.2.0) and helm upgrade / re-apply — the chart renders the init-container image tag for you.

  • By hand on an existing Deployment:

# repoint the wheel-bundle init container to the desired connector version
kubectl -n <vllm-namespace> set image deployment/<vllm-deployment> \
stage-lcf-wheels=<registry>/lcf/lcf-vllm-plugin-wheels:x.y.z
# roll the pod so the launch shim reinstalls from the new bundle
kubectl -n <vllm-namespace> rollout restart deployment/<vllm-deployment>

Either way, changing the connector version = changing which wheel bundle lands in /lcf-wheels, then restarting the pod.

On that restart the LCF Deployment is already wired (the --kv-transfer-config, the env vars, and the /var/run/lcf socket mount are all present), so the connector auto-connects to the node-local LCFC at engine startup — there is no separate connect step. But because the connector is fail_open, if LCFC is not reachable on that node vLLM still comes up and serves without LCF; so "loaded" ≠ "connected" — always confirm via the logs below and a warm-turn LCF cache hit.

Either way you end up with lcf-vllm-connector==x.y.z and pylcf==x.y.z installed in the vLLM Python environment.

Connector Environment and Launch Flags

The vLLM container needs this env var:

env:
- { name: LCF_VLLM_RUNTIME_CONFIG, value: "/etc/lcf-vllm/runtime.yaml" }

Add VLLM_SERVER_DEV_MODE=1 if you intend to run the cold/warm proof — its POST /reset_prefix_cache call is only served in dev mode, and nothing sets it for you:

- { name: VLLM_SERVER_DEV_MODE, value: "1" }
Dev mode is a debug surface

Dev mode exposes additional debug endpoints on the engine's HTTP port, so treat a pod that carries it as you would any debug surface. It is not needed for serving; only for that test.

VLLM_USE_V2_MODEL_RUNNER

Optionally, VLLM_USE_V2_MODEL_RUNNER=0 pins vLLM to the v1 model runner. Nothing in Inferra reads it and working deployments do not set it; set it only if Lightbits Support asks you to for your release.

Two optional variables control distributed-trace volume from this pod. Both are safe to leave unset; set them only while investigating a latency problem, and unset them afterwards.

Variable

Default

Effect

LCF_TRACE_SAMPLE_RATE

100

Trace 1 request in N, on both the read (GET) and write (PUT) paths. 1 traces every request — roughly 600 OTLP log records each, which will overwhelm a small collector. 0 turns tracing off even when an OTLP endpoint is configured.

LCF_TRACE_GET_CHUNKS

unset

Per-chunk detail inside a traced request. boundary adds four rows per request; all adds one row per chunk (thousands per long-prefix request). Set the same value on the LCFC daemon or the two halves of the trace will not match.

Tracing also requires telemetry to be on; with telemetry turned off (below), no trace is produced at any sample rate.

Where the connector sends its telemetry

The connector pushes its metrics and traces over plaintext OTLP to http://lcf-observability.lcf.svc.cluster.local:24317 — the observability service of an Inferra release installed in the namespace lcf. One variable changes that:

LCF_TELEMETRY__OTLP_ENDPOINT

Effect

unset

The default above.

a URL

Telemetry goes there instead.

set to an empty value

Telemetry is off: no metrics and no traces from this pod.

If Inferra is installed in any namespace other than lcf, you must set it

Set it to http://lcf-observability.<namespace>.svc.cluster.local:24317. Otherwise the connector keeps sending to an address that does not exist, its metrics never reach the dashboard, and nothing reports an error.

LCF_VLLM_RUNTIME_CONFIG points at the runtime ConfigMap; omit it and the connector falls back to its own default path. The socket is not set by an env var: it comes from lcf_ipc_path in --kv-transfer-config (below), or from ipc_path in the runtime ConfigMap.

You also need the /var/run/lcf host path mounted in (this is the socket the node-local LCFC exposes), plus the runtime ConfigMap (enabled / load_enabled / store_enabled / fail_open: true, ipc_path: /var/run/lcf/lcfc.sock).

The pod must also run hostIPC: true

This is a hard requirement, not a tuning option. The SDK reaches LCFC over POSIX shared memory in the host's /dev/shm — request/response rings and a shared data pool. Shared-memory objects live in an IPC namespace, and a pod gets its own by default, so without hostIPC the SDK's shm_open returns ENOENT, the engine silently falls back to local-only, and serves traffic normally with no caching and no error in vLLM's own metrics. A deployment can run like that indefinitely; the only symptom is a cache hit rate that never rises.

See Why the vLLM Pod Needs Host IPC for the full rationale, including why the UCX mm transport is a different thing and not a substitute.

Do not mount a volume at /dev/shm

That is the emptyDir with medium: Memory that many vLLM manifests carry. With hostIPC: true it hides the host's /dev/shm from the pod, and the SDK falls back exactly as it does without hostIPC. The host's /dev/shm is not bounded by the container default, so the engine does not need the mount.

So the pod spec needs all three:

spec:
hostIPC: true # REQUIRED — SDK <-> LCFC shared-memory rings
containers:
- name: vllm
env:
- { name: LCF_VLLM_RUNTIME_CONFIG, value: "/etc/lcf-vllm/runtime.yaml" }
volumeMounts:
- { name: lcf-socket, mountPath: /var/run/lcf }
- { name: lcf-runtime, mountPath: /etc/lcf-vllm }
volumes:
- name: lcf-socket
hostPath: { path: /var/run/lcf, type: Directory }
- name: lcf-runtime
configMap: { name: lcf-vllm-runtime }

The LCFC side is already configured for this: the chart runs LCFC with hostIPC: true by default, so that SDK clients see the same POSIX SHM namespace. Enabling hostIPC on the vLLM pod is the half you have to do.

What this guide does not specify, and you must

The fragments above are the connector's requirements, not a complete Deployment. There is no LCF chart for the vLLM pod on a released build, so the following are yours to decide and this guide deliberately does not prescribe them — they depend on your model, your hardware and your platform conventions:

  • the GPU resource request and limit (nvidia.com/gpu),

  • how the model weights reach the container — a PVC, a hostPath, or a download at startup,

  • a Service to reach the engine, and any Ingress in front of it,

  • readiness and liveness probes, remembering that a large model takes a long time to load (see Apply and Restart the vLLM Pod below),

  • --max-model-len, which must cover the longest prompt you intend to serve.

Get the connector pieces right and the engine pieces are an ordinary vLLM deployment.

The vllm serve launch line

The connector is selected by appending --kv-transfer-config to the vllm serve command. Leave the hybrid KV-cache manager at its default — do not add --disable-hybrid-kv-cache-manager (see Keep the Hybrid KV-cache Manager On in Supported vLLM Versions and Models):

vllm serve <model-path> \
--served-model-name <served-model-name> \
--tensor-parallel-size 8 \
--max-model-len <N> \
--enable-prefix-caching \
--enable-chunked-prefill \
--trust-remote-code \
--kv-cache-dtype auto \
--kv-transfer-config '{"kv_connector":"LcfVllmConnectorV1","kv_connector_module_path":"lcf_vllm_connector.connector","kv_role":"kv_both","kv_load_failure_policy":"recompute","kv_connector_extra_config":{"lcf_ipc_path":"/var/run/lcf/lcfc.sock","lcf_namespace":"<lcf-cache-scope>"}}'
Replicas share the cache

Set the Deployment's replicas to what your traffic needs. The cache key an engine computes carries no per-replica identity, so every replica of the same model bound to the same lcf_namespace reuses the KV the others stored — a prefix one replica computed is a warm hit on the next. Give engines separate lcf_namespace values only when their cached KV must be kept apart.

Qwen 3.5 on vLLM 0.23 only — required on H100-class GPUs

Pass --additional-config '{"gdn_prefill_backend": "triton"}'. This selects the Triton GDN (Gated DeltaNet, "mamba-style") prefill backend. It is not needed on vLLM 0.27.1 or later, including the 0.30.0 default, where the affected kernel is no longer used.

Without it, vLLM 0.23 auto-selects a FlashInfer CUTLASS GDN prefill kernel that deadlocks the GPU on SM90 (H100/H800). The symptom is not a slowdown and not a crash: the engine stops retiring work, reports 100% GPU utilisation at idle power, and does not recover — every subsequent request times out until you restart the pod. It has reproduced within about three minutes of sustained load.

The cause is upstream in FlashInfer 0.6.12, which vLLM 0.23 pins, and it reproduces on unmodified vLLM, so it is not specific to Inferra or the connector. On vLLM 0.23, treat this flag as part of the required launch line for Qwen, not as a tuning preference.

In that command, <model-path>, <served-model-name>, and <N> (max-model-len) are model-specific, and <lcf-cache-scope> is a value you choose — see below.

Choosing lcf_namespace and lcf_tenant_id

lcf_namespace is not a Kubernetes namespace

lcf_namespace (shown above as <lcf-cache-scope>) is a logical KV-cache scope label, not a Kubernetes namespace — the connector stamps it onto every key it writes.

You choose it, and there is no tooling on a released build that will choose it for you (the connector is wheels in your own vLLM pod, not a separate deployment). What matters:

  • Pods that must share cached KV have to use the identical string. Two engines serving the same model with different values each build their own cache and neither ever hits the other's — with no error, just a hit rate that stays lower than it should.

  • Pods that must not share it need different strings — different models, in particular, since KV from one model is meaningless to another.

  • Omitted entirely, the connector defaults to vllm.

  • Anything stable and descriptive works. A <model>-lcf convention keeps it obvious which cache belongs to which engine; naming it after the served model is the easiest rule to remember.

Lightbits Support can recommend a scheme if you are running many engines.

lcf_tenant_id (optional) is the LCF tenant the pod serves. The connector opens one fabric session per engine, so a vLLM pod serves exactly one tenant — deploy one pod per tenant. Set it to the name of a tenant created through the LCF API. That tenant must own a logical namespace whose name equals lcf_namespace; the pod's session then binds to that tenant's namespace, and its KV is isolated from every other tenant's. Omitted, it defaults to vllm-connector, the value every deployment used before the knob existed. An empty value is rejected at startup.

Both must already exist in the cluster, and a fresh install creates neither

A clean LCF install has one tenant (system) and one logical namespace (logical-namespace-0), so the connector's defaults — lcf_tenant_id: vllm-connector and lcf_namespace: vllm — match nothing.

The session then comes up unbound, and LCF does not reject it — it routes your KV to logical namespace 0 instead, the bootstrap namespace owned by the system tenant. Nothing errors on either side and the cache may look like it is working, but the isolation you configured is not in effect: your blocks share a namespace with every other unbound client, and quota and data-at-rest encryption are resolved from namespace 0 rather than from your tenant. Treat it as a misconfiguration to fix before serving real traffic, not as a performance symptom.

Create the pair before starting the engine — Step 8b: Create the Tenant and Logical Namespace the Connector Binds To in LCF-CLI Deployment has the commands. Then confirm with Confirm the Connector Loaded below: the LCF session: line must be accompanied by session created (bound) in the lcfc log, not session created UNBOUND. Creating the pair for an engine that is already running does bind it — the session rebinds when the namespace reaches the cluster directory, with no pod restart.

Apply and Restart the vLLM Pod

A config/wheel change only takes effect on a fresh engine, so roll the Deployment:

kubectl -n <vllm-namespace> rollout restart deployment/<vllm-deployment>
kubectl -n <vllm-namespace> rollout status deployment/<vllm-deployment> --timeout=120m
A long Ready time is normal

The vLLM engine takes a long time to become Ready (weight load + CUDA-graph capture); a 1–2 h progressDeadlineSeconds / status timeout is normal. The pod uses a Recreate strategy because it holds all GPUs — the old pod must exit before the new one is admitted.

Confirm the Connector Loaded

kubectl -n <vllm-namespace> logs deploy/<vllm-deployment> -c vllm \
| grep -iE "LcfVllmConnectorV1|lcfc\.sock|lcf_namespace|LCF session"

Seeing LcfVllmConnectorV1 (and the socket / namespace) in the logs confirms the connector registered and is active. The LCF session: namespace=... tenant_id=... line shows the effective cache scope and tenant the pod is serving.

Registered is not the same as caching

Because the connector is fail_open, a registered connector can still be serving nothing. Finish with the cold-versus-warm test.

Prompts Larger Than GPU Memory — VP Streaming (Preview)

New in 1.2.0 Preview Stock vLLM cannot serve a request whose KV cache is larger than the engine's GPU KV pool: it refuses a --max-model-len above that capacity at startup. VP streaming (virtual paging with streamed attention) lifts that limit for a single request. Once the request's early KV has been stored to Inferra, the connector frees those GPU blocks for the rest of the prompt; each attention layer then reads the evicted prefix back from Inferra one window at a time and combines it exactly with the part still in GPU memory. Requests that fit in GPU memory are not affected and run the normal path.

Off by default — enable it only together with Lightbits Support

It changes nothing until you enable it. Because it is a preview, enable it only together with Lightbits Support, who will help you size the deployment. Validated so far: Llama-4-Scout-17B-16E at TP8 on 8x NVIDIA H100 (Hopper) GPUs, vLLM 0.30.0, with a 128,000-token prompt about 3.3 times the engine's GPU KV capacity — the output matched the same prompt served without streaming.

Enabling it takes both of the following; either one alone does nothing:

  • the environment variable LCF_STREAMING_SHIM=1 on the vLLM container, and

  • "streaming_enabled": true and "virtual_paging_enabled": true in kv_connector_extra_config. "streaming_window_tokens" sets the size of one window (default 8192).

--kv-transfer-config '{"kv_connector":"LcfVllmConnectorV1","kv_connector_module_path":"lcf_vllm_connector.connector","kv_role":"kv_both","kv_load_failure_policy":"recompute","kv_connector_extra_config":{"lcf_ipc_path":"/var/run/lcf/lcfc.sock","lcf_namespace":"<lcf-cache-scope>","streaming_enabled":true,"virtual_paging_enabled":true}}'

Requirements and limits in this release:

  • One sequence at a time: --max-num-seqs 1. The streamed attention serves one request per step. With a larger value the connector refuses streaming at startup.

  • No sliding-window attention layers. On a model with sliding-window layers (gpt-oss, for example) the connector refuses streaming at startup. Chunked-local attention, as in Llama-4-Scout, is supported.

  • Piecewise CUDA graphs. With streaming enabled the connector asks vLLM for piecewise CUDA-graph mode, because the streamed path must run on every decode step. Expect lower decode throughput than the same engine without streaming.

  • Host memory bounds the context. Each step brings one layer's whole evicted prefix into host memory before it reaches the GPU, so the longest prompt you can stream depends on the vLLM pod's host memory. Give the pod a memory limit with headroom and size it with Lightbits Support.

  • A refusal is loud. If --max-model-len exceeds the GPU KV capacity and streaming cannot run — one of the requirements above is not met, or the connector configuration does not ask for it — the engine fails at startup and names the cause, rather than starting with requests that can never be scheduled.

  • No partial KV. If a streamed request's evicted prefix cannot be read back, the request ends with an error finish reason. It never returns output computed from incomplete KV.