Prove It's Actually Using LCF

The simplest way to show the connector is live is a cold-then-warm test against the vLLM OpenAI-compatible API.

The idea

Cold-then-warm against the OpenAI-compatible API

Four steps. The whole point is the gap between step 2 and step 3.

Reach the engine

Expose the vLLM service — for example port-forward it to localhost:8000 — so you can send it requests.

Cold turn

Send a long prompt (tens to hundreds of thousands of tokens) once. Nothing is cached yet, so vLLM must prefill the whole prompt — note the time-to-first-token (TTFT). Stream the response so you can measure TTFT separately from decode time.

Warm turn

Send the identical prompt again. The KV cache for that prompt is now resident on the LCF fabric, so prefill is skipped and TTFT drops sharply.

Compare

A large, repeatable warm TTFT ≪ cold TTFT gap is the proof the connector served the KV from LCF instead of recomputing it.

kubectl -n <vllm-namespace> port-forward deploy/<vllm-deployment> 8000:8000

Make it a definitive proof: clear the GPU prefix cache

For a stronger result, clear vLLM's local GPU prefix cache between the cold and warm turns — otherwise a warm hit could be served from the GPU and tell you nothing about LCF. vLLM exposes this as an HTTP call:

curl -X POST http://localhost:8000/reset_prefix_cache
That endpoint needs dev mode, and you have to set it yourself

/reset_prefix_cache is served only when the engine was started with VLLM_SERVER_DEV_MODE=1. There is no LCF deployment template on a released build, so add it to the pod's env as shown in Connector Environment and Launch Flags and restart before running this test.

With the GPU cache cleared, any warm-turn speed-up can only come from KV reloaded over the LCF fabric, not from the GPU — that is the definitive proof the connector is serving cache.

Clearing the GPU cache is not the same as clearing Inferra's

Resetting the prefix cache empties vLLM's GPU-side cache only; the KV stays in the Inferra fabric, which is the point. If you also want to force the read past the node-local RAM tier and down to the storage tier — useful when establishing both figures — run lcf-cli node clear-ram-tier between the turns as well.

Use any client you like

You can drive this with any HTTP/benchmark client you already use (a short streaming script, or a standard LLM load-testing tool). Lightbits Support can provide a ready-made cold/warm benchmark helper on request.

For the cluster-side view of the same test — what the dashboard should show, and the three causes of a flat hit rate — see Post-Deployment Validation.