Post-Deployment Validation
A healthy cluster and a working cache are different claims. This page is the check to run once, right after deploying, to prove Inferra is actually serving KV blocks — and to learn what the numbers should look like so a later regression is recognizable.
The Shape of the Test
The measurement is a cold-versus-warm comparison on the same prompts.
Run both passes against the same engine, with the observability dashboard open.
Send a set of long-prefix prompts to a freshly started engine with an empty cache. Every request prefills in full. Record time to first token and throughput. This is the baseline — what inference costs without Inferra.
Send the same prompts again. Their prefix blocks are now in the fabric, so the connector loads them instead of prefilling. Record the same numbers.
The difference between the two passes is Inferra's contribution. A single pass in isolation does not tell you anything.
Two properties matter for the test to mean anything:
Long prompts. Cache benefit scales with how much prefix is reused, and prefill — the work Inferra removes — grows faster than linearly with sequence length. Short prompts have little to save and a test built from them will correctly show almost no benefit.
Reuse is between passes, not between prompts. The prompts in the set are distinct from each other and byte-identical across the two passes. This is the point people get wrong: if the prompts in the set shared a prefix with each other, prompts 2..N would hit on prompt 1's blocks and the cold pass would not be cold. Multi-turn conversation traffic is the realistic production shape for cache benefit, but it is the wrong shape for this measurement.
A concrete recipe, which is what our own release validation runs, so the figures are comparable to the ones in the Release Notes:
Parameter | Value |
|---|---|
Prompts | 8, distinct from one another, reused verbatim in the warm pass |
Prompt length | ~120,000 tokens each |
Output tokens | 48 — you are measuring first-token latency, not decode |
Concurrency | 1, so queueing does not enter the number |
Between passes | Restart or reset the engine's GPU prefix cache (see below) |
Shorter prompts work too — 12K and 60K are the other sizes we measure — but expect the speed-up to fall with them.
What to Expect
Measure | Cold pass | Warm pass |
|---|---|---|
KV-cache hit rate | Near zero | Close to 1; the headline number |
Time to first token | Full prefill cost | Lower — this is where the benefit appears first |
Output throughput | Baseline | Similar or better; decode is unchanged, so do not expect a large move |
Fabric get latency | n/a (no hits) | Should be a small fraction of the prefill it replaced |
Time to first token is the measure to watch. Inferra replaces prefill work, and prefill dominates TTFT on long prompts; decode is untouched, so end-to-end throughput moves much less than TTFT does. A test that reports only average end-to-end latency will understate the effect.
Absolute numbers depend on the model, prompt length, hardware and concurrency, so this guide does not quote targets. Capture your own cold and warm figures at install time and keep them: they are the baseline that makes a later regression obvious. The Release Notes give the figures we measure on our own validation hardware, which is the closest thing to an expectation; ask Lightbits Support what to expect for your configuration.
Measuring the Hit Rate
The hit rate is the headline number, so it is worth being exact about which one you are reading. The connector reports it through the vLLM engine's own metrics endpoint, as a pair of counters:
Both are cumulative since engine start, so take the difference across a pass rather than the absolute value. Scrape them from inside the pod:
That output also shows vllm:inferra_real_ext_hit_tokens_total, which counts tokens the fabric actually delivered rather than blocks vLLM expected to be available. The two can disagree — the numerator above is vLLM's own optimistic accounting, and the inferra_ counter is the stricter one. Use the pair above for the headline rate, and reach for the inferra_ counter when a high hit rate is not producing the TTFT improvement it should.
Watching It Happen
Run the two passes with the observability dashboard open. During the warm pass you should see the cache hit rate climb and fabric get activity appear. If hit rate stays flat through a warm pass, work through these three before concluding the cache is ineffective — all three present as "no benefit", and none of them reports an error:
Check
hostIPCon the vLLM pod, and that no volume is mounted at/dev/shm— Why the vLLM Pod Needs Host IPC. This is the most common cause by a wide margin.Check the connector actually loaded — see Confirm the Connector Loaded.
Check you are reading a metric name the build exports — Observability, trap 3. A flat line and a mistyped metric name look identical.
A session that came up UNBOUND (Step 8b in LCF-CLI Deployment) does not usually show as a flat hit rate: the data path falls back to namespace 0 and the cache appears to work. Confirm it anyway before you trust a good result, because a hit rate measured in the bootstrap namespace is not a measurement of the tenant you meant to test.
Any HTTP or benchmark client you already use will do: the whole test is "send the set, record TTFT, reset, send it again". Stream the responses so you measure time to first token rather than total time. Lightbits also maintains benchmark harnesses that automate the cycle and report hit rate, TTFT and throughput per pass; they are not shipped with the product, so ask Support if you would rather not write the driver yourself.
To make the warm pass prove that Inferra served the KV — rather than the engine's own GPU prefix cache — clear that GPU cache between passes. The connector guide's Prove It's Actually Using LCF page covers how, including the VLLM_SERVER_DEV_MODE=1 the reset endpoint needs. Without that step a warm hit could have come from the GPU and does not tell you anything about the fabric.