Observability
Inferra ships its own observability stack as a subchart, lcf-observability, enabled by default. You do not install Prometheus or Grafana: the stack bundles metric storage, log storage, alerting and a web UI in one pod.
What the Stack Contains
Component | Purpose | Port |
|---|---|---|
Inferra dashboard | Web UI and query API |
|
VictoriaMetrics | Metric storage |
|
VictoriaLogs | Log storage |
|
OpenTelemetry Collector | Ingest, labeling, forwarding |
|
vmalert + Alertmanager | Rule evaluation and routing |
|
Envoy | mTLS termination for ingest |
|
lcft and lcfc push metrics and traces to a local Envoy sidecar over a Unix socket; Envoy terminates mTLS to the collector. Nothing needs a node port and nothing crosses the network unauthenticated.
Reaching the Dashboard
The UI Service is a NodePort by default, so it is reachable from outside the cluster without a port-forward.
The UI requires authentication. The admin account is seeded at install with a random password stored in the lcf-observability-admin Secret — there is deliberately no shipped default, because a NodePort means anyone routable to a node IP can reach the login page:
The Secret is named after the observability subchart, not after your Helm release, so it is lcf-observability-admin whatever you called the release.
The password must be changed at first login: it is a bootstrap credential, not a standing one. It survives helm upgrade and is never rotated by an upgrade. If it is lost afterwards, reset it from inside the pod:
The diagnostics API key. The chart also makes an API key for lcf-cli diag collect (Troubleshooting), in Secret lcf-observability-diag-key. It can only create, download and delete call-home bundles, and changing the admin password does not affect it. To list the service's API keys, or revoke one, log in as an administrator:
A revoke is kept in the observability service's database, so it is lost if that database is lost (for example, when the pod moves to another node). To revoke or replace the key for good, delete the Secret and upgrade the release; the service restarts to take the new key, and the old one never works again. Set lcf-observability.diag.apiKey.enabled=false to have no such key.
A NodePort is reachable by anyone who can route to a node IP. Keep it on a trusted network, put a TLS-terminating ingress in front of it, or set lcf-observability.service.type=ClusterIP and reach it by port-forward.
Metrics Worth Watching
Signal | What it tells you |
|---|---|
KV-cache hit rate | The share of prefix blocks served from Inferra rather than recomputed. This is the number the product exists to move. |
Time to first token (TTFT) | Cold versus warm difference is where cache benefit shows up first. |
Fabric get/put latency | Inferra's own contribution to TTFT, separate from prefill. The fabric GET latency covers reads fetched over the fabric only; |
| Fills, evictions, and per-device health. |
RAM-tier residency |
|
RAM-tier hit split |
|
Certificate expiry |
|
Set clusterId per deployment. It is stamped onto every signal, and without it signals from two clusters are indistinguishable once forwarded anywhere shared.
Three Traps That Cost Real Debugging Time
These are not hypothetical. Each one has already cost people time.
1. The plaintext OTLP port is 24317, not 4317
The daemons use the mTLS port (4317) through their Envoy sidecars. The vLLM connector cannot: pylcf runs inside the vLLM pod, which the serving fleet owns, so no Envoy sidecar and no certificate mount can be added to it. It therefore pushes to the plaintext ingest port, 24317, on the ClusterIP service.
If you configure the connector against 4317, its telemetry is silently dropped.
2. Narrowing the plaintext port must cover every vLLM node and namespace
By default the subchart's NetworkPolicy admits plaintext OTLP from any in-cluster source, because a vLLM instance spanning several nodes runs with hostNetwork and its pods carry no namespace identity. Once you set lcf-observability.networkPolicy.otlpFromCidrs, plaintext OTLP is allowed only from those CIDRs and from the namespaces in lcf-observability.networkPolicy.otlpPlaintextFromNamespaces (default vllm).
Telemetry from a node or namespace outside them is denied by our own policy, and nothing in any log says so. Include your vLLM namespace, or you will see a healthy cluster with no connector metrics at all.
3. All-zero counters mean one of two different things
The connector's metric namespace differs by build: older builds export vllm:external_prefix_cache_*, newer ones also export vllm:inferra_*. A dashboard or a query written against the wrong prefix reads exactly like a cluster serving no traffic. Before concluding "the cache is not working", check which names actually exist:
What you see | What it means |
|---|---|
No output at all | The connector is not reporting — start with trap 1 and trap 2. |
Names present but zero | The connector is reporting and genuinely serving nothing yet, which on a cold cache is the correct answer. |
Retention
Metrics are retained for 30 days by default (victoriaMetrics.retentionPeriod). Logs are capped by both age and volume: under a log storm the oldest per-day partitions are dropped so the stack stays up rather than filling its volume. Sizing presets are values-small.yaml, values-medium.yaml (the default) and values-large.yaml.
If the stack runs in hostPath mode, its data lives on one node. A reschedule elsewhere starts empty and strands the history on the previous node. Set global.pinNodeName to keep it put, or use persistent mode with a real StorageClass.
Collecting a Support Bundle
New in 1.2.0 When Lightbits Support asks for diagnostics, one command collects the observability stack's metrics and logs for a time window, plus a snapshot of the cluster's namespaces, devices and components, into a single archive on your machine. That file is what you send to Support.
For a support case, start with Troubleshooting (from 1.2.0). The command in Collecting Diagnostics for Support, lcf-cli diag collect, includes this bundle (it reaches the observability service through a port-forward and logs in with the diagnostics API key, so it needs no password), together with the Kubernetes, host and vLLM state support also needs, in one file. Use obs bundle on its own when support asks only for metrics and logs.
Inferra opens no outbound connection: the bundle is built inside the cluster when you ask for it, you download it, and you decide whether and how to send it.
Before you start
The command authenticates to the observability service as an administrator of its UI — the admin account from Reaching the dashboard above, or another administrator. A freshly installed admin must change its bootstrap password at first login before anything else is accepted, so log in to the UI once first.
There is no TLS listener, so the command refuses an http:// address unless you pass --insecure-http, and the password crosses the network in clear. Run it from a host on a trusted network, or through a port-forward.
Collect
Read the password into the environment rather than passing it as an argument, which would leave it in the process list and your shell history:
--obs-endpoint, --username, --password and --insecure-http belong to obs bundle itself, so lcf-cli obs bundle create --insecure-http … is rejected with unexpected argument. The endpoint and password above come from the environment, so only --insecure-http needs placing.
The archive is written to the current directory as lcf-callhome-<cluster>-<id>.tar.gz and its full path is printed when it finishes; nothing appears at that name until the whole download has succeeded, so an interrupted run cannot leave a truncated file to send by mistake. Useful options:
Option | Effect |
|---|---|
| How far back to collect, e.g. |
| Collect an exact window instead — an incident is usually bounded on both sides. |
| Collect logs only from that component, e.g. |
| Collect a long window as a series of bundles of that length, numbered in time order and named for the window each holds, each removed from the cluster once downloaded. Send all of them. |
| Write the archive somewhere other than the current directory. |
| Remove the bundle from the cluster once it is downloaded. Off by default, so a failed download can be retried. |
| Collect without redaction. Only if Support asks for it. |
Labeled fields that hold credentials or user identity — password, token, cookie, session, user, email and the like — are erased. Host names, IP addresses, pod names and tenants in labeled fields are replaced by stable pseudonyms (node-2), consistent across the whole bundle, so Support can still tell one node from another without learning which machine it is. Inside the free text of a log message, inline secrets (password=…, token=…, Bearer …), JWTs and IP addresses are removed; user names, e-mail addresses and host names that appear in a message's text are not. If your logs contain them, review the bundle before you send it. The namespace, device, and component inventory is kept, as that is what Support needs to read the bundle.
Bundles kept on the cluster
Each bundle is also kept on the cluster, so you can fetch it again:
They live on a volume of their own (lcf-observability.storage.callhome.size, default 4Gi), and after each new bundle the oldest are deleted once there are more than lcf-observability.callhome.maxBundles (default 10) or they total more than lcf-observability.callhome.maxTotalBytes (default 2 GiB). The newest bundle is never deleted. One bundle is built at a time — a second request while one is building is refused, so retry it — and a single window is limited to seven days (lcf-observability.callhome.maxWindowSecs).