Observability

Inferra ships its own observability stack as a subchart, lcf-observability, enabled by default. You do not install Prometheus or Grafana: the stack bundles metric storage, log storage, alerting and a web UI in one pod.

What the Stack Contains

Component

Purpose

Port

Inferra dashboard

Web UI and query API

8080

VictoriaMetrics

Metric storage

8428

VictoriaLogs

Log storage

9428

OpenTelemetry Collector

Ingest, labeling, forwarding

4317 (mTLS), 24317 (plaintext)

vmalert + Alertmanager

Rule evaluation and routing

8880, 9093

Envoy

mTLS termination for ingest

9901 (admin)

lcft and lcfc push metrics and traces to a local Envoy sidecar over a Unix socket; Envoy terminates mTLS to the collector. Nothing needs a node port and nothing crosses the network unauthenticated.

Reaching the Dashboard

The UI Service is a NodePort by default, so it is reachable from outside the cluster without a port-forward.

kubectl -n "$LCF_NAMESPACE" get svc -l app.kubernetes.io/name=lcf-observability

The UI requires authentication. The admin account is seeded at install with a random password stored in the lcf-observability-admin Secret — there is deliberately no shipped default, because a NodePort means anyone routable to a node IP can reach the login page:

kubectl -n "$LCF_NAMESPACE" get secret lcf-observability-admin \
-o jsonpath='{.data.password}' | base64 -d; echo

The Secret is named after the observability subchart, not after your Helm release, so it is lcf-observability-admin whatever you called the release.

The password must be changed at first login: it is a bootstrap credential, not a standing one. It survives helm upgrade and is never rotated by an upgrade. If it is lost afterwards, reset it from inside the pod:

kubectl exec -n "$LCF_NAMESPACE" <pod> -c lcf-observability -- \
lcf-observability admin reset-password --username admin --password <new>

The diagnostics API key. The chart also makes an API key for lcf-cli diag collect (Troubleshooting), in Secret lcf-observability-diag-key. It can only create, download and delete call-home bundles, and changing the admin password does not affect it. To list the service's API keys, or revoke one, log in as an administrator:

export LCF_OBS_ENDPOINT=http://127.0.0.1:8080 # or http://<node-ip>:31080
read -rs LCF_OBS_PASSWORD && export LCF_OBS_PASSWORD
lcf-cli obs api-key --insecure-http list
lcf-cli obs api-key --insecure-http revoke diag

A revoke is kept in the observability service's database, so it is lost if that database is lost (for example, when the pod moves to another node). To revoke or replace the key for good, delete the Secret and upgrade the release; the service restarts to take the new key, and the old one never works again. Set lcf-observability.diag.apiKey.enabled=false to have no such key.

Exposure

A NodePort is reachable by anyone who can route to a node IP. Keep it on a trusted network, put a TLS-terminating ingress in front of it, or set lcf-observability.service.type=ClusterIP and reach it by port-forward.

Metrics Worth Watching

Signal

What it tells you

KV-cache hit rate

The share of prefix blocks served from Inferra rather than recomputed. This is the number the product exists to move.

Time to first token (TTFT)

Cold versus warm difference is where cache benefit shows up first.

Fabric get/put latency

Inferra's own contribution to TTFT, separate from prefill. The fabric GET latency covers reads fetched over the fabric only; pylcf_sdk_get_serve_latency_ns, labeled source="tier" or source="fabric", times RAM-tier and fabric reads side by side.

lcft device and capacity

Fills, evictions, and per-device health.

RAM-tier residency

lcf_ram_tier_used_bytes against lcf_ram_tier_capacity_bytes — how full the host-RAM tier is. A tier that never fills is larger than the working set; one that is always full is the constraint on hit rate.

RAM-tier hit split

lcf_ram_tier_read_hits_total (served from host RAM), lcf_ram_tier_read_miss_fabric_hits_total (missed the tier, served from the storage tier) and lcf_ram_tier_read_miss_notfound_total (not cached anywhere). The first two are both cache hits; the split tells you which tier is doing the work.

Certificate expiry

days_until_first_cert_expiring on each Envoy admin endpoint — see Troubleshooting.

Always set clusterId

Set clusterId per deployment. It is stamped onto every signal, and without it signals from two clusters are indistinguishable once forwarded anywhere shared.

Three Traps That Cost Real Debugging Time

These are not hypothetical. Each one has already cost people time.

1. The plaintext OTLP port is 24317, not 4317

The daemons use the mTLS port (4317) through their Envoy sidecars. The vLLM connector cannot: pylcf runs inside the vLLM pod, which the serving fleet owns, so no Envoy sidecar and no certificate mount can be added to it. It therefore pushes to the plaintext ingest port, 24317, on the ClusterIP service.

Silent drop

If you configure the connector against 4317, its telemetry is silently dropped.

2. Narrowing the plaintext port must cover every vLLM node and namespace

By default the subchart's NetworkPolicy admits plaintext OTLP from any in-cluster source, because a vLLM instance spanning several nodes runs with hostNetwork and its pods carry no namespace identity. Once you set lcf-observability.networkPolicy.otlpFromCidrs, plaintext OTLP is allowed only from those CIDRs and from the namespaces in lcf-observability.networkPolicy.otlpPlaintextFromNamespaces (default vllm).

Denied by our own policy, with nothing in any log

Telemetry from a node or namespace outside them is denied by our own policy, and nothing in any log says so. Include your vLLM namespace, or you will see a healthy cluster with no connector metrics at all.

helm upgrade ... \
--set 'lcf-observability.networkPolicy.otlpPlaintextFromNamespaces[0]=REPLACE_WITH_VLLM_NAMESPACE'

3. All-zero counters mean one of two different things

The connector's metric namespace differs by build: older builds export vllm:external_prefix_cache_*, newer ones also export vllm:inferra_*. A dashboard or a query written against the wrong prefix reads exactly like a cluster serving no traffic. Before concluding "the cache is not working", check which names actually exist:

kubectl -n REPLACE_WITH_VLLM_NAMESPACE exec <vllm-pod> -- \
curl -s localhost:8000/metrics | grep -E 'external_prefix_cache|inferra_'

What you see

What it means

No output at all

The connector is not reporting — start with trap 1 and trap 2.

Names present but zero

The connector is reporting and genuinely serving nothing yet, which on a cold cache is the correct answer.

Retention

Metrics are retained for 30 days by default (victoriaMetrics.retentionPeriod). Logs are capped by both age and volume: under a log storm the oldest per-day partitions are dropped so the stack stays up rather than filling its volume. Sizing presets are values-small.yaml, values-medium.yaml (the default) and values-large.yaml.

Storage

If the stack runs in hostPath mode, its data lives on one node. A reschedule elsewhere starts empty and strands the history on the previous node. Set global.pinNodeName to keep it put, or use persistent mode with a real StorageClass.

Collecting a Support Bundle

New in 1.2.0 When Lightbits Support asks for diagnostics, one command collects the observability stack's metrics and logs for a time window, plus a snapshot of the cluster's namespaces, devices and components, into a single archive on your machine. That file is what you send to Support.

For a support case, start with Troubleshooting (from 1.2.0). The command in Collecting Diagnostics for Support, lcf-cli diag collect, includes this bundle (it reaches the observability service through a port-forward and logs in with the diagnostics API key, so it needs no password), together with the Kubernetes, host and vLLM state support also needs, in one file. Use obs bundle on its own when support asks only for metrics and logs.

Nothing leaves your cluster on its own

Inferra opens no outbound connection: the bundle is built inside the cluster when you ask for it, you download it, and you decide whether and how to send it.

Before you start

The command authenticates to the observability service as an administrator of its UI — the admin account from Reaching the dashboard above, or another administrator. A freshly installed admin must change its bootstrap password at first login before anything else is accepted, so log in to the UI once first.

The service listens on plain HTTP

There is no TLS listener, so the command refuses an http:// address unless you pass --insecure-http, and the password crosses the network in clear. Run it from a host on a trusted network, or through a port-forward.

kubectl -n "$LCF_NAMESPACE" port-forward svc/lcf-observability-ui 8080:8080

Collect

Read the password into the environment rather than passing it as an argument, which would leave it in the process list and your shell history:

export LCF_OBS_ENDPOINT=http://127.0.0.1:8080 # or http://<node-ip>:31080
read -rs LCF_OBS_PASSWORD && export LCF_OBS_PASSWORD

lcf-cli obs bundle --insecure-http create --since 6h
Connection options go before the subcommand, not after it

--obs-endpoint, --username, --password and --insecure-http belong to obs bundle itself, so lcf-cli obs bundle create --insecure-http … is rejected with unexpected argument. The endpoint and password above come from the environment, so only --insecure-http needs placing.

The archive is written to the current directory as lcf-callhome-<cluster>-<id>.tar.gz and its full path is printed when it finishes; nothing appears at that name until the whole download has succeeded, so an interrupted run cannot leave a truncated file to send by mistake. Useful options:

Option

Effect

--since <duration>

How far back to collect, e.g. 90m, 6h, 2d. Default 1h.

--start <unix-seconds> --end <unix-seconds>

Collect an exact window instead — an incident is usually bounded on both sides.

--component <name>

Collect logs only from that component, e.g. lcfc, lcft. Repeatable.

--split <duration>

Collect a long window as a series of bundles of that length, numbered in time order and named for the window each holds, each removed from the cluster once downloaded. Send all of them.

--out-file <path>

Write the archive somewhere other than the current directory.

--delete-remote

Remove the bundle from the cluster once it is downloaded. Off by default, so a failed download can be retried.

--no-redact

Collect without redaction. Only if Support asks for it.

Redaction is on by default, and it is not complete

Labeled fields that hold credentials or user identity — password, token, cookie, session, user, email and the like — are erased. Host names, IP addresses, pod names and tenants in labeled fields are replaced by stable pseudonyms (node-2), consistent across the whole bundle, so Support can still tell one node from another without learning which machine it is. Inside the free text of a log message, inline secrets (password=…, token=…, Bearer …), JWTs and IP addresses are removed; user names, e-mail addresses and host names that appear in a message's text are not. If your logs contain them, review the bundle before you send it. The namespace, device, and component inventory is kept, as that is what Support needs to read the bundle.

Bundles kept on the cluster

Each bundle is also kept on the cluster, so you can fetch it again:

lcf-cli obs bundle --insecure-http list
lcf-cli obs bundle --insecure-http download <id>
lcf-cli obs bundle --insecure-http delete <id>

They live on a volume of their own (lcf-observability.storage.callhome.size, default 4Gi), and after each new bundle the oldest are deleted once there are more than lcf-observability.callhome.maxBundles (default 10) or they total more than lcf-observability.callhome.maxTotalBytes (default 2 GiB). The newest bundle is never deleted. One bundle is built at a time — a second request while one is building is refused, so retry it — and a single window is limited to seven days (lcf-observability.callhome.maxWindowSecs).