Troubleshooting
This page applies to both Inferra deployment methods:
Install method | Where it is described |
|---|---|
| |
Direct Helm chart |
If you are opening a support case, start with Collecting Diagnostics for Support below: one command collects what Lightbits support needs, in one file.
Use the other commands to monitor lcft, lcfc, Envoy, certificate issuance/rotation, and bootstrap JWT access after a deployment.
Set common variables first
The commands below use ${KUBECTL[@]} and ${HELM[@]}. For plain commands, replace ${KUBECTL[@]} with kubectl --context REPLACE_WITH_CONTEXT and ${HELM[@]} with helm --kube-context REPLACE_WITH_CONTEXT.
Collecting Diagnostics for Support
New in 1.2.0 When you open a support case, run this first. One command collects what Lightbits support needs to analyze the problem into a single archive on your machine, and you send that file. Nothing is uploaded: you decide whether and how to send it.
It runs kubectl and helm as you, like lcf-cli deploy. It reads the LCF and vLLM namespaces, runs commands in their pods (kubectl exec), and starts a short-lived DaemonSet in the LCF namespace. It collects the last 6 hours (--since):
Part | What it holds |
|---|---|
Kubernetes | Nodes, pods, events, |
Hosts | For each node: kernel messages, memory and hugepages, what backs the |
Daemons | For each |
vLLM | For each running vLLM pod in |
Call-home | The observability stack's metrics and logs (Observability). The command reaches the service itself, through |
| The cluster's own view: Raft and convergence, targets and the transport each connection uses, clients, devices, namespaces, rebalances, recent events, cluster params, and each node's configuration. The command finds |
The prompt. The command first prints exactly what it will collect, then asks Continue? [y/N]. --dry-run prints the plan and stops. In a script, or anywhere without a terminal, pass --yes: without it the command collects nothing.
Credentials. To log in, the command reads two values from Secrets in the LCF namespace: the observability API key the chart made for diagnostics (Secret lcf-observability-diag-key) and the lcf-api bootstrap token (Secret lcf-bootstrap-token). It reads only that one key of each, uses it only to log in, and never writes it to the archive or to the screen. The plan names both Secrets. --no-cluster-credentials turns this off. The bootstrap token is used only with the lcf-api the command finds itself: if LCF_ENDPOINT is set (or --endpoint, LCF_CA_FILE, --insecure-skip-tls-verify), also set LCF_TOKEN or LCF_TOKEN_FILE, or the lcf-api part is skipped.
The API key can only create, download and delete call-home bundles: it opens no dashboard, query or user setting, and changing the admin password does not affect it. To revoke it, or to make a new one, delete the Secret and upgrade the release:
The observability service restarts to take the new key. To use the admin password instead, export LCF_OBS_PASSWORD.
Reaching the observability service. The command reaches it with kubectl port-forward to the lcf-observability-ui Service, and stops the port-forward at the end. The traffic goes through the Kubernetes API server's encrypted connection, so nothing is sent in clear, and your machine does not need to reach the nodes. You need permission to port-forward in the LCF namespace (pods/portforward). Without it, call-home is skipped and the summary says why. To use another address instead, set LCF_OBS_ENDPOINT; an http:// address also needs --insecure-http, so use one only on a trusted network.
The bootstrap token lasts 90 days after the install. On an older install, give a current admin token in LCF_TOKEN to include the lcf-api view.
Redaction. There are three levels. --redact removes everything the default removes, and more.
Level | Option | What is removed |
|---|---|---|
| (default) | Passwords, tokens, API keys, JWTs, private keys, credentials in URLs, and webhook URLs (Slack, Discord, Teams, Google Chat). Host names, IP addresses and tenant ids are kept, because support needs them to read the bundle. |
|
| Also replaces host and tenant names with pseudonyms ( |
|
| Nothing. Use it only when Lightbits support asks for it. The call-home part stays at |
Redaction is best effort, not a guarantee. If that matters at your site, review the files before you send them. Logs can also hold the text of prompts: vLLM logs its requests when request logging is on, and no level removes them. If your prompts must not leave your site, check the vLLM logs in the archive (namespaces/<vLLM namespace>/logs/) before you send it.
The key file. With --redact, the command also writes lcf-diag-<cluster>-<time>.key.json next to the archive, readable only by you (mode 0600). It maps each pseudonym to the real name. Keep it, and do not send it. When support asks "which machine is node-2?", look it up there and answer.
Options.
Option | Effect |
|---|---|
| How far back logs, kernel messages and call-home go: |
| The namespace of your vLLM pods. Repeat it for several. Default |
| Where to write the archive. Default: the current directory. It must have room for |
| Also run NVIDIA's |
| Also copy the core dump files of the last 24 hours ( |
| Skip that part. |
| Do not read the two Secrets above. Call-home then needs |
| Size caps: 1 GiB for the whole archive and 50 MiB per file. A file cut to fit keeps its newest part. |
| How much of the start of each |
| A limit for the whole run. Default |
To stop a run, press Ctrl-C: the command stops, deletes its host DaemonSet, and writes no archive. Only if lcf-cli itself is killed outright does the DaemonSet stay; its pods then stop working when the run's --timeout is up, and you delete it with the command the plan printed: "${KUBECTL[@]}" -n "$LCF_NAMESPACE" delete daemonset lcf-diag-host-<run id>. Each run has its own DaemonSet, so a later run does not remove it. To list what such runs left: "${KUBECTL[@]}" -n "$LCF_NAMESPACE" get daemonset -l app.kubernetes.io/name=lcf-diag-host.
Exit codes.
Code | Meaning |
|---|---|
| Everything was collected. |
| The archive was written, but some parts failed. |
| No archive was written: an option was wrong, |
| You stopped it with Ctrl-C (another signal gives 128 plus its number). No archive was written. |
What to send. At the end, the command prints the archive's path, its size and its SHA-256. The archive, the cores archive and the key file are readable only by you (mode 0600). Send the lcf-diag-<cluster>-<time>.tar.gz file to Lightbits support, with its SHA-256. Do not send the .key.json file. Send the -cores.tar.gz file only if support asks for it.
lcf-cli diag list shows the archives in a directory; lcf-cli diag rm <name> deletes one, with its key file and cores archive.
Workload Inventory
LCFT Logs
lcft pods have two containers: lcft and envoy.
LCFC Logs
lcfc pods have two containers: lcfc and envoy.
Raising Daemon Log Levels at Runtime
New in 1.1.0
lcf-cli log changes the log level of the running lcfc or lcft daemons on every node of a component, the way RUST_LOG would at start, without a restart. Every directive carries a mandatory --ttl and a --reason; the daemon reverts on its own when the TTL passes. set and reset need the ADMIN role, and each one is written to the audit trail. It uses the same connection settings as every other lcf-cli command (LCF-CLI Deployment, Step 7).
Directives are written exactly as in RUST_LOG — target=level, or a bare level for the whole daemon. lcf-cli log targets lists the named diagnostic targets.
Reset them when you are done: debug output on a busy node is large.
Envoy Logs
Envoy logs include mTLS ingress, JWT-authenticated API ingress, upstream routing, response codes, and latency fields.
Look for non-2xx responses in recent Envoy logs:
Envoy Admin Readiness and Certificate Expiry
The Envoy images may not include curl or wget. Use kubectl port-forward from the operator workstation to query the admin listener.
Admin ports:
Workload | Container | Admin port |
|---|---|---|
|
|
|
|
|
|
|
|
|
Expected readiness output is LIVE. The days_until_first_cert_expiring stat is relevant when present; absence means Envoy has not emitted that stat on the queried listener yet, so use the cert-manager Certificate status and Secret inspection commands below as the source of truth.
Certificate Status and Rotation
List certificate resources:
Show readiness, renewal time, and expiry for every LCF certificate:
Inspect the certificate spec, including duration and renewal policy:
Inspect the actual issued certificate from Kubernetes Secrets:
Check which issuer each certificate references:
Check the expected shared StepClusterIssuer:
If the Certificate resources are not Ready or the StepClusterIssuer is missing, inspect the bootstrap job and controller logs:
Watch for certificate renewal events:
Confirm the Helm values that drive certificate duration and renewal window:
After cert-manager rotates a certificate Secret, Envoy should reload it from the watched /certs directory. Confirm by checking Envoy logs and the days_until_first_cert_expiring stat through the admin endpoint commands above.
Fetch Bootstrap JWT and Root CA
Fetch the bootstrap admin JWT:
Check the token starts with a JWT header:
Check when the bootstrap token Secret was created:
Fetch the API trust bundle from the issued API certificate Secret:
When the deployment has a populated StepClusterIssuer.spec.caBundle, this equivalent command also works:
Discover the external NodePort endpoint:
Verify CLI access:
Use --insecure-skip-tls-verify only for lab troubleshooting. Production clients should use --ca-file and a server endpoint covered by lcf-api.certificate.ipAddresses or lcf-api.certificate.dnsNames.