Troubleshooting

This page applies to both Inferra deployment methods:

Install method

Where it is described

lcf-cli deploy

LCF-CLI Deployment

Direct Helm chart

Direct Helm Chart Deployment

If you are opening a support case, start with Collecting Diagnostics for Support below: one command collects what Lightbits support needs, in one file.

Use the other commands to monitor lcft, lcfc, Envoy, certificate issuance/rotation, and bootstrap JWT access after a deployment.

Set common variables first

export LCF_NAMESPACE=lcf
export LCF_RELEASE=lcf
# Optional. Leave KUBECONFIG and KUBE_CONTEXT unset if the current kubectl
# context already points at the target cluster.
# export KUBECONFIG=/path/to/kubeconfig.yaml
# export KUBE_CONTEXT=REPLACE_WITH_CONTEXT

KUBECTL=(kubectl)
HELM=(helm)

if [ -n "${KUBE_CONTEXT:-}" ]; then
KUBECTL=(kubectl --context "$KUBE_CONTEXT")
HELM=(helm --kube-context "$KUBE_CONTEXT")
fi
If you are not using the wrappers

The commands below use ${KUBECTL[@]} and ${HELM[@]}. For plain commands, replace ${KUBECTL[@]} with kubectl --context REPLACE_WITH_CONTEXT and ${HELM[@]} with helm --kube-context REPLACE_WITH_CONTEXT.

Collecting Diagnostics for Support

New in 1.2.0 When you open a support case, run this first. One command collects what Lightbits support needs to analyze the problem into a single archive on your machine, and you send that file. Nothing is uploaded: you decide whether and how to send it.

lcf-cli diag collect --namespace "$LCF_NAMESPACE"
# add --context "$KUBE_CONTEXT" if the current kubectl context is another cluster

It runs kubectl and helm as you, like lcf-cli deploy. It reads the LCF and vLLM namespaces, runs commands in their pods (kubectl exec), and starts a short-lived DaemonSet in the LCF namespace. It collects the last 6 hours (--since):

Part

What it holds

Kubernetes

Nodes, pods, events, describe output, the logs of every container and of its previous instance, Helm status and values, ConfigMaps, PVCs, storage classes, and the version of every image. Secret names only, never Secret contents.

Hosts

For each node: kernel messages, memory and hugepages, what backs the lcfc RAM tier and how much memory the lcfc and lcft processes hold, NVMe devices and /dataN mounts, RDMA, GPU, and the list of core dump files. To read them, a privileged pod runs read-only commands on every node (DaemonSet lcf-diag-host-<run id>, new for each run, image busybox:1.37, the same as the deploy device probe; --probe-image for your mirror). It is deleted at the end. --no-host skips this part.

Daemons

For each lcfc and lcft: its LCF_*, UCX_* and RUST_LOG* settings, read on the node from the running process, and what UCX finds in its container (ucx_info -d and ucx_info -c, run with kubectl exec). A variable set from a Kubernetes Secret keeps its name but not its value. If the command cannot list the LCF pods, every LCF_* variable keeps only its name.

vLLM

For each running vLLM pod in --vllm-namespace (default vllm): vllm collect-env, the vLLM, torch and LCF package versions, and its LCF_*, VLLM_*, UCX_*, NCCL_* and CUDA_* variables. A variable set from a Kubernetes Secret keeps its name but not its value.

Call-home

The observability stack's metrics and logs (Observability). The command reaches the service itself, through kubectl port-forward, and logs in with an API key the chart made for this (see Credentials below). If it cannot, this part is skipped with the reason, and the rest is still collected.

lcf-api

The cluster's own view: Raft and convergence, targets and the transport each connection uses, clients, devices, namespaces, rebalances, recent events, cluster params, and each node's configuration. The command finds lcf-api itself and logs in with the lcf-api bootstrap token (see Credentials below).

The prompt. The command first prints exactly what it will collect, then asks Continue? [y/N]. --dry-run prints the plan and stops. In a script, or anywhere without a terminal, pass --yes: without it the command collects nothing.

Credentials. To log in, the command reads two values from Secrets in the LCF namespace: the observability API key the chart made for diagnostics (Secret lcf-observability-diag-key) and the lcf-api bootstrap token (Secret lcf-bootstrap-token). It reads only that one key of each, uses it only to log in, and never writes it to the archive or to the screen. The plan names both Secrets. --no-cluster-credentials turns this off. The bootstrap token is used only with the lcf-api the command finds itself: if LCF_ENDPOINT is set (or --endpoint, LCF_CA_FILE, --insecure-skip-tls-verify), also set LCF_TOKEN or LCF_TOKEN_FILE, or the lcf-api part is skipped.

The API key can only create, download and delete call-home bundles: it opens no dashboard, query or user setting, and changing the admin password does not affect it. To revoke it, or to make a new one, delete the Secret and upgrade the release:

"${KUBECTL[@]}" -n "$LCF_NAMESPACE" delete secret lcf-observability-diag-key
# then run your usual `lcf-cli deploy` or `helm upgrade` for the release

The observability service restarts to take the new key. To use the admin password instead, export LCF_OBS_PASSWORD.

Reaching the observability service. The command reaches it with kubectl port-forward to the lcf-observability-ui Service, and stops the port-forward at the end. The traffic goes through the Kubernetes API server's encrypted connection, so nothing is sent in clear, and your machine does not need to reach the nodes. You need permission to port-forward in the LCF namespace (pods/portforward). Without it, call-home is skipped and the summary says why. To use another address instead, set LCF_OBS_ENDPOINT; an http:// address also needs --insecure-http, so use one only on a trusted network.

The bootstrap token lasts 90 days after the install. On an older install, give a current admin token in LCF_TOKEN to include the lcf-api view.

Redaction. There are three levels. --redact removes everything the default removes, and more.

Level

Option

What is removed

secrets

(default)

Passwords, tokens, API keys, JWTs, private keys, credentials in URLs, and webhook URLs (Slack, Discord, Teams, Google Chat). Host names, IP addresses and tenant ids are kept, because support needs them to read the bundle.

full

--redact

Also replaces host and tenant names with pseudonyms (node-1, tenant-1) and masks IP addresses.

none

--no-redact

Nothing. Use it only when Lightbits support asks for it. The call-home part stays at secrets unless you also give the admin password (LCF_OBS_PASSWORD): the diagnostics API key may not have credentials kept.

Redaction is best effort, not a guarantee. If that matters at your site, review the files before you send them. Logs can also hold the text of prompts: vLLM logs its requests when request logging is on, and no level removes them. If your prompts must not leave your site, check the vLLM logs in the archive (namespaces/<vLLM namespace>/logs/) before you send it.

The key file. With --redact, the command also writes lcf-diag-<cluster>-<time>.key.json next to the archive, readable only by you (mode 0600). It maps each pseudonym to the real name. Keep it, and do not send it. When support asks "which machine is node-2?", look it up there and answer.

Options.

Option

Effect

--since <duration>

How far back logs, kernel messages and call-home go: 90m, 6h, 2d. Default 6h.

--vllm-namespace <namespace>

The namespace of your vLLM pods. Repeat it for several. Default vllm.

--output-dir <dir>

Where to write the archive. Default: the current directory. It must have room for --max-size, twice that when call-home runs (its download waits there), plus --cores-max-size with --include-cores.

--nvidia-bug-report

Also run NVIDIA's nvidia-bug-report.sh on each node that has it. Slow (up to 10 minutes) and large. NVIDIA's report is copied exactly as the tool writes it, so it is not redacted, at any level: it holds the node's host name, addresses, and kernel and system logs.

--include-cores

Also copy the core dump files of the last 24 hours (--cores-since) into a second archive, lcf-diag-<cluster>-<time>-cores.tar.gz, up to 8 GiB (--cores-max-size). A core is a copy of a process's memory, so it is not redacted, at any level. It can hold mTLS private keys, the lcf-api JWT signing key, short-lived copies of encryption keys and other data held in memory. Cached KV data is left out. Send it only when support asks for it.

--no-host, --no-vllm, --no-callhome

Skip that part.

--no-cluster-credentials

Do not read the two Secrets above. Call-home then needs LCF_OBS_PASSWORD (the admin password) or LCF_OBS_API_KEY, and the lcf-api view LCF_TOKEN.

--max-size, --log-max

Size caps: 1 GiB for the whole archive and 50 MiB per file. A file cut to fit keeps its newest part.

--log-head-max

How much of the start of each lcfc, lcft, lcf-api and vLLM log to keep as well, when the container started before the window: what it logged at startup. Default 4 MiB; 0 keeps none.

--timeout <duration>

A limit for the whole run. Default 30m.

To stop a run, press Ctrl-C: the command stops, deletes its host DaemonSet, and writes no archive. Only if lcf-cli itself is killed outright does the DaemonSet stay; its pods then stop working when the run's --timeout is up, and you delete it with the command the plan printed: "${KUBECTL[@]}" -n "$LCF_NAMESPACE" delete daemonset lcf-diag-host-<run id>. Each run has its own DaemonSet, so a later run does not remove it. To list what such runs left: "${KUBECTL[@]}" -n "$LCF_NAMESPACE" get daemonset -l app.kubernetes.io/name=lcf-diag-host.

Exit codes.

Code

Meaning

0

Everything was collected.

2

The archive was written, but some parts failed. SUMMARY.txt inside it says which and why. Send it anyway.

1

No archive was written: an option was wrong, --output-dir had too little free space, the prompt was answered no, or there was no terminal and no --yes.

130

You stopped it with Ctrl-C (another signal gives 128 plus its number). No archive was written.

What to send. At the end, the command prints the archive's path, its size and its SHA-256. The archive, the cores archive and the key file are readable only by you (mode 0600). Send the lcf-diag-<cluster>-<time>.tar.gz file to Lightbits support, with its SHA-256. Do not send the .key.json file. Send the -cores.tar.gz file only if support asks for it.

lcf-cli diag list shows the archives in a directory; lcf-cli diag rm <name> deletes one, with its key file and cores archive.

Workload Inventory

"${KUBECTL[@]}" -n "$LCF_NAMESPACE" get pods,svc,jobs -o wide
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" get pods -l app=lcft \
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.nodeName}{"\t"}{.status.phase}{"\n"}{end}'
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" get pods -l app.kubernetes.io/name=lcfc \
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.nodeName}{"\t"}{.status.phase}{"\n"}{end}'
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" rollout status daemonset/lcft --timeout=10m
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" rollout status daemonset/lcfc --timeout=20m
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" rollout status deployment/lcf-api --timeout=10m
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" rollout status deployment/step-ca --timeout=10m

LCFT Logs

lcft pods have two containers: lcft and envoy.

"${KUBECTL[@]}" -n "$LCF_NAMESPACE" logs -l app=lcft -c lcft -f --tail=200
LCFT_POD=$("${KUBECTL[@]}" -n "$LCF_NAMESPACE" get pod -l app=lcft \
-o jsonpath='{.items[0].metadata.name}')
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" logs "$LCFT_POD" -c lcft --tail=200
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" logs "$LCFT_POD" -c lcft --previous --tail=200
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" describe pod "$LCFT_POD"
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" get events --sort-by=.lastTimestamp

LCFC Logs

lcfc pods have two containers: lcfc and envoy.

"${KUBECTL[@]}" -n "$LCF_NAMESPACE" logs -l app.kubernetes.io/name=lcfc -c lcfc -f --tail=200
LCFC_POD=$("${KUBECTL[@]}" -n "$LCF_NAMESPACE" get pod -l app.kubernetes.io/name=lcfc \
-o jsonpath='{.items[0].metadata.name}')
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" logs "$LCFC_POD" -c lcfc --tail=200
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" logs "$LCFC_POD" -c lcfc --previous --tail=200
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" describe pod "$LCFC_POD"

Raising Daemon Log Levels at Runtime

New in 1.1.0 lcf-cli log changes the log level of the running lcfc or lcft daemons on every node of a component, the way RUST_LOG would at start, without a restart. Every directive carries a mandatory --ttl and a --reason; the daemon reverts on its own when the TTL passes. set and reset need the ADMIN role, and each one is written to the audit trail. It uses the same connection settings as every other lcf-cli command (LCF-CLI Deployment, Step 7).

lcf-cli log set --component lcft lcft::coordinator=debug --ttl 30m --reason "support case"
lcf-cli log get --component lcft --nodes
lcf-cli log list
lcf-cli log reset --component lcft

Directives are written exactly as in RUST_LOG — target=level, or a bare level for the whole daemon. lcf-cli log targets lists the named diagnostic targets.

Raise levels only when Lightbits Support asks

Reset them when you are done: debug output on a busy node is large.

Envoy Logs

Envoy logs include mTLS ingress, JWT-authenticated API ingress, upstream routing, response codes, and latency fields.

"${KUBECTL[@]}" -n "$LCF_NAMESPACE" logs -l app=lcft -c envoy -f --tail=200
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" logs -l app.kubernetes.io/name=lcfc -c envoy -f --tail=200
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" logs -l app=lcf-api -c envoy -f --tail=200

Look for non-2xx responses in recent Envoy logs:

"${KUBECTL[@]}" -n "$LCF_NAMESPACE" logs -l app=lcf-api -c envoy --tail=500 | \
grep -E '"response_code":(4|5)[0-9][0-9]'

Envoy Admin Readiness and Certificate Expiry

No curl inside the Envoy image

The Envoy images may not include curl or wget. Use kubectl port-forward from the operator workstation to query the admin listener.

Admin ports:

Workload

Container

Admin port

lcft

envoy

9901

lcfc

envoy

19901

lcf-api

envoy

9901

LCFT_POD=$("${KUBECTL[@]}" -n "$LCF_NAMESPACE" get pod -l app=lcft \
-o jsonpath='{.items[0].metadata.name}')
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" port-forward "pod/${LCFT_POD}" 19991:9901 >/tmp/lcft-envoy-port-forward.log 2>&1 & PF_PID=$!
sleep 2
curl -fsS http://127.0.0.1:19991/ready
curl -fsS http://127.0.0.1:19991/stats | grep days_until_first_cert_expiring || true
kill "$PF_PID"
LCFC_POD=$("${KUBECTL[@]}" -n "$LCF_NAMESPACE" get pod -l app.kubernetes.io/name=lcfc \
-o jsonpath='{.items[0].metadata.name}')
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" port-forward "pod/${LCFC_POD}" 19902:19901 >/tmp/lcfc-envoy-port-forward.log 2>&1 & PF_PID=$!
sleep 2
curl -fsS http://127.0.0.1:19902/ready
curl -fsS http://127.0.0.1:19902/stats | grep days_until_first_cert_expiring || true
kill "$PF_PID"
API_POD=$("${KUBECTL[@]}" -n "$LCF_NAMESPACE" get pod -l app=lcf-api \
-o jsonpath='{.items[0].metadata.name}')
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" port-forward "pod/${API_POD}" 19903:9901 >/tmp/lcf-api-envoy-port-forward.log 2>&1 & PF_PID=$!
sleep 2
curl -fsS http://127.0.0.1:19903/ready
curl -fsS http://127.0.0.1:19903/stats | grep days_until_first_cert_expiring || true
kill "$PF_PID"

Expected readiness output is LIVE. The days_until_first_cert_expiring stat is relevant when present; absence means Envoy has not emitted that stat on the queried listener yet, so use the cert-manager Certificate status and Secret inspection commands below as the source of truth.

Certificate Status and Rotation

List certificate resources:

"${KUBECTL[@]}" -n "$LCF_NAMESPACE" get certificates,certificaterequests

Show readiness, renewal time, and expiry for every LCF certificate:

for cert in lcft-tls lcfc-tls lcf-api-tls; do
printf '%s\t' "$cert"
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" get certificate "$cert" \
-o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{"\t"}{.status.renewalTime}{"\t"}{.status.notAfter}{"\n"}'
done

Inspect the certificate spec, including duration and renewal policy:

"${KUBECTL[@]}" -n "$LCF_NAMESPACE" describe certificate lcft-tls
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" describe certificate lcfc-tls
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" describe certificate lcf-api-tls

Inspect the actual issued certificate from Kubernetes Secrets:

"${KUBECTL[@]}" -n "$LCF_NAMESPACE" get secret lcft-tls \
-o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject -issuer -dates
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" get secret lcfc-tls \
-o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject -issuer -dates
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" get secret lcf-api-tls \
-o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject -issuer -dates

Check which issuer each certificate references:

for cert in lcft-tls lcfc-tls lcf-api-tls; do
printf '%s\t' "$cert"
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" get certificate "$cert" \
-o jsonpath='{.spec.issuerRef.kind}{"\t"}{.spec.issuerRef.name}{"\n"}'
done

Check the expected shared StepClusterIssuer:

ISSUER_NAME="${LCF_RELEASE}-${LCF_NAMESPACE}-step-issuer"
"${KUBECTL[@]}" get stepclusterissuer "$ISSUER_NAME" -o yaml

If the Certificate resources are not Ready or the StepClusterIssuer is missing, inspect the bootstrap job and controller logs:

"${KUBECTL[@]}" -n "$LCF_NAMESPACE" get jobs -l app=step-issuer-bootstrap
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" logs -l app=step-issuer-bootstrap --tail=200
"${KUBECTL[@]}" -n "$LCF_NAMESPACE" logs deployment/step-issuer --tail=200
"${KUBECTL[@]}" -n cert-manager logs deployment/cert-manager --tail=200

Watch for certificate renewal events:

"${KUBECTL[@]}" -n "$LCF_NAMESPACE" get events --sort-by=.lastTimestamp | grep -E 'Certificate|CertificateRequest|Issuer|Renew|Ready'

Confirm the Helm values that drive certificate duration and renewal window:

"${HELM[@]}" get values "$LCF_RELEASE" -n "$LCF_NAMESPACE" -o yaml | grep -A5 '^global:'
"${HELM[@]}" get manifest "$LCF_RELEASE" -n "$LCF_NAMESPACE" | grep -A20 'kind: Certificate'

After cert-manager rotates a certificate Secret, Envoy should reload it from the watched /certs directory. Confirm by checking Envoy logs and the days_until_first_cert_expiring stat through the admin endpoint commands above.

Fetch Bootstrap JWT and Root CA

Fetch the bootstrap admin JWT:

"${KUBECTL[@]}" -n "$LCF_NAMESPACE" get secret lcf-bootstrap-token \
-o jsonpath='{.data.token}' | base64 -d > ./lcf-bootstrap-token.jwt
chmod 600 ./lcf-bootstrap-token.jwt

Check the token starts with a JWT header:

cut -c1-20 ./lcf-bootstrap-token.jwt

Check when the bootstrap token Secret was created:

"${KUBECTL[@]}" -n "$LCF_NAMESPACE" get secret lcf-bootstrap-token \
-o jsonpath='{.metadata.creationTimestamp}{"\n"}'

Fetch the API trust bundle from the issued API certificate Secret:

"${KUBECTL[@]}" -n "$LCF_NAMESPACE" get secret lcf-api-tls \
-o jsonpath='{.data.ca\.crt}' | base64 -d > ./lcf-root-ca.pem

When the deployment has a populated StepClusterIssuer.spec.caBundle, this equivalent command also works:

ISSUER_NAME="${LCF_RELEASE}-${LCF_NAMESPACE}-step-issuer"
"${KUBECTL[@]}" get stepclusterissuer "$ISSUER_NAME" \
-o jsonpath='{.spec.caBundle}' | base64 -d > ./lcf-root-ca.pem

Discover the external NodePort endpoint:

API_NODE=$("${KUBECTL[@]}" get nodes -o jsonpath='{.items[0].metadata.name}')
API_IP=$("${KUBECTL[@]}" get node "$API_NODE" \
-o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}')
API_PORT=$("${KUBECTL[@]}" -n "$LCF_NAMESPACE" get svc lcf-api-external \
-o jsonpath='{.spec.ports[?(@.name=="grpc-jwt")].nodePort}')
export LCF_ENDPOINT="https://${API_IP}:${API_PORT}"
export LCF_TOKEN_FILE="${PWD}/lcf-bootstrap-token.jwt"
export LCF_CA_FILE="${PWD}/lcf-root-ca.pem"

Verify CLI access:

lcf-cli --endpoint "$LCF_ENDPOINT" \
--token-file "$LCF_TOKEN_FILE" \
--ca-file "$LCF_CA_FILE" \
list targets
lcf-cli --endpoint "$LCF_ENDPOINT" \
--token-file "$LCF_TOKEN_FILE" \
--ca-file "$LCF_CA_FILE" \
list clients
Lab-only shortcut

Use --insecure-skip-tls-verify only for lab troubleshooting. Production clients should use --ca-file and a server endpoint covered by lcf-api.certificate.ipAddresses or lcf-api.certificate.dnsNames.