LCF-CLI Deployment

lcf-cli deploy is the recommended way to install Inferra. It discovers the cluster, classifies nodes, sizes resources, generates Helm values, applies node labels, runs Helm, and writes the post-install access files for you.

Before you start

Kubernetes, Helm, and kubectl must already be installed and operational — see Prerequisites. If you prefer to drive Helm yourself, use Direct Helm Chart Deployment instead.

Download lcf-cli

export TOKEN=<CLOUDSMITH_TOKEN>
export VERSION=1.2.0
export SEMVER=1.2.0
export ARCH=x86_64 # or arm64
curl -fL -o lcf-cli "https://dl.lightbitslabs.com/${TOKEN}/lightbits-inferra/raw/names/lcf-cli-${ARCH}/versions/${VERSION}/lcf-cli-v${SEMVER}-linux-${ARCH}"
chmod +x lcf-cli
Two version forms, one URL

The download URL carries the version twice: VERSION is the PEP 440 form (1.2.0, in the versions/ segment) and SEMVER is the SemVer form (1.2.0, inside the filename). Mixing them up produces a 404.

What lcf-cli Deploy Does

Stage

Behavior

Preflight

Verifies Helm, kubectl, Kubernetes version, namespace write access, registry pull access, network pull bandwidth, and container image storage.

Discovery

Reads Kubernetes nodes, versions, labels, taints, addresses, GPU count, memory, CPU, and existing release state.

Device probe

Runs a temporary privileged probe DaemonSet to inspect host storage inventory and mounted /data* paths when --probe-devices=true.

Classification

Chooses storage, inference, dual-role, or excluded node roles. Existing LCF labels override heuristics.

Sizing

Computes CPU, memory, and cpuset recommendations. Manual mode lets the operator edit memory limits.

Values

Writes generated Helm values under the deploy config directory.

Labels

Applies node-role.kubernetes.io/storage=true and node-role.kubernetes.io/inference=true based on the plan.

Helm

Runs helm upgrade --install --wait.

Post-install

Reads the bootstrap JWT, root CA, API exposure, and writes access files.

Prerequisites

Requirement

Minimum

Notes

lcf-cli

The binary from this release's download URL

The examples use the deploy subcommand built into lcf-cli. lcf-cli --version does not report the release version — it prints an internal build version and will not confirm which release you fetched. The download URL above is what pins it, so re-download rather than trusting a binary already on PATH.

Kubernetes

1.28

Enforced by deploy preflight.

Helm

3.x

Used by deploy to run the chart.

kubectl

Compatible with the cluster

Used by deploy for discovery, labels, probes, and post-install.

Cloudsmith credential

Read access

The examples use username token and password $CLOUDSMITH_TOKEN.

Storage nodes

Pre-mounted XFS data paths

lcft uses host directories such as /data0, /data1, and /dataN.

Inference nodes

GPU nodes or CPU inference nodes

lcfc runs on inference-labeled nodes.

2 MiB huge pages

Recommended on every inference and dual-role node

vm.nr_hugepages sized for the RAM tier you want, applied before the kubelet starts. Not required — the tier falls back to smaller pages — but it is what makes the tier fast and quick to come up. See The lcfc memory limit is the RAM-tier size in Step 5.

Set common variables:

export LCF_NAMESPACE=lcf
export LCF_RELEASE=lcf
export LCF_DEPLOYMENT_NAME=inferra-prod
# SemVer form of the release. Helm chart versions and container image tags use
# it. This is NOT the PEP 440 form (1.2.0) used in the lcf-cli download URL
# above; that form is only valid for Cloudsmith raw and Python packages.
export LCF_VERSION=1.2.0
export LCF_IMAGE_TAG=v${LCF_VERSION}
export CLOUDSMITH_REGISTRY=docker.cloudsmith.io
export CLOUDSMITH_REPO=lightbits/lightbits-inferra
export CLOUDSMITH_TOKEN='REPLACE_WITH_CLOUDSMITH_READ_TOKEN'
# lcf-cli reads the registry token for the image pull secret from this variable.
# Keep the token off the command line: an argument such as --pull-secret-token
# is visible to other users of the host in the process list while it runs.
export LCF_REGISTRY_TOKEN="$CLOUDSMITH_TOKEN"
export LCF_CHART_OCI="oci://${CLOUDSMITH_REGISTRY}/${CLOUDSMITH_REPO}/charts/lcf"

Authenticate Helm to Cloudsmith and pull the exact chart version.

helm registry login does not work with an entitlement token

The entitlement token that Lightbits Support issues you is accepted for basic authentication but is rejected by the OCI login handshake, which fails with Error: Get "https://docker.cloudsmith.io/v2/": unauthorized. Write the credential into a registry configuration file of its own and point helm pull at it — the pull then succeeds with the same token, and your existing Helm registry logins are left untouched. If Support issued you a Cloudsmith API key, which is a different credential, helm registry login --username token --password "$CLOUDSMITH_API_KEY" does work and you can use it in place of the first two commands below.

export INFERRA_HELM_REGISTRY_CONFIG="${PWD}/inferra-helm-registry.json"
rm -f "$INFERRA_HELM_REGISTRY_CONFIG"
(umask 077 && cat > "$INFERRA_HELM_REGISTRY_CONFIG" <<EOF
{"auths":{"${CLOUDSMITH_REGISTRY}":{"auth":"$(printf 'token:%s' "$CLOUDSMITH_TOKEN" | base64 -w0)"}}}
EOF
)
mkdir -p "$LCF_CHART_DIR"
helm pull "$LCF_CHART_OCI" \
--registry-config "$INFERRA_HELM_REGISTRY_CONFIG" \
--version "$LCF_VERSION" \
--destination "$LCF_CHART_DIR"
test -s "$LCF_CHART"

That file holds your token in clear text, so it is created readable only by you — the umask 077 above. The rm -f before it matters when you rerun this step: umask applies only when a file is created, so writing into a copy left from an earlier run would keep that copy's permissions. Remove the file when you are done installing, with rm -f "$INFERRA_HELM_REGISTRY_CONFIG".

Do not write the credential into ~/.config/helm/registry/config.json

That replaces every registry login Helm already holds for your account.

Create the namespace before running deploy discover in the final installation namespace:

kubectl create namespace "$LCF_NAMESPACE" \
--dry-run=client -o yaml | kubectl apply -f -

Step 1: Discover the Cluster

Run discovery first. JSON output is useful for reviewing and changing records.

lcf-cli -o json deploy discover \
--namespace "$LCF_NAMESPACE" \
--probe-devices true \
--probe-timeout 180 > inferra-discovery.json
lcf-cli deploy discover \
--namespace "$LCF_NAMESPACE" \
--probe-devices true \
--probe-timeout 180

Classification rules used by deploy

Rules are evaluated in order, and the first match wins.

Condition

Role

Node is unschedulable (cordoned)

Excluded

Control-plane node tainted NoExecute (node-role.kubernetes.io/control-plane or master)

Excluded

Node already has storage and inference labels

Dual role

Node already has storage label

Storage

Node already has inference label

Inference

Control-plane node tainted NoSchedule (node-role.kubernetes.io/control-plane or master) without explicit LCF label

Excluded

Only node LCF can run on, with no GPU node elsewhere in the cluster

Dual role

GPU and usable local storage

Dual role

GPU without usable local storage

Inference

Usable local storage without GPU

Storage

No GPU and no usable storage

Excluded

Cordoning and NoExecute win over labels

Note the first two rows: a cordoned node and a NoExecute control-plane node are excluded before any label is read, so neither is rescued by an explicit LCF label.

Only the control-plane taint excludes a node

The control-plane role label alone does not. An untainted control-plane node — the usual shape for an evaluation or a single-node proof-of-concept — is classified by its hardware like any other node, and as the only node LCF can run on it is promoted to dual role. A control-plane node tainted NoSchedule is excluded unless you label it: the label rows above its row are the way in (see Step 2). lcft and lcfc do not tolerate NoExecute, so a NoExecute control-plane node is excluded even when labeled; remove that taint before deploying to it.

Step 2: Pre-Labeling — Required on a Tainted Control-Plane Node

lcf-cli deploy install can apply labels automatically, and on a cluster of ordinary worker nodes you can skip this step. Pre-label nodes when the deployment policy must override the hardware heuristics, and always in these two cases:

  • The node you are deploying to is a tainted control-plane node. Without an explicit LCF label it is excluded, whatever its hardware — including when it is the only node in the cluster. For a NoSchedule taint, label it as shown below. For a NoExecute taint, remove the taint first: a label alone does not let lcft or lcfc run there.

  • A node is cordoned. No label rescues it; the unschedulable row wins first. Uncordon it before discovery.

kubectl label node storage-01 node-role.kubernetes.io/storage=true --overwrite
kubectl label node storage-02 node-role.kubernetes.io/storage=true --overwrite
kubectl label node storage-03 node-role.kubernetes.io/storage=true --overwrite
kubectl label node gpu-01 node-role.kubernetes.io/inference=true --overwrite
kubectl label node gpu-02 node-role.kubernetes.io/inference=true --overwrite
kubectl label node single-node-01 \
node-role.kubernetes.io/storage=true \
node-role.kubernetes.io/inference=true \
--overwrite

Verify labels:

kubectl get nodes \
-L node-role.kubernetes.io/storage,node-role.kubernetes.io/inference

Step 3: Prepare XFS Mount Points

The supported storage model is mount point mode. Prepare XFS filesystems and mount them on the host before running lcf-cli deploy. The deploy command should see storage as /data0, /data1, and /dataN mount points. Formatting is not performed by lcf-cli deploy — it is an operator-controlled host preparation step.

From 1.2.0 a disk of any type — NVMe, SATA, or a virtual disk — is accepted as storage when it is mounted only at a /dataN path; raw devices that are not mounted are still selected only if they are NVMe. If the generated values would leave lcft with no storage at all, the install stops before anything is installed and names the reason for each node: mount a dedicated disk at /data0, or set lcft.storage.* explicitly with --set or --values.

Destructive formatting

mkfs.xfs -f deletes data on the selected device. Run it only on devices dedicated to LCF.

Example for one storage node with three LCF data devices:

# Run on storage-01, then repeat on every storage node with that node's devices.
sudo lsblk -o NAME,TYPE,SIZE,FSTYPE,MOUNTPOINT,MODEL,SERIAL
DEV0=/dev/disk/by-id/REPLACE_WITH_STORAGE_DEVICE_0
DEV1=/dev/disk/by-id/REPLACE_WITH_STORAGE_DEVICE_1
DEV2=/dev/disk/by-id/REPLACE_WITH_STORAGE_DEVICE_2
sudo mkfs.xfs -f "$DEV0"
sudo mkfs.xfs -f "$DEV1"
sudo mkfs.xfs -f "$DEV2"
sudo mkdir -p /data0 /data1 /data2
sudo mount -o noatime,nodiratime "$DEV0" /data0
sudo mount -o noatime,nodiratime "$DEV1" /data1
sudo mount -o noatime,nodiratime "$DEV2" /data2
findmnt -no SOURCE,TARGET,FSTYPE,OPTIONS /data0 /data1 /data2

Persist the mounts across reboot:

DEV0=/dev/disk/by-id/REPLACE_WITH_STORAGE_DEVICE_0
DEV1=/dev/disk/by-id/REPLACE_WITH_STORAGE_DEVICE_1
DEV2=/dev/disk/by-id/REPLACE_WITH_STORAGE_DEVICE_2
UUID0=$(sudo blkid -s UUID -o value "$DEV0")
UUID1=$(sudo blkid -s UUID -o value "$DEV1")
UUID2=$(sudo blkid -s UUID -o value "$DEV2")
printf 'UUID=%s /data0 xfs defaults,noatime,nodiratime,nofail 0 2\n' "$UUID0" | sudo tee -a /etc/fstab
printf 'UUID=%s /data1 xfs defaults,noatime,nodiratime,nofail 0 2\n' "$UUID1" | sudo tee -a /etc/fstab
printf 'UUID=%s /data2 xfs defaults,noatime,nodiratime,nofail 0 2\n' "$UUID2" | sudo tee -a /etc/fstab
sudo mount -a
findmnt -no SOURCE,TARGET,FSTYPE /data0 /data1 /data2

Run discovery after the mount points exist:

lcf-cli deploy discover \
--namespace "$LCF_NAMESPACE" \
--probe-devices true \
--probe-timeout 180

Manual mode is recommended for first production installs because it lets operators confirm roles, LCFT devices, and memory limits before Helm runs.

lcf-cli deploy install \
--mode manual \
--namespace "$LCF_NAMESPACE" \
--release "$LCF_RELEASE" \
--deployment-name "$LCF_DEPLOYMENT_NAME" \
--chart "$LCF_CHART" \
--registry "$LCF_IMAGE_REGISTRY" \
--tag "$LCF_IMAGE_TAG" \
--pull-secret cloudsmith-pull-secret \
--pull-secret-username token \
--pull-secret-email ops@example.com \
--probe-devices true \
--probe-timeout 180

If the cluster has one schedulable node

lcf-cli deploy install already handles this — no override needed. The chart requests two lcf-api replicas with hard pod anti-affinity, so on one schedulable node the second would be permanently Pending. lcf-cli derives the replica count from the number of nodes a pod can actually land on: one placeable node gets lcf-api.replicas: 1, two or more still get the HA pair on distinct nodes. This applies to both --mode manual and --mode auto.

A single-replica lcf-api has no redundancy, which suits a single-node evaluation and not production. If you deploy the chart directly with helm instead of through lcf-cli — Direct Helm Chart Deployment — the chart's own default stays at replicas: 2, and you must set lcf-api.replicas: 1 yourself; see Step 6 there.

If your cluster already runs cert-manager

lcf-cli deploy does not detect an existing cert-manager

It reports cluster.cert_manager_installed: false and generates certManager.install: true on every cluster, including one where cert-manager is running healthily in its own namespace with all its CRDs present. An operator who trusts the generated values has the chart install a second, bundled cert-manager over the working one.

Check for yourself before installing, and override if the answer is yes:

kubectl get pods -n cert-manager
kubectl get crd | grep cert-manager.io

If cert-manager is present, add this to the deploy install command above (and to the auto-mode command in Step 5):

--set certManager.install=false

The chart then issues its certificates through your cert-manager rather than bringing its own. This also removes the four quay.io/jetstack/* images from the mirror set — see Minimum Image Mirror Set in Prerequisites.

Manual Install Input Screen Fields

Field

Example

Notes

Kubeconfig

empty or /path/to/kubeconfig

Empty uses the default kubectl behavior.

Context

empty or prod

Empty uses current context.

Namespace

lcf

Target namespace.

Release

lcf

Helm release name.

Deployment name

inferra-prod

Stable artifact directory name.

Chart

./cloudsmith-charts/lcf-1.2.0.tgz

Local chart package pulled from the tagged Cloudsmith OCI chart.

Tag

v1.2.0

Image tag with leading v.

Pull secret

cloudsmith-pull-secret

Generated or existing Kubernetes Secret name.

Pull secret token

$LCF_REGISTRY_TOKEN

Read from the environment, never passed on the command line. Hidden by deploy logging.

Pull secret username

token

Use the username required by the Cloudsmith credential.

Pull secret email

ops@example.com

Stored in the docker-registry Secret.

Clean install

false

If true, deletes the namespace first. Avoid for existing data.

Version tags in this table

The chart and tag values above are a worked example for v1.2.0. Confirm the version tag for your release with Lightbits Support.

Manual Wizard Checks

Screen

Operator action

Node roles

Confirm storage, inference, dual-role, and excluded nodes.

Storage devices

Select only mounted LCFT XFS data paths such as /data0, /data1, and /data2.

Resource sizing

For lcft, accept the generated value unless you know the node has memory to spare — it is computed from your selected storage, and the request is set equal to the limit, so an over-large limit leaves the pod Pending. For lcfc, remember the limit is the RAM-tier size (limit − 16Gi), not headroom — raise it toward the node's memory on a dedicated node, and set it deliberately on a colocated one. See Step 5.

Summary

Review generated Helm values and node labels before deploy.

Deploying

Watch Helm and pod readiness progress.

Post-install

Note the API endpoint, token file, CA file, and lcf-cli command.

LCFT Mount Point Selection in Manual Mode

Storage path state

Recommended action

XFS mount at /data0, /data1, or /dataN dedicated to LCF

Select it.

OS, kubelet, container runtime, or application mount

Do not select it.

Missing /dataN path

Format and mount the host filesystem first, then rerun discovery/deploy.

Step 5: Auto Mode for Scripted Installs

Auto mode is intended for CI, labs, and repeatable installs after the deployment policy is known. Use it only after the host XFS mount points exist and a dry run shows mountPoints.enabled: true with the expected /data0..dataN paths.

Preview first:

lcf-cli deploy install \
--mode auto \
--dry-run \
--namespace "$LCF_NAMESPACE" \
--release "$LCF_RELEASE" \
--deployment-name "$LCF_DEPLOYMENT_NAME" \
--chart "$LCF_CHART" \
--registry "$LCF_IMAGE_REGISTRY" \
--tag "$LCF_IMAGE_TAG" \
--pull-secret cloudsmith-pull-secret \
--pull-secret-username token \
--pull-secret-email ops@example.com \
--probe-devices true \
--probe-timeout 180 \
--output-values ./inferra-generated-values.yaml

Inspect the generated values before a real auto install:

grep -n "imageRegistry" ./inferra-generated-values.yaml
grep -n "memory:" ./inferra-generated-values.yaml
grep -n "mountPoints" ./inferra-generated-values.yaml
grep -n "/data" ./inferra-generated-values.yaml
Stop if the mount points are missing

Stop if the generated values do not contain the expected lcft.storage.deviceSelector.mountPoints.paths list. Rerun in manual mode and select the mount points explicitly.

Run the install:

lcf-cli deploy install \
--mode auto \
--namespace "$LCF_NAMESPACE" \
--release "$LCF_RELEASE" \
--deployment-name "$LCF_DEPLOYMENT_NAME" \
--chart "$LCF_CHART" \
--registry "$LCF_IMAGE_REGISTRY" \
--tag "$LCF_IMAGE_TAG" \
--pull-secret cloudsmith-pull-secret \
--pull-secret-username token \
--pull-secret-email ops@example.com \
--probe-devices true \
--probe-timeout 180

Resource sizing formulas

The dry run is the authoritative number

To see what a node will actually be asked for, run deploy install --dry-run and read the generated values file. That is what auto mode computed for your devices, your GPUs, and your labels, and it cannot drift from the code the way a formula in a document can.

The formulas auto mode applies are below, for capacity planning before you have a cluster to run --dry-run against:

Component

Formula

lcft CPU request

3 management cores plus 2 cores per usable storage device.

lcft memory

5Gi runtime overhead, plus 6Gi per TiB of usable selected storage, plus 512Mi per lcft data-plane core.

lcfc CPU request

3 control-plane cores plus one RDMA data-plane core per GPU. Over TCP — when the data-plane transport to an lcft target is TCP rather than RDMA — lcfc instead sizes for one receive core per lcft data connection it owns on TCP targets, never below the RDMA per-GPU figure and capped at half the node; until an lcft target is known it keeps the RDMA per-GPU figure.

lcfc memory limit

The RAM tier plus a 16Gi reserve. The tier is half the node's memory, floored at 32Gi — except that a 2 MiB huge-page reservation smaller than that share, but still at least 32Gi, bounds the tier instead, so it stays on the pages you reserved. A reservation-bounded tier lands slightly under the reservation: 4 huge pages per data-plane core are left outside it, and the result is rounded down to a whole GiB. On a unified-memory GPU node — one whose GPU shares the CPU's RAM pool, such as an NVIDIA GB10 (detected by a nvidia.com/gpu.product label naming a GB10, or by a GPU node that carries no nvidia.com/gpu.memory label — see Check the lcfc limit on a GPU node without GPU Feature Discovery labels below) — the tier is sized instead from what is left for it, because the engine's KV cache draws from the same pool. By default the tier is a conservative bound (16Gi), but it is capped to what physically fits a heavy engine on the node — node RAM − 0.65×node RAM − 18Gi system − 16Gi — so on a tight single-node lab (which also runs the Kubernetes control plane) the default lands well under 16Gi rather than over-committing the node, while a roomy node keeps the full 16Gi. The engine's real footprint is fixed by the model you load after install, which this deploy-time sizer cannot know, so it errs toward the engine (a starved engine will not start; a smaller tier only lowers hit rate). Pass --engine-memory-utilization <fraction> when you know the engine's peak gpu_memory_utilization, and the tier becomes the exact remainder node RAM − utilization×node RAM − 18Gi system − 16Gi; the deploy refuses rather than ship an over-committing tier if that leaves nothing. Either way the sizer warns you of the maximum engine gpu_memory_utilization the split allows — derived from the real budget — and the engine must stay at or below it.

lcfc memory request

Equal to the limit whenever the tier has no hugepages-2Mi grant — the tier is then pre-faulted into lcfc's own memory, so the scheduler must count the whole thing or it can admit pods the node cannot hold. Only when a huge-page reservation backs the tier does the request shrink to lcfc's own working set, since the tier's pages are then a separate hugetlb allocation.

lcfc hugepages-2Mi

Equal request and limit covering the whole tier, emitted only when the node's reservation covers it. Omitted otherwise, which leaves the tier on transparent huge pages.

Check the lcfc limit on a GPU node without GPU Feature Discovery labels

Auto mode tells a discrete GPU (an H100, for example) from a unified-memory one by the nvidia.com/gpu.memory node label, which NVIDIA GPU Feature Discovery (GFD) sets to the GPU's on-board memory. A GPU node that runs the NVIDIA device plugin without GFD has no such label, so auto mode sizes it as a unified-memory node: the tier gets the conservative 16Gi bound instead of half the node, and nothing warns you. Check the labels before installing — an empty GPU.MEMORY column on a discrete-GPU node means the tier will be undersized.

kubectl get nodes -L nvidia.com/gpu.product -L nvidia.com/gpu.memory

Either deploy GFD (it ships with the NVIDIA GPU Operator) and rerun --dry-run, or set the limit yourself — in manual mode by editing it, in auto mode with --set, which overrides the generated values. If the node reserves enough 2 MiB huge pages to hold the tier, grant them too — request and limit equal to the tier, which is how auto mode itself emits them:

--set lcfc.resources.limits.memory=<tier + 16>Gi
--set lcfc.resources.requests.hugepages-2Mi=<tier>Gi \
--set lcfc.resources.limits.hugepages-2Mi=<tier>Gi

Whichever route you take, confirm the lcfc memory limit in the --dry-run values file before the real install.

Do not force an lcft memory limit the node cannot schedule

Earlier guidance said to raise every lcft memory limit to at least 256Gi. Treat that as a large-node figure, not a floor, and size from the formula above or from --dry-run instead.

Two reasons, both of which bite on a modest node:

  • The number auto mode produces is usually correct. A node with 4.9 TiB of selected storage generates about 49Gi, which is exactly what the formula in the table above predicts. That is not auto mode under-provisioning; it is the sizing working.

  • lcf-cli sets the lcft memory request equal to its limit. A 256Gi limit is therefore a 256Gi request, and on a node with less allocatable memory than that — easily the case once the RAM tier's huge pages are reserved — the lcft pod is simply Pending forever.

Raise the limit above the generated value when you know the node has the memory and you want headroom for growth in the selected storage. Check what the node can actually give you first:

kubectl get node <node> -o jsonpath='{.status.allocatable.memory}{"\n"}'

Overriding sizing without the interactive wizard

Step 4's manual mode is a terminal wizard: it needs a TTY and cannot run from a script, from CI, or over a non-interactive SSH command. When you need to change a generated value non-interactively, use auto mode with --set or --values instead of --mode manual:

lcf-cli deploy install --mode auto \
--set lcft.resources.limits.memory=96Gi \
--set lcft.resources.requests.memory=96Gi \
… the rest of the auto-mode arguments …

Set the request alongside the limit for lcft: they are generated equal, and raising only the limit leaves the request where it was. For structured or checked-in overrides prefer --values — see Appendix B.

CPU is trimmed to what the node has left (from 1.2.0). The CPU formulas above are what auto mode asks for when the node can give it. When it cannot, auto mode trims lcft and lcfc to the node's CPU budget — its allocatable CPU, minus what pods in other namespaces request, minus a small margin and the daemons' proxy sidecars — and prints a warning for each daemon it trims. On a single-node cluster the budget also leaves room for Inferra's own utility pods (the certificate authority, lcf-api and the observability stack).

Choosing the data-plane NICs (new in 1.2.0)

lcft and lcfc now limit their data traffic to the node's data-plane network interfaces. By default each daemon uses every physical interface that has an IPv4 address, plus any active InfiniBand port; loopback, software interfaces such as a CNI bridge, and interfaces without an address are left out. An interface's reported link speed no longer matters.

To choose the interfaces yourself, set dataPlane.ucxNetDevices on both daemons to a comma-separated list. Name an RDMA port as <device>:<port>; a network interface name on its own (ens1f0) carries TCP:

lcf-cli deploy install --mode auto \
--set lcft.dataPlane.ucxNetDevices=mlx5_0:1\,mlx5_1:1 \
--set lcfc.dataPlane.ucxNetDevices=mlx5_0:1\,mlx5_1:1 \
… the rest of the auto-mode arguments …
  • Never name a bare RDMA card (mlx5_0). Always give the port (mlx5_0:1): a bare card name does not reliably select the port you mean.

  • Escape each comma (\,) inside --set, or put the list in a --values file.

  • The value you set is used exactly as written, and sizing follows it: a daemon whose list names no RDMA port is sized for TCP.

  • The chart refuses a list containing whitespace, ^ or [, or more than 64 names.

  • lcf-cli deploy discover does not read --set or --values, so its sizing report ignores this setting; deploy install applies it.

The lcfc memory limit is the RAM-tier size

This is the one sizing number whose meaning changed, and treating it as ordinary headroom will quietly cost you most of the cache.

lcfc keeps a host-RAM KV tier in front of the storage tier, and it derives that tier from its own memory limit: limit − 16Gi. The limit is therefore the cache-capacity dial, not a safety ceiling. A limit of 128Gi — the chart's default — is a tier of roughly 112 GiB. A limit of 64Gi is a tier of 48 GiB, and a limit near the reserve leaves no usable tier at all.

Aim far above the GPU's own KV cache

The tier earns its place by holding what will not fit in HBM; a tier the size of HBM duplicates the GPU's cache instead of extending it, and the engine's own prefix cache already serves that case faster than any host-memory read can. Think in multiples of the HBM KV cache, not in absolute gigabytes — that ratio is what decides how much of your working set stays off the storage tier.

Three consequences worth planning for:

  • On a dedicated storage-client node, make the limit most of the node's memory. Auto mode already aims at half; raise it if the node does nothing else.

  • On a colocated node — an inference engine on the same host — size it explicitly as node memory − engine budget − 16Gi. Auto mode's half-the-node default does not know your engine's budget. Exception: on a unified-memory GPU node (a GB10, where the GPU shares the CPU's RAM), auto mode does this for you — by default it gives the tier a conservative bound (16Gi) capped to what physically fits a heavy engine, so on a tight single-node lab the default shrinks to fit rather than over-committing the node, and it warns you of the maximum engine gpu_memory_utilization the split allows; keep the engine at or below it. If you know the engine's utilization, pass --engine-memory-utilization <fraction> (for example 0.65) and the tier is sized from the exact remainder (node − utilization×node − 18Gi system − 16Gi); the deploy refuses if that leaves no room for a tier rather than shipping one that breaks engine startup.

  • A limit at or above the node's physical memory is not an error, but the daemon treats it as half the node rather than honoring it, so it is not a way to ask for more.

Reserve 2 MiB huge pages for it. The tier is fastest, and comes up fastest, on reserved hugetlb pages, and it can only use them if the pod carries a matching hugepages-2Mi limit — a pod cannot exceed its hugetlb cgroup limit however many free pages the node has. lcf-cli deploy emits that limit automatically when the node's reservation covers the tier it planned. To make that happen, on every inference and dual-role node:

# Size for the tier you want, plus a little slack. 2 MiB pages, so
# a 112 GiB tier is 57344 pages.
sudo sysctl -w vm.nr_hugepages=57600
echo 'vm.nr_hugepages = 57600' | sudo tee /etc/sysctl.d/90-lcf-hugepages.conf
# The kubelet reads hugepage capacity at startup, so restart it before deploying.
sudo systemctl restart kubelet
kubectl get node <node> -o jsonpath='{.status.allocatable.hugepages-2Mi}{"\n"}'

Expect the tier to land a little under what you reserved. When the reservation is what bounds the tier, lcf-cli leaves 4 huge pages per data-plane core outside it and rounds down to a whole GiB, so a 196 GiB reservation on a 7-core node plans a 195 GiB tier — a 211Gi memory limit and a 199680Mi huge-page grant. That is the intended behavior: asking for the exact reservation would round the limit up and demand more pages than exist, which refuses the grant outright and drops the tier to a slower rung. Reserve a little more than the tier you want rather than exactly it.

Skipping this is supported, not fatal. Without a reservation the tier falls back to transparent huge pages, and below that to 4 KiB pages — it still works and still caches, but it comes up more slowly, is sized from free memory rather than from your request, and serves more slowly. Confirm which one you got from the lcfc log line RAM tier backing plan at startup, or from the lcfc_ram_tier_backing metric.

Bring-up takes longer than it used to. lcfc faults in and registers the whole tier before it reports ready — seconds for a small tier on hugetlb, and several minutes for a large one on a lower rung. The chart's startup budget is 15 minutes for exactly this reason. See Step 6.

Restarting lcfc keeps its tier (from 1.1.0)

The tier lives in a named file on the node. When lcfc restarts — a pod deletion, a rolling upgrade, a crash — the replacement adopts that same tier: same memory, same backing, no reclaim wait and no fresh pre-fault. A running vLLM pod keeps its mapping across the restart and reattaches without being restarted itself, so there is no longer any need to scale vLLM down around an lcfc restart.

The adopted tier starts empty: the blocks the previous lcfc held in RAM are not served from it. Every one of them is still in the storage tier, so warm reads continue from there while the RAM tier refills with traffic.

Earlier releases needed a scale-down around every lcfc restart

They re-planned the tier on every lcfc restart, against the memory that was free at that moment, and told you never to restart lcfc on its own. That procedure is no longer needed on this release.

From 1.2.0 the tier directory is its own huge-page tmpfs. lcfc now mounts the directory that holds a tier not on reserved huge pages (lcfc.ramTier.shmemHostPath, default /dev/shm/lcf-ram-tier) as a dedicated tmpfs that uses 2 MiB pages, so that tier needs no node preparation. The mount is made on the host by a short privileged init container, mount-thp-tier, and the parent directory of lcfc.ramTier.shmemHostPath must exist on the node. The mount outlives the pod, so later restarts adopt the tier as described above. The one exception is the first rollout of 1.2.0 on such a node: the init container removes the tier file left in the plain directory by the previous release, and lcfc plans a new tier. Restart the vLLM pods on that node after lcfc has rolled — you do so anyway to upgrade the connector wheel — so that no engine keeps a mapping of the old tier. A tier on reserved huge pages is not affected.

Resizing the tier is an explicit step, not a restart

Because a restart adopts the existing tier, the tier never resizes on its own — not when you raise lcfc's memory limit, and not when huge pages appear. To pick up a new size or a better backing, invalidate the tier first, then restart:

  1. With lcfc running, invalidate the tier on the node (omit --node-uuid for every node):

    lcf-cli node clear-ram-tier --recreate --node-uuid REPLACE_WITH_NODE_UUID

    The command returns once every selected lcfc has acknowledged. The running tier keeps serving; only its filename is gone.

  2. Apply the change — the new lcfc memory limit, the new huge-page reservation, or both — and roll lcfc. The vLLM pods do not need to be scaled down: a connected engine releases its hold on the old tier when lcfc restarts.

  3. Confirm the new size from the lcfc log line RAM tier backing plan at startup, or from the lcfc_ram_tier_backing metric. The replacement lcfc waits a bounded time for the old tier's memory to return before planning, so a large tier on a tmpfs backing can take a little longer to come up.

A tier left on the node by a previous Inferra deployment is retired rather than adopted, so a clean reinstall plans a fresh tier at the new deployment's size.

Step 6: Monitor Deployment

lcf-cli deploy install prints progress stages and the Helm command. With --deployment-name inferra-prod, persistent files are written to stable paths.

File

Default path with --deployment-name inferra-prod

Generated Helm values

${XDG_CONFIG_HOME:-~/.config}/inferra-cli/deploy-runs/inferra-prod/helm-values.yaml

Bootstrap JWT

${XDG_CONFIG_HOME:-~/.config}/inferra-cli/deploy-runs/inferra-prod/bootstrap-token.jwt

Root CA

${XDG_CONFIG_HOME:-~/.config}/inferra-cli/deploy-runs/inferra-prod/root-ca.pem

Debug log

${XDG_STATE_HOME:-~/.local/state}/inferra-cli/deploy-runs/inferra-prod/deploy.log

Short-lived pre-pull pods come first (new in 1.1.0)

Before Helm creates the release, pre-install and pre-upgrade hooks pull each role's images onto that role's nodes — one puller DaemonSet per role, and an awaiter Job that waits for them and then removes them — so the install itself starts with the large images already on the node. The hooks create their own ServiceAccount, Role and RoleBinding in the release namespace. They can never fail the install: after imagePrepull.awaiter.timeoutSeconds (default 300) the awaiter gives up, removes the pullers and lets the install proceed. On a cluster whose admission policy does not allow those objects, set imagePrepull.enabled: false in your Helm values.

kubectl -n "$LCF_NAMESPACE" get pods -o wide -w
kubectl -n "$LCF_NAMESPACE" rollout status deployment/step-ca --timeout=10m
kubectl -n "$LCF_NAMESPACE" rollout status daemonset/lcft --timeout=10m
kubectl -n "$LCF_NAMESPACE" rollout status daemonset/lcfc --timeout=20m
kubectl -n "$LCF_NAMESPACE" rollout status deployment/lcf-api --timeout=10m
helm status "$LCF_RELEASE" -n "$LCF_NAMESPACE"
helm history "$LCF_RELEASE" -n "$LCF_NAMESPACE"
kubectl -n "$LCF_NAMESPACE" get jobs
kubectl -n "$LCF_NAMESPACE" get certificates,certificaterequests
kubectl get stepclusterissuer "${LCF_RELEASE}-${LCF_NAMESPACE}-step-issuer"
tail -f "${XDG_STATE_HOME:-$HOME/.local/state}/inferra-cli/deploy-runs/${LCF_DEPLOYMENT_NAME}/deploy.log"
lcfc is the slow one, and that is expected

It faults in and registers its whole RAM tier before reporting ready. A small tier on reserved huge pages is ready in well under a minute; a large one on a fallback rung can take several. The chart allows 15 minutes before it gives up, which is why the lcfc rollout line waits 20 and the others wait 10. A lcfc pod that is Running but not Ready for a few minutes after install is normal — check its log for RAM tier backing plan to see what it is doing.

Investigate a Blocked Pod:

kubectl -n "$LCF_NAMESPACE" get events --sort-by=.lastTimestamp
POD_NAME=REPLACE_WITH_POD_NAME
kubectl -n "$LCF_NAMESPACE" describe pod "$POD_NAME"
kubectl -n "$LCF_NAMESPACE" get pod "$POD_NAME" \
-o jsonpath='{.spec.containers[*].name}{"\n"}'
CONTAINER_NAME=REPLACE_WITH_CONTAINER_NAME
kubectl -n "$LCF_NAMESPACE" logs "$POD_NAME" -c "$CONTAINER_NAME"

Step 7: Use the Saved JWT With lcf-cli

After a successful deploy, lcf-cli deploy saves the bootstrap token and root CA.

export LCF_TOKEN_FILE="${XDG_CONFIG_HOME:-$HOME/.config}/inferra-cli/deploy-runs/${LCF_DEPLOYMENT_NAME}/bootstrap-token.jwt"
export LCF_CA_FILE="${XDG_CONFIG_HOME:-$HOME/.config}/inferra-cli/deploy-runs/${LCF_DEPLOYMENT_NAME}/root-ca.pem"

If deploy printed a try: command, use that endpoint. If not, build the endpoint from the NodePort service:

API_NODE=storage-01
API_IP=$(kubectl get node "$API_NODE" \
-o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}')
API_PORT=$(kubectl -n "$LCF_NAMESPACE" get svc lcf-api-external \
-o jsonpath='{.spec.ports[?(@.name=="grpc-jwt")].nodePort}')
export LCF_ENDPOINT="https://${API_IP}:${API_PORT}"

Run Management Commands:

lcf-cli list targets
lcf-cli list devices
lcf-cli list clients
lcf-cli \
--endpoint "$LCF_ENDPOINT" \
--token-file "$LCF_TOKEN_FILE" \
--ca-file "$LCF_CA_FILE" \
list targets
Keep TLS verification on outside a lab

For lab TLS debugging only, use --insecure-skip-tls-verify. Production installs should include the endpoint IP or DNS name in lcf-api.certificate.ipAddresses or lcf-api.certificate.dnsNames and keep TLS verification enabled.

--insecure-skip-tls-verify cannot be combined with --ca-file

The CLI refuses the pair outright:

error: the argument '--insecure-skip-tls-verify' cannot be used with '--ca-file <CA_FILE>'

LCF_CA_FILE counts as --ca-file. The exports above set it, so the lab-debugging escape hatch fails with that error in the very flow this step just walked you through. Clear the variable for the one command:

env -u LCF_CA_FILE lcf-cli --endpoint "$LCF_ENDPOINT" \
--token-file "$LCF_TOKEN_FILE" --insecure-skip-tls-verify list targets

Decide at install time how you will reach the API

The endpoint deploy prints is a NodePort on a node's own address, and the lcf-api certificate covers only the node addresses discovered during install. From a workstation that cannot route to the node network there is no supported route afterwards: kubectl port-forward reaches the service, but 127.0.0.1 is not in the certificate's SANs, so it then needs --insecure-skip-tls-verify — which is the lab-only option above, not an answer for production.

The fix belongs in the install, not in the client. Add the addresses you will actually connect from — a load-balancer IP, a DNS name, or 127.0.0.1 if you intend to port-forward — to the certificate before you install.

Add them, do not overwrite them

lcf-cli deploy has already generated lcf-api.certificate.ipAddresses[0] = the node's internal IP, which is the address the try: line prints. Helm's --set on an explicit list index replaces that element, so ipAddresses[0]=127.0.0.1 silently deletes the node IP and leaves the printed NodePort endpoint uncovered by the certificate. Append at the next free index instead.

--set 'lcf-api.certificate.ipAddresses[1]=127.0.0.1' \
--set 'lcf-api.certificate.dnsNames[0]=lcf-api.example.internal'

ipAddresses[1] because index 0 is taken; dnsNames[0] because deploy generates no DNS names, so index 0 is free there. Confirm before you commit to it — run deploy install --dry-run and read the generated values file, or render the certificate and check that every address you need is in the list:

grep -A6 'certificate:' ./inferra-generated-values.yaml

Changing them later means reissuing the certificate.

Step 8: Verify LCFT Devices

Verify targets and devices through the API:

lcf-cli list targets
lcf-cli list devices
Every storage node must report DEVICES greater than 0

A target with 0 devices in lcf-cli list targets has no storage tier: the install still reports healthy, and that node serves from its host-RAM tier only. Check that the node's /dataN mount points are present, and that the node has a network interface with an IPv4 address on the data network (see Choosing the data-plane NICs above). If it still shows 0, contact Lightbits support.

Verify lcft pods are on the intended storage nodes:

kubectl -n "$LCF_NAMESPACE" get pods -l app=lcft -o wide

If devices are missing, inspect the generated values file first:

grep -n "mountPoints" \
"${XDG_CONFIG_HOME:-$HOME/.config}/inferra-cli/deploy-runs/${LCF_DEPLOYMENT_NAME}/helm-values.yaml"

Troubleshooting Reference

Symptom

Likely cause

Fix

No lcft pod on a storage node

Missing storage label

Label the node or rerun manual deploy with the correct role.

Expected /dataN path is missing

Host filesystem was not mounted before discovery

Format as XFS, mount it, persist it in /etc/fstab, and rerun discovery/deploy.

Non-LCF mount appears in selector

Manual selection included the wrong mount path

Rerun manual deploy and select only LCF-owned /data0..dataN paths.

TLS endpoint not printed

API certificate SAN does not cover the chosen node IP

Use an IP from the SAN list or reinstall with the correct certificate IPs.

For ongoing kubectl operations — including lcft and lcfc logs, Envoy logs, Envoy admin readiness, certificate rotation checks, and JWT/root CA extraction — see Troubleshooting.

Step 8b: Create the Tenant and Logical Namespace the Connector Binds To

Do not skip this step

A fresh install does not create the tenant or the logical namespace the vLLM connector expects, and the connector does not fail when they are missing — it falls back to the cluster's bootstrap namespace.

That fallback is deliberate, and it is why this is easy to miss: an unbound session's reads and writes are routed to logical namespace 0 (the one named logical-namespace-0, owned by the system tenant) instead of being rejected. Nothing errors, the engine serves, the cache may even appear to work — and you have lost the isolation you were relying on:

  • Your KV lands in the bootstrap namespace, alongside every other unbound client on the cluster. Two engines that were supposed to be separated by tenant are not.

  • Per-namespace settings do not apply. Quota and the data-at-rest encryption configuration are resolved from namespace 0, not from the tenant you intended, so a namespace you created with its own encryption key is not the one being used.

Treat UNBOUND as a stop-and-fix condition before you serve production traffic, not as a performance problem to investigate later.

The connector opens one fabric session, identified by two values it sends with every request:

Connector setting

Default

Must name

lcf_tenant_id

vllm-connector

A tenant that exists in the cluster.

lcf_namespace

vllm

A logical namespace of that tenant, with exactly this name.

A clean install gives you neither. It creates one tenant, system, and one logical namespace, logical-namespace-0, belonging to it. Confirm that for yourself:

lcf-cli list tenants
lcf-cli list logical-namespaces

Create the pair. <lcf-namespace> is whatever you will configure as the connector's lcf_namespace — pods that must share cached KV use the same value, so decide it now and use it in both places:

PNS=$(lcf-cli -o json list physical-namespaces | jq -r '.items[0].uid')
echo "physical namespace: $PNS"

lcf-cli create tenant --name vllm-connector

lcf-cli create logical-namespace \
--name REPLACE_WITH_LCF_NAMESPACE \
--physical-namespace-uid "$PNS" \
--tenant-id vllm-connector \
--wait

Two things to be aware of:

  • --tenant-id takes the tenant's name, not a UUID — vllm-connector here, matching --name on the command above it.

  • Use your own tenant name if you set lcf_tenant_id. vllm-connector is only the connector's default; if you configure a different tenant, create that one instead. One vLLM pod serves exactly one tenant.

Replicas share the cache

Several vLLM replicas of the same model can bind to the same logical namespace: the cache key an engine computes carries no per-replica identity, so each replica reuses the KV the others stored. Create a separate logical namespace only for engines whose cached KV must be kept apart.

Confirm the session actually bound once the vLLM pod is running. Because the data path falls back rather than failing, this log line is the only place the difference is visible:

kubectl -n "$LCF_NAMESPACE" logs -l app.kubernetes.io/name=lcfc -c lcfc --tail=200 | grep -i 'session created'

You want session created (bound) with your namespace and tenant. If instead you see session created UNBOUND — namespace not in directory, the pair does not exist or the names do not match exactly — recheck both against lcf-cli list logical-namespaces and the connector's lcf_namespace and lcf_tenant_id.

Creating the pair afterwards does fix a running engine. An unbound session binds itself once the namespace appears in the cluster directory — the log line says as much (will bind on next discovery push) — so you do not have to restart the vLLM pod. Re-run the command above to confirm it bound, and treat anything cached before that moment as having gone to namespace 0.

Step 8c: Clearing the RAM Tier

Emptying the host-RAM tier forces the next read of every block to come from the storage tier. This is a measurement tool rather than a maintenance task: it is how you separate a warm read served from RAM from one served from NVMe when establishing your baselines (Post-Deployment Validation). It does not delete cached data — the storage tier still holds it.

lcf-cli node clear-ram-tier
lcf-cli node clear-ram-tier --node-uuid REPLACE_WITH_NODE_UUID

lcf-cli list targets reports the node UUIDs. The tier refills as traffic resumes; nothing needs restarting.

Clearing is not resizing

The command above empties the tier of cached blocks and leaves it exactly the same size on the same backing — it is a measurement tool. To make the tier bigger, after raising lcfc's memory limit or reserving huge pages, add --recreate and restart lcfc: see Resizing the tier is an explicit step, not a restart in Step 5.

Step 9: Uninstall

Remove the release and wait for pods to disappear:

lcf-cli deploy uninstall \
--namespace "$LCF_NAMESPACE" \
--release "$LCF_RELEASE" \
--wait
What uninstall does not remove

Uninstall removes the Helm release. It does not delete Cloudsmith credentials stored outside the namespace, external DNS records, or any host data that lcft wrote to selected devices.

What it does beyond removing the release, and what it leaves — worth knowing before you reinstall into the same cluster:



Removed, though you may not expect it

The node-role.kubernetes.io/storage and node-role.kubernetes.io/inference labels. If you pre-labeled deliberately (Step 2), re-apply them before the next install.

Left in the namespace

The completed lcf-api-jwt-bootstrap job, and the cloudsmith-pull-secret, lcf-jwt-signing-key and lcf-observability-admin Secrets. A reinstall reuses or replaces them; delete the namespace if you want a genuinely clean start.

Removed, and must be

Every cert-manager-issued workload TLS Secret, lcf-observability-tls included. These carry no owner references, so nothing else would collect them — and a reinstall regenerates the cluster root, against which cert-manager only re-issues a Secret that is gone. A survivor would keep presenting the previous root.

Left outside the namespace

The lcf-device-probe namespace, created by the device-probe stage. It is empty and harmless, and kubectl delete namespace lcf-device-probe removes it.

Left on the hosts

Everything lcft wrote under the selected mount points — /data0/hrwdev*, /data1/hrwdev*, /data0/cluster.redb and so on. Removing these destroys the cached data and the cluster's metadata, so do it only when you intend to discard the deployment, and only on paths you confirmed are LCF's.

Confirm the namespace is otherwise empty before reinstalling:

kubectl -n "$LCF_NAMESPACE" get all,secrets,jobs
kubectl get namespace lcf-device-probe --ignore-not-found

Cleaning up after an install that never finished

If a previous deploy install partially failed — the single-node lcf-api case in Step 4 is the common one — Helm leaves the release in pending-install, and the next deploy install refuses to start rather than stepping on it:

preflight found Helm release "lcf" in namespace "lcf" is pending-install: …;
wait for the operation to finish, run deploy uninstall, or rerun with --clean-install

You do not need helm uninstall for this. lcf-cli deploy uninstall handles a pending-install release; run it, then install again with whatever --set override the first attempt was missing.

Appendix A: lcf-cli Deploy Parameters and Defaults

Global lcf-cli Options

Parameter

Default

Environment

Description

--endpoint

http://localhost:8080

LCF_ENDPOINT

lcf-api endpoint for management commands. Not used by deploy install.

-o, --output

tui

LCF_OUTPUT

Output format, tui or json.

--token

unset

LCF_TOKEN

JWT literal. Prefer --token-file.

--token-file

unset

LCF_TOKEN_FILE

JWT file path.

--ca-file

unset

LCF_CA_FILE

Root CA PEM bundle for HTTPS endpoint validation. Cannot be combined with --insecure-skip-tls-verify.

--insecure-skip-tls-verify

false

LCF_INSECURE_SKIP_TLS_VERIFY

Disable HTTPS certificate verification for lab/debug. Mutually exclusive with --ca-file, including when that comes from LCF_CA_FILE — clear the variable to use this.

-v, --verbose

false

none

Enable debug logging.

deploy install

Parameter

Default

Environment

Description

--kubeconfig

kubectl default

KUBECONFIG through kubectl

Kubeconfig path.

--context

current context

none

Kubernetes context.

--namespace

lcf

none

Target namespace.

--release

lcf

none

Helm release name.

--deployment-name

timestamp plus UUID

none

Stable name for deploy artifacts under XDG config and state roots.

--chart

charts/lcf

none

Local chart path or OCI chart reference. For pinned Cloudsmith installs, pass the .tgz produced by helm pull --version.

--registry

ghcr.io/lightbitslabs

LCF_IMAGE_REGISTRY

Image registry prefix. Set to the Cloudsmith registry for this installation.

--pull-secret

ghcr-pull-secret

none

Existing or generated imagePullSecret name.

--pull-secret-token

unset

LCF_REGISTRY_TOKEN

Registry token used to create or update the pull secret.

--pull-secret-username

GITHUB_ACTOR, then USER, then lcf-cli

LCF_REGISTRY_USERNAME

Registry username for generated pull secret.

--pull-secret-email

lcf-deploy@lightbitslabs.com

LCF_REGISTRY_EMAIL

Email field for generated docker-registry Secret.

--clean-install

false

none

Delete target namespace before installing. Refuses protected namespaces.

--mode

auto when --tag is set, otherwise manual

none

Deploy mode.

--dry-run

false

none

Print values and Helm command without mutating the cluster.

--output-values

XDG config deploy-run helm-values.yaml

none

Path to write generated values.

--probe-devices

true

none

Run device probe DaemonSet.

--probe-image

busybox:1.37

none

Probe container image.

--probe-timeout

120

none

Max probe wait in seconds.

--tag

unset

none

Image tag. Required for auto mode.

--set

none

none

Override one generated Helm value, KEY=VALUE. Repeatable, forwarded to helm --set, and takes precedence over everything else. This is how lcf-api.replicas=1 and certManager.install=false are applied — see Appendix B.

--values

none

none

Override generated values from a file. Repeatable. Beaten by --set, beats the generated values. See Appendix B.

deploy discover

Parameter

Default

Environment

Description

--kubeconfig

kubectl default

KUBECONFIG through kubectl

Kubeconfig path.

--context

current context

none

Kubernetes context.

--namespace

default

none

Namespace used for temporary probe resources.

--probe-devices

true

none

Run device probe DaemonSet.

--probe-image

busybox:1.37

none

Probe container image.

--probe-timeout

120

none

Max probe wait in seconds.

deploy uninstall

Parameter

Default

Environment

Description

--kubeconfig

kubectl default

KUBECONFIG through kubectl

Kubeconfig path.

--context

current context

none

Kubernetes context.

--namespace

lcf

none

Target namespace.

--release

lcf

none

Helm release name.

--wait

false

none

Wait for deployed pods to disappear.

Appendix B: Generated Helm Values

lcf-cli deploy writes the Helm values below from discovery, sizing, and any manual edits. You can override them at install time:

  • --set KEY=VALUE, repeatable, for individual leaves. Forwarded to helm --set and takes precedence over the generated values.

  • --values FILE, repeatable, for structured overrides. Rendered after the generated values and before any --set, so a file beats the generated values and --set beats the file. Unlike --set, a values file keeps explicit YAML types and carries whole nested structures, which makes it the reviewable, checked-in way to override something like lcft.storage.deviceSelector. Files are validated before the slow discovery and probe stages, so a malformed file fails immediately.

The exact output depends on discovery and manual edits.

Helm value

Generated from

Default or behavior

global.imageRegistry

--registry

Defaults to ghcr.io/lightbitslabs; set Cloudsmith explicitly.

global.imageTag

--tag

Required for auto mode.

global.imagePullSecrets

--pull-secret

Included when a pull secret name is configured.

certManager.install

Deploy generator

Always true. Deploy does not detect an existing cert-manager, so on a cluster that already runs one you must override this — see If your cluster already runs cert-manager in Step 4.

stepIssuer.install

Deploy generator

true.

clusterIdentity.uuid

Deploy generator

New UUID for first install, existing UUID reused on upgrade.

lcft.enabled

Node classification

true when at least one storage or dual-role node exists.

lcft.storage.deviceSelector

Device probe and manual device selection

Mount point selector based on selected /data0..dataN paths.

lcft.storage.dataHostPath

Device probe (first selected mount point, or /var/lib/lcf on block-device nodes)

Kept from the existing release on upgrade: lcft keeps the cluster state there, and a new path would leave the cluster without it. A clean install takes the probed path. An upgrade that the release notes say needs a clean install does not keep this state: remove it first (see Step 9: Uninstall).

lcft.resources

Sizing and manual edits

Memory request and limit are generated equal, from the selected storage capacity and data-plane core count. Raise both together, and only to something the node can schedule.

lcft.raft.bootstrapNodeName

First storage-capable node by name

Required by chart.

lcft.raft.seedNodes

Bootstrap storage node IP

Usually the bootstrap storage node IP with port 8443, for example 10.20.30.11:8443.

lcft.certificate.ipAddresses

Storage node internal IPs

Needed for mTLS SAN validation.

lcfc.enabled

Node classification

true when at least one inference or dual-role node exists.

lcfc.resources

Sizing and manual edits

The memory limit sizes the host-RAM KV tier (limit − 16Gi), so it is the cache-capacity dial rather than a ceiling. Auto mode aims at half the node — except on a unified-memory GPU node (a GB10), where it defaults the tier to a conservative bound (16Gi) capped to what physically fits a heavy engine — so it shrinks to fit a tight node — or sizes it from the remainder when you pass --engine-memory-utilization. Includes a hugepages-2Mi request and limit when the node's reservation covers the planned tier, and only then is the memory request smaller than the limit; without that grant the memory request equals the limit, since the tier is pre-faulted into lcfc's own memory. On a GPU node without GPU Feature Discovery labels, check this value before installing — Step 5.

lcfc.certificate.ipAddresses

Inference node internal IPs

Used for certificate SANs.

lcf-api.enabled

Storage bootstrap availability

true when lcft is enabled and seed nodes exist.

lcf-api.service.nodePort

Deploy generator

30443.

lcf-api.raft.seedNodes

Same seed list as lcft

Lets API query lcft voters.

lcf-api.certificate.ipAddresses

Internal and external node IPs from deployable nodes

Lets deploy print a TLS-valid NodePort endpoint.