LCF-CLI Deployment
lcf-cli deploy is the recommended way to install Inferra. It discovers the cluster, classifies nodes, sizes resources, generates Helm values, applies node labels, runs Helm, and writes the post-install access files for you.
Kubernetes, Helm, and kubectl must already be installed and operational — see Prerequisites. If you prefer to drive Helm yourself, use Direct Helm Chart Deployment instead.
Download lcf-cli
The download URL carries the version twice: VERSION is the PEP 440 form (1.2.0, in the versions/ segment) and SEMVER is the SemVer form (1.2.0, inside the filename). Mixing them up produces a 404.
What lcf-cli Deploy Does
Stage | Behavior |
|---|---|
Preflight | Verifies Helm, kubectl, Kubernetes version, namespace write access, registry pull access, network pull bandwidth, and container image storage. |
Discovery | Reads Kubernetes nodes, versions, labels, taints, addresses, GPU count, memory, CPU, and existing release state. |
Device probe | Runs a temporary privileged probe DaemonSet to inspect host storage inventory and mounted |
Classification | Chooses storage, inference, dual-role, or excluded node roles. Existing LCF labels override heuristics. |
Sizing | Computes CPU, memory, and cpuset recommendations. Manual mode lets the operator edit memory limits. |
Values | Writes generated Helm values under the deploy config directory. |
Labels | Applies |
Helm | Runs |
Post-install | Reads the bootstrap JWT, root CA, API exposure, and writes access files. |
Prerequisites
Requirement | Minimum | Notes |
|---|---|---|
| The binary from this release's download URL | The examples use the |
Kubernetes | 1.28 | Enforced by deploy preflight. |
Helm | 3.x | Used by deploy to run the chart. |
kubectl | Compatible with the cluster | Used by deploy for discovery, labels, probes, and post-install. |
Cloudsmith credential | Read access | The examples use username |
Storage nodes | Pre-mounted XFS data paths |
|
Inference nodes | GPU nodes or CPU inference nodes |
|
2 MiB huge pages | Recommended on every inference and dual-role node |
|
Set common variables:
Authenticate Helm to Cloudsmith and pull the exact chart version.
The entitlement token that Lightbits Support issues you is accepted for basic authentication but is rejected by the OCI login handshake, which fails with Error: Get "https://docker.cloudsmith.io/v2/": unauthorized. Write the credential into a registry configuration file of its own and point helm pull at it — the pull then succeeds with the same token, and your existing Helm registry logins are left untouched. If Support issued you a Cloudsmith API key, which is a different credential, helm registry login --username token --password "$CLOUDSMITH_API_KEY" does work and you can use it in place of the first two commands below.
That file holds your token in clear text, so it is created readable only by you — the umask 077 above. The rm -f before it matters when you rerun this step: umask applies only when a file is created, so writing into a copy left from an earlier run would keep that copy's permissions. Remove the file when you are done installing, with rm -f "$INFERRA_HELM_REGISTRY_CONFIG".
That replaces every registry login Helm already holds for your account.
Create the namespace before running deploy discover in the final installation namespace:
Step 1: Discover the Cluster
Run discovery first. JSON output is useful for reviewing and changing records.
Classification rules used by deploy
Rules are evaluated in order, and the first match wins.
Condition | Role |
|---|---|
Node is unschedulable (cordoned) | Excluded |
Control-plane node tainted | Excluded |
Node already has storage and inference labels | Dual role |
Node already has storage label | Storage |
Node already has inference label | Inference |
Control-plane node tainted | Excluded |
Only node LCF can run on, with no GPU node elsewhere in the cluster | Dual role |
GPU and usable local storage | Dual role |
GPU without usable local storage | Inference |
Usable local storage without GPU | Storage |
No GPU and no usable storage | Excluded |
Note the first two rows: a cordoned node and a NoExecute control-plane node are excluded before any label is read, so neither is rescued by an explicit LCF label.
The control-plane role label alone does not. An untainted control-plane node — the usual shape for an evaluation or a single-node proof-of-concept — is classified by its hardware like any other node, and as the only node LCF can run on it is promoted to dual role. A control-plane node tainted NoSchedule is excluded unless you label it: the label rows above its row are the way in (see Step 2). lcft and lcfc do not tolerate NoExecute, so a NoExecute control-plane node is excluded even when labeled; remove that taint before deploying to it.
Step 2: Pre-Labeling — Required on a Tainted Control-Plane Node
lcf-cli deploy install can apply labels automatically, and on a cluster of ordinary worker nodes you can skip this step. Pre-label nodes when the deployment policy must override the hardware heuristics, and always in these two cases:
The node you are deploying to is a tainted control-plane node. Without an explicit LCF label it is excluded, whatever its hardware — including when it is the only node in the cluster. For a
NoScheduletaint, label it as shown below. For aNoExecutetaint, remove the taint first: a label alone does not letlcftorlcfcrun there.A node is cordoned. No label rescues it; the unschedulable row wins first. Uncordon it before discovery.
Verify labels:
Step 3: Prepare XFS Mount Points
The supported storage model is mount point mode. Prepare XFS filesystems and mount them on the host before running lcf-cli deploy. The deploy command should see storage as /data0, /data1, and /dataN mount points. Formatting is not performed by lcf-cli deploy — it is an operator-controlled host preparation step.
From 1.2.0 a disk of any type — NVMe, SATA, or a virtual disk — is accepted as storage when it is mounted only at a /dataN path; raw devices that are not mounted are still selected only if they are NVMe. If the generated values would leave lcft with no storage at all, the install stops before anything is installed and names the reason for each node: mount a dedicated disk at /data0, or set lcft.storage.* explicitly with --set or --values.
mkfs.xfs -f deletes data on the selected device. Run it only on devices dedicated to LCF.
Example for one storage node with three LCF data devices:
Persist the mounts across reboot:
Run discovery after the mount points exist:
Step 4: Recommended Install With Manual Mode
Manual mode is recommended for first production installs because it lets operators confirm roles, LCFT devices, and memory limits before Helm runs.
If the cluster has one schedulable node
lcf-cli deploy install already handles this — no override needed. The chart requests two lcf-api replicas with hard pod anti-affinity, so on one schedulable node the second would be permanently Pending. lcf-cli derives the replica count from the number of nodes a pod can actually land on: one placeable node gets lcf-api.replicas: 1, two or more still get the HA pair on distinct nodes. This applies to both --mode manual and --mode auto.
A single-replica lcf-api has no redundancy, which suits a single-node evaluation and not production. If you deploy the chart directly with helm instead of through lcf-cli — Direct Helm Chart Deployment — the chart's own default stays at replicas: 2, and you must set lcf-api.replicas: 1 yourself; see Step 6 there.
If your cluster already runs cert-manager
It reports cluster.cert_manager_installed: false and generates certManager.install: true on every cluster, including one where cert-manager is running healthily in its own namespace with all its CRDs present. An operator who trusts the generated values has the chart install a second, bundled cert-manager over the working one.
Check for yourself before installing, and override if the answer is yes:
If cert-manager is present, add this to the deploy install command above (and to the auto-mode command in Step 5):
The chart then issues its certificates through your cert-manager rather than bringing its own. This also removes the four quay.io/jetstack/* images from the mirror set — see Minimum Image Mirror Set in Prerequisites.
Manual Install Input Screen Fields
Field | Example | Notes |
|---|---|---|
Kubeconfig | empty or | Empty uses the default kubectl behavior. |
Context | empty or | Empty uses current context. |
Namespace |
| Target namespace. |
Release |
| Helm release name. |
Deployment name |
| Stable artifact directory name. |
Chart |
| Local chart package pulled from the tagged Cloudsmith OCI chart. |
Tag |
| Image tag with leading |
Pull secret |
| Generated or existing Kubernetes Secret name. |
Pull secret token |
| Read from the environment, never passed on the command line. Hidden by deploy logging. |
Pull secret username |
| Use the username required by the Cloudsmith credential. |
Pull secret email |
| Stored in the docker-registry Secret. |
Clean install |
| If |
The chart and tag values above are a worked example for v1.2.0. Confirm the version tag for your release with Lightbits Support.
Manual Wizard Checks
Screen | Operator action |
|---|---|
Node roles | Confirm storage, inference, dual-role, and excluded nodes. |
Storage devices | Select only mounted LCFT XFS data paths such as |
Resource sizing | For |
Summary | Review generated Helm values and node labels before deploy. |
Deploying | Watch Helm and pod readiness progress. |
Post-install | Note the API endpoint, token file, CA file, and |
LCFT Mount Point Selection in Manual Mode
Storage path state | Recommended action |
|---|---|
XFS mount at | Select it. |
OS, kubelet, container runtime, or application mount | Do not select it. |
Missing | Format and mount the host filesystem first, then rerun discovery/deploy. |
Step 5: Auto Mode for Scripted Installs
Auto mode is intended for CI, labs, and repeatable installs after the deployment policy is known. Use it only after the host XFS mount points exist and a dry run shows mountPoints.enabled: true with the expected /data0..dataN paths.
Preview first:
Inspect the generated values before a real auto install:
Stop if the generated values do not contain the expected lcft.storage.deviceSelector.mountPoints.paths list. Rerun in manual mode and select the mount points explicitly.
Run the install:
Resource sizing formulas
To see what a node will actually be asked for, run deploy install --dry-run and read the generated values file. That is what auto mode computed for your devices, your GPUs, and your labels, and it cannot drift from the code the way a formula in a document can.
The formulas auto mode applies are below, for capacity planning before you have a cluster to run --dry-run against:
Component | Formula |
|---|---|
| 3 management cores plus 2 cores per usable storage device. |
| 5Gi runtime overhead, plus 6Gi per TiB of usable selected storage, plus 512Mi per |
| 3 control-plane cores plus one RDMA data-plane core per GPU. Over TCP — when the data-plane transport to an |
| The RAM tier plus a 16Gi reserve. The tier is half the node's memory, floored at 32Gi — except that a 2 MiB huge-page reservation smaller than that share, but still at least 32Gi, bounds the tier instead, so it stays on the pages you reserved. A reservation-bounded tier lands slightly under the reservation: 4 huge pages per data-plane core are left outside it, and the result is rounded down to a whole GiB. On a unified-memory GPU node — one whose GPU shares the CPU's RAM pool, such as an NVIDIA GB10 (detected by a |
| Equal to the limit whenever the tier has no |
| Equal request and limit covering the whole tier, emitted only when the node's reservation covers it. Omitted otherwise, which leaves the tier on transparent huge pages. |
Auto mode tells a discrete GPU (an H100, for example) from a unified-memory one by the nvidia.com/gpu.memory node label, which NVIDIA GPU Feature Discovery (GFD) sets to the GPU's on-board memory. A GPU node that runs the NVIDIA device plugin without GFD has no such label, so auto mode sizes it as a unified-memory node: the tier gets the conservative 16Gi bound instead of half the node, and nothing warns you. Check the labels before installing — an empty GPU.MEMORY column on a discrete-GPU node means the tier will be undersized.
Either deploy GFD (it ships with the NVIDIA GPU Operator) and rerun --dry-run, or set the limit yourself — in manual mode by editing it, in auto mode with --set, which overrides the generated values. If the node reserves enough 2 MiB huge pages to hold the tier, grant them too — request and limit equal to the tier, which is how auto mode itself emits them:
Whichever route you take, confirm the lcfc memory limit in the --dry-run values file before the real install.
The lcft memory term with no analog elsewhere is the last one. Each lcft data-plane core pins two RMA buffers, 256Mi for reads and 256Mi for writes, and lcft pre-faults them at startup, so they are resident from the moment the pod runs rather than paged in on demand.
Data-plane cores come from device count (2 per usable device), while the per-TiB term comes from device capacity, so on a node with many small devices the two diverge sharply: 24 devices of 400 GB yield 32 data-plane cores, which is 16Gi of pools on its own.
Size from all three terms, or size from --dry-run. Sizing from the per-TiB term alone under-provisions such a node.
Do not force an lcft memory limit the node cannot schedule
Earlier guidance said to raise every lcft memory limit to at least 256Gi. Treat that as a large-node figure, not a floor, and size from the formula above or from --dry-run instead.
Two reasons, both of which bite on a modest node:
The number auto mode produces is usually correct. A node with 4.9 TiB of selected storage generates about
49Gi, which is exactly what the formula in the table above predicts. That is not auto mode under-provisioning; it is the sizing working.lcf-cli sets the lcft memory request equal to its limit. A
256Gilimit is therefore a256Girequest, and on a node with less allocatable memory than that — easily the case once the RAM tier's huge pages are reserved — thelcftpod is simplyPendingforever.
Raise the limit above the generated value when you know the node has the memory and you want headroom for growth in the selected storage. Check what the node can actually give you first:
Overriding sizing without the interactive wizard
Step 4's manual mode is a terminal wizard: it needs a TTY and cannot run from a script, from CI, or over a non-interactive SSH command. When you need to change a generated value non-interactively, use auto mode with --set or --values instead of --mode manual:
Set the request alongside the limit for lcft: they are generated equal, and raising only the limit leaves the request where it was. For structured or checked-in overrides prefer --values — see Appendix B.
CPU is trimmed to what the node has left (from 1.2.0). The CPU formulas above are what auto mode asks for when the node can give it. When it cannot, auto mode trims lcft and lcfc to the node's CPU budget — its allocatable CPU, minus what pods in other namespaces request, minus a small margin and the daemons' proxy sidecars — and prints a warning for each daemon it trims. On a single-node cluster the budget also leaves room for Inferra's own utility pods (the certificate authority, lcf-api and the observability stack).
Choosing the data-plane NICs (new in 1.2.0)
lcft and lcfc now limit their data traffic to the node's data-plane network interfaces. By default each daemon uses every physical interface that has an IPv4 address, plus any active InfiniBand port; loopback, software interfaces such as a CNI bridge, and interfaces without an address are left out. An interface's reported link speed no longer matters.
To choose the interfaces yourself, set dataPlane.ucxNetDevices on both daemons to a comma-separated list. Name an RDMA port as <device>:<port>; a network interface name on its own (ens1f0) carries TCP:
Never name a bare RDMA card (
mlx5_0). Always give the port (mlx5_0:1): a bare card name does not reliably select the port you mean.Escape each comma (
\,) inside--set, or put the list in a--valuesfile.The value you set is used exactly as written, and sizing follows it: a daemon whose list names no RDMA port is sized for TCP.
The chart refuses a list containing whitespace,
^or[, or more than 64 names.lcf-cli deploy discoverdoes not read--setor--values, so its sizing report ignores this setting;deploy installapplies it.
The lcfc memory limit is the RAM-tier size
This is the one sizing number whose meaning changed, and treating it as ordinary headroom will quietly cost you most of the cache.
lcfc keeps a host-RAM KV tier in front of the storage tier, and it derives that tier from its own memory limit: limit − 16Gi. The limit is therefore the cache-capacity dial, not a safety ceiling. A limit of 128Gi — the chart's default — is a tier of roughly 112 GiB. A limit of 64Gi is a tier of 48 GiB, and a limit near the reserve leaves no usable tier at all.
The tier earns its place by holding what will not fit in HBM; a tier the size of HBM duplicates the GPU's cache instead of extending it, and the engine's own prefix cache already serves that case faster than any host-memory read can. Think in multiples of the HBM KV cache, not in absolute gigabytes — that ratio is what decides how much of your working set stays off the storage tier.
Three consequences worth planning for:
On a dedicated storage-client node, make the limit most of the node's memory. Auto mode already aims at half; raise it if the node does nothing else.
On a colocated node — an inference engine on the same host — size it explicitly as
node memory − engine budget − 16Gi. Auto mode's half-the-node default does not know your engine's budget. Exception: on a unified-memory GPU node (a GB10, where the GPU shares the CPU's RAM), auto mode does this for you — by default it gives the tier a conservative bound (16Gi) capped to what physically fits a heavy engine, so on a tight single-node lab the default shrinks to fit rather than over-committing the node, and it warns you of the maximum enginegpu_memory_utilizationthe split allows; keep the engine at or below it. If you know the engine's utilization, pass--engine-memory-utilization <fraction>(for example0.65) and the tier is sized from the exact remainder (node − utilization×node − 18Gi system − 16Gi); the deploy refuses if that leaves no room for a tier rather than shipping one that breaks engine startup.A limit at or above the node's physical memory is not an error, but the daemon treats it as half the node rather than honoring it, so it is not a way to ask for more.
Reserve 2 MiB huge pages for it. The tier is fastest, and comes up fastest, on reserved hugetlb pages, and it can only use them if the pod carries a matching hugepages-2Mi limit — a pod cannot exceed its hugetlb cgroup limit however many free pages the node has. lcf-cli deploy emits that limit automatically when the node's reservation covers the tier it planned. To make that happen, on every inference and dual-role node:
Expect the tier to land a little under what you reserved. When the reservation is what bounds the tier, lcf-cli leaves 4 huge pages per data-plane core outside it and rounds down to a whole GiB, so a 196 GiB reservation on a 7-core node plans a 195 GiB tier — a 211Gi memory limit and a 199680Mi huge-page grant. That is the intended behavior: asking for the exact reservation would round the limit up and demand more pages than exist, which refuses the grant outright and drops the tier to a slower rung. Reserve a little more than the tier you want rather than exactly it.
Skipping this is supported, not fatal. Without a reservation the tier falls back to transparent huge pages, and below that to 4 KiB pages — it still works and still caches, but it comes up more slowly, is sized from free memory rather than from your request, and serves more slowly. Confirm which one you got from the lcfc log line RAM tier backing plan at startup, or from the lcfc_ram_tier_backing metric.
Bring-up takes longer than it used to. lcfc faults in and registers the whole tier before it reports ready — seconds for a small tier on hugetlb, and several minutes for a large one on a lower rung. The chart's startup budget is 15 minutes for exactly this reason. See Step 6.
Restarting lcfc keeps its tier (from 1.1.0)
The tier lives in a named file on the node. When lcfc restarts — a pod deletion, a rolling upgrade, a crash — the replacement adopts that same tier: same memory, same backing, no reclaim wait and no fresh pre-fault. A running vLLM pod keeps its mapping across the restart and reattaches without being restarted itself, so there is no longer any need to scale vLLM down around an lcfc restart.
The adopted tier starts empty: the blocks the previous lcfc held in RAM are not served from it. Every one of them is still in the storage tier, so warm reads continue from there while the RAM tier refills with traffic.
They re-planned the tier on every lcfc restart, against the memory that was free at that moment, and told you never to restart lcfc on its own. That procedure is no longer needed on this release.
From 1.2.0 the tier directory is its own huge-page tmpfs. lcfc now mounts the directory that holds a tier not on reserved huge pages (lcfc.ramTier.shmemHostPath, default /dev/shm/lcf-ram-tier) as a dedicated tmpfs that uses 2 MiB pages, so that tier needs no node preparation. The mount is made on the host by a short privileged init container, mount-thp-tier, and the parent directory of lcfc.ramTier.shmemHostPath must exist on the node. The mount outlives the pod, so later restarts adopt the tier as described above. The one exception is the first rollout of 1.2.0 on such a node: the init container removes the tier file left in the plain directory by the previous release, and lcfc plans a new tier. Restart the vLLM pods on that node after lcfc has rolled — you do so anyway to upgrade the connector wheel — so that no engine keeps a mapping of the old tier. A tier on reserved huge pages is not affected.
Resizing the tier is an explicit step, not a restart
Because a restart adopts the existing tier, the tier never resizes on its own — not when you raise lcfc's memory limit, and not when huge pages appear. To pick up a new size or a better backing, invalidate the tier first, then restart:
With lcfc running, invalidate the tier on the node (omit
--node-uuidfor every node):lcf-cli node clear-ram-tier --recreate --node-uuid REPLACE_WITH_NODE_UUIDThe command returns once every selected
lcfchas acknowledged. The running tier keeps serving; only its filename is gone.Apply the change — the new
lcfcmemory limit, the new huge-page reservation, or both — and rolllcfc. The vLLM pods do not need to be scaled down: a connected engine releases its hold on the old tier whenlcfcrestarts.Confirm the new size from the
lcfclog lineRAM tier backing planat startup, or from thelcfc_ram_tier_backingmetric. The replacementlcfcwaits a bounded time for the old tier's memory to return before planning, so a large tier on a tmpfs backing can take a little longer to come up.
A tier left on the node by a previous Inferra deployment is retired rather than adopted, so a clean reinstall plans a fresh tier at the new deployment's size.
Step 6: Monitor Deployment
lcf-cli deploy install prints progress stages and the Helm command. With --deployment-name inferra-prod, persistent files are written to stable paths.
File | Default path with |
|---|---|
Generated Helm values |
|
Bootstrap JWT |
|
Root CA |
|
Debug log |
|
Before Helm creates the release, pre-install and pre-upgrade hooks pull each role's images onto that role's nodes — one puller DaemonSet per role, and an awaiter Job that waits for them and then removes them — so the install itself starts with the large images already on the node. The hooks create their own ServiceAccount, Role and RoleBinding in the release namespace. They can never fail the install: after imagePrepull.awaiter.timeoutSeconds (default 300) the awaiter gives up, removes the pullers and lets the install proceed. On a cluster whose admission policy does not allow those objects, set imagePrepull.enabled: false in your Helm values.
It faults in and registers its whole RAM tier before reporting ready. A small tier on reserved huge pages is ready in well under a minute; a large one on a fallback rung can take several. The chart allows 15 minutes before it gives up, which is why the lcfc rollout line waits 20 and the others wait 10. A lcfc pod that is Running but not Ready for a few minutes after install is normal — check its log for RAM tier backing plan to see what it is doing.
Investigate a Blocked Pod:
Step 7: Use the Saved JWT With lcf-cli
After a successful deploy, lcf-cli deploy saves the bootstrap token and root CA.
If deploy printed a try: command, use that endpoint. If not, build the endpoint from the NodePort service:
Run Management Commands:
For lab TLS debugging only, use --insecure-skip-tls-verify. Production installs should include the endpoint IP or DNS name in lcf-api.certificate.ipAddresses or lcf-api.certificate.dnsNames and keep TLS verification enabled.
The CLI refuses the pair outright:
LCF_CA_FILE counts as --ca-file. The exports above set it, so the lab-debugging escape hatch fails with that error in the very flow this step just walked you through. Clear the variable for the one command:
Decide at install time how you will reach the API
The endpoint deploy prints is a NodePort on a node's own address, and the lcf-api certificate covers only the node addresses discovered during install. From a workstation that cannot route to the node network there is no supported route afterwards: kubectl port-forward reaches the service, but 127.0.0.1 is not in the certificate's SANs, so it then needs --insecure-skip-tls-verify — which is the lab-only option above, not an answer for production.
The fix belongs in the install, not in the client. Add the addresses you will actually connect from — a load-balancer IP, a DNS name, or 127.0.0.1 if you intend to port-forward — to the certificate before you install.
lcf-cli deploy has already generated lcf-api.certificate.ipAddresses[0] = the node's internal IP, which is the address the try: line prints. Helm's --set on an explicit list index replaces that element, so ipAddresses[0]=127.0.0.1 silently deletes the node IP and leaves the printed NodePort endpoint uncovered by the certificate. Append at the next free index instead.
ipAddresses[1] because index 0 is taken; dnsNames[0] because deploy generates no DNS names, so index 0 is free there. Confirm before you commit to it — run deploy install --dry-run and read the generated values file, or render the certificate and check that every address you need is in the list:
Changing them later means reissuing the certificate.
Step 8: Verify LCFT Devices
Verify targets and devices through the API:
A target with 0 devices in lcf-cli list targets has no storage tier: the install still reports healthy, and that node serves from its host-RAM tier only. Check that the node's /dataN mount points are present, and that the node has a network interface with an IPv4 address on the data network (see Choosing the data-plane NICs above). If it still shows 0, contact Lightbits support.
Verify lcft pods are on the intended storage nodes:
If devices are missing, inspect the generated values file first:
Troubleshooting Reference
Symptom | Likely cause | Fix |
|---|---|---|
No | Missing storage label | Label the node or rerun manual deploy with the correct role. |
Expected | Host filesystem was not mounted before discovery | Format as XFS, mount it, persist it in |
Non-LCF mount appears in selector | Manual selection included the wrong mount path | Rerun manual deploy and select only LCF-owned |
TLS endpoint not printed | API certificate SAN does not cover the chosen node IP | Use an IP from the SAN list or reinstall with the correct certificate IPs. |
For ongoing kubectl operations — including lcft and lcfc logs, Envoy logs, Envoy admin readiness, certificate rotation checks, and JWT/root CA extraction — see Troubleshooting.
Step 8b: Create the Tenant and Logical Namespace the Connector Binds To
A fresh install does not create the tenant or the logical namespace the vLLM connector expects, and the connector does not fail when they are missing — it falls back to the cluster's bootstrap namespace.
That fallback is deliberate, and it is why this is easy to miss: an unbound session's reads and writes are routed to logical namespace 0 (the one named logical-namespace-0, owned by the system tenant) instead of being rejected. Nothing errors, the engine serves, the cache may even appear to work — and you have lost the isolation you were relying on:
Your KV lands in the bootstrap namespace, alongside every other unbound client on the cluster. Two engines that were supposed to be separated by tenant are not.
Per-namespace settings do not apply. Quota and the data-at-rest encryption configuration are resolved from namespace 0, not from the tenant you intended, so a namespace you created with its own encryption key is not the one being used.
Treat UNBOUND as a stop-and-fix condition before you serve production traffic, not as a performance problem to investigate later.
The connector opens one fabric session, identified by two values it sends with every request:
Connector setting | Default | Must name |
|---|---|---|
|
| A tenant that exists in the cluster. |
|
| A logical namespace of that tenant, with exactly this name. |
A clean install gives you neither. It creates one tenant, system, and one logical namespace, logical-namespace-0, belonging to it. Confirm that for yourself:
Create the pair. <lcf-namespace> is whatever you will configure as the connector's lcf_namespace — pods that must share cached KV use the same value, so decide it now and use it in both places:
Two things to be aware of:
--tenant-id takes the tenant's name, not a UUID —
vllm-connectorhere, matching--nameon the command above it.Use your own tenant name if you set lcf_tenant_id.
vllm-connectoris only the connector's default; if you configure a different tenant, create that one instead. One vLLM pod serves exactly one tenant.
Several vLLM replicas of the same model can bind to the same logical namespace: the cache key an engine computes carries no per-replica identity, so each replica reuses the KV the others stored. Create a separate logical namespace only for engines whose cached KV must be kept apart.
Confirm the session actually bound once the vLLM pod is running. Because the data path falls back rather than failing, this log line is the only place the difference is visible:
You want session created (bound) with your namespace and tenant. If instead you see session created UNBOUND — namespace not in directory, the pair does not exist or the names do not match exactly — recheck both against lcf-cli list logical-namespaces and the connector's lcf_namespace and lcf_tenant_id.
Creating the pair afterwards does fix a running engine. An unbound session binds itself once the namespace appears in the cluster directory — the log line says as much (will bind on next discovery push) — so you do not have to restart the vLLM pod. Re-run the command above to confirm it bound, and treat anything cached before that moment as having gone to namespace 0.
Step 8c: Clearing the RAM Tier
Emptying the host-RAM tier forces the next read of every block to come from the storage tier. This is a measurement tool rather than a maintenance task: it is how you separate a warm read served from RAM from one served from NVMe when establishing your baselines (Post-Deployment Validation). It does not delete cached data — the storage tier still holds it.
lcf-cli list targets reports the node UUIDs. The tier refills as traffic resumes; nothing needs restarting.
The command above empties the tier of cached blocks and leaves it exactly the same size on the same backing — it is a measurement tool. To make the tier bigger, after raising lcfc's memory limit or reserving huge pages, add --recreate and restart lcfc: see Resizing the tier is an explicit step, not a restart in Step 5.
Step 9: Uninstall
Remove the release and wait for pods to disappear:
Uninstall removes the Helm release. It does not delete Cloudsmith credentials stored outside the namespace, external DNS records, or any host data that lcft wrote to selected devices.
What it does beyond removing the release, and what it leaves — worth knowing before you reinstall into the same cluster:
Removed, though you may not expect it | The |
Left in the namespace | The completed |
Removed, and must be | Every cert-manager-issued workload TLS Secret, |
Left outside the namespace | The |
Left on the hosts | Everything |
Confirm the namespace is otherwise empty before reinstalling:
Cleaning up after an install that never finished
If a previous deploy install partially failed — the single-node lcf-api case in Step 4 is the common one — Helm leaves the release in pending-install, and the next deploy install refuses to start rather than stepping on it:
You do not need helm uninstall for this. lcf-cli deploy uninstall handles a pending-install release; run it, then install again with whatever --set override the first attempt was missing.
Appendix A: lcf-cli Deploy Parameters and Defaults
Global lcf-cli Options
Parameter | Default | Environment | Description |
|---|---|---|---|
|
|
|
|
|
|
| Output format, |
| unset |
| JWT literal. Prefer |
| unset |
| JWT file path. |
| unset |
| Root CA PEM bundle for HTTPS endpoint validation. Cannot be combined with |
|
|
| Disable HTTPS certificate verification for lab/debug. Mutually exclusive with --ca-file, including when that comes from |
|
| none | Enable debug logging. |
deploy install
Parameter | Default | Environment | Description |
|---|---|---|---|
| kubectl default |
| Kubeconfig path. |
| current context | none | Kubernetes context. |
|
| none | Target namespace. |
|
| none | Helm release name. |
| timestamp plus UUID | none | Stable name for deploy artifacts under XDG config and state roots. |
|
| none | Local chart path or OCI chart reference. For pinned Cloudsmith installs, pass the |
|
|
| Image registry prefix. Set to the Cloudsmith registry for this installation. |
|
| none | Existing or generated |
| unset |
| Registry token used to create or update the pull secret. |
|
|
| Registry username for generated pull secret. |
|
|
| Email field for generated docker-registry Secret. |
|
| none | Delete target namespace before installing. Refuses protected namespaces. |
|
| none | Deploy mode. |
|
| none | Print values and Helm command without mutating the cluster. |
| XDG config deploy-run | none | Path to write generated values. |
|
| none | Run device probe DaemonSet. |
|
| none | Probe container image. |
|
| none | Max probe wait in seconds. |
| unset | none | Image tag. Required for auto mode. |
| none | none | Override one generated Helm value, |
| none | none | Override generated values from a file. Repeatable. Beaten by |
deploy discover
Parameter | Default | Environment | Description |
|---|---|---|---|
| kubectl default |
| Kubeconfig path. |
| current context | none | Kubernetes context. |
|
| none | Namespace used for temporary probe resources. |
|
| none | Run device probe DaemonSet. |
|
| none | Probe container image. |
|
| none | Max probe wait in seconds. |
deploy uninstall
Parameter | Default | Environment | Description |
|---|---|---|---|
| kubectl default |
| Kubeconfig path. |
| current context | none | Kubernetes context. |
|
| none | Target namespace. |
|
| none | Helm release name. |
|
| none | Wait for deployed pods to disappear. |
Appendix B: Generated Helm Values
lcf-cli deploy writes the Helm values below from discovery, sizing, and any manual edits. You can override them at install time:
--set KEY=VALUE, repeatable, for individual leaves. Forwarded tohelm --setand takes precedence over the generated values.--values FILE, repeatable, for structured overrides. Rendered after the generated values and before any--set, so a file beats the generated values and--setbeats the file. Unlike--set, a values file keeps explicit YAML types and carries whole nested structures, which makes it the reviewable, checked-in way to override something likelcft.storage.deviceSelector. Files are validated before the slow discovery and probe stages, so a malformed file fails immediately.
The exact output depends on discovery and manual edits.
Helm value | Generated from | Default or behavior |
|---|---|---|
|
| Defaults to |
|
| Required for auto mode. |
|
| Included when a pull secret name is configured. |
| Deploy generator | Always true. Deploy does not detect an existing cert-manager, so on a cluster that already runs one you must override this — see If your cluster already runs cert-manager in Step 4. |
| Deploy generator |
|
| Deploy generator | New UUID for first install, existing UUID reused on upgrade. |
| Node classification |
|
| Device probe and manual device selection | Mount point selector based on selected |
| Device probe (first selected mount point, or | Kept from the existing release on upgrade: |
| Sizing and manual edits | Memory request and limit are generated equal, from the selected storage capacity and data-plane core count. Raise both together, and only to something the node can schedule. |
| First storage-capable node by name | Required by chart. |
| Bootstrap storage node IP | Usually the bootstrap storage node IP with port 8443, for example |
| Storage node internal IPs | Needed for mTLS SAN validation. |
| Node classification |
|
| Sizing and manual edits | The memory limit sizes the host-RAM KV tier ( |
| Inference node internal IPs | Used for certificate SANs. |
| Storage bootstrap availability |
|
| Deploy generator |
|
| Same seed list as | Lets API query |
| Internal and external node IPs from deployable nodes | Lets deploy print a TLS-valid NodePort endpoint. |