# kuber `kuber` is a Docker Compose to Kubernetes translation layer backed by a self-hosted management service (`kuber-server`). It reads a local Compose file, renders Kubernetes resources, and applies them to the cluster through an authenticated v2 API. Image builds happen inside the cluster with rootless BuildKit, so the workstation needs neither Docker nor `kubectl`. ## Architecture ``` workstation (kuber CLI) ──HTTPS──▶ kuber-server (in-cluster pod) │ ├─ CAS (content-addressed blobs on RWX PVC) ├─ BuildKit Jobs (rootless) ──▶ registry └─ Kubernetes API (RBAC-scoped) ``` The CLI authenticates to the server with a Bearer token acquired by `kuber login`. It snapshots the repository into a content-addressed workspace, uploads only the blobs the server is missing, and submits a build request. The server materializes that workspace onto a shared RWX PVC and runs a rootless BuildKit Job that builds and pushes the image, then the CLI pins the resulting immutable registry digest into the rendered Deployment. The server also owns reconciliation: it plans, applies, and prunes resources in a per-project namespace, reconciles managed Postgres and S3 claims, rolls deployments back, streams logs, and exposes interactive `exec` sessions over a WebSocket. ## Directory Trust Before `kuber up`, run `kuber trust` from the configured project directory. Trust is exactly the configured namespace plus a SHA-256 fingerprint of the resolved current working directory. The local mode-0600 store lets `up` fail before builds from an untrusted directory. The server stores only namespace and fingerprint registrations in labelled ConfigMaps in its `kuber-system` control plane namespace, then checks the pair when resource reconciliation begins. This is an accidental-targeting safeguard, not a security boundary: a client that intentionally forges a registered fingerprint can pass it. `kuber trust status` shows local/server awareness without printing paths; `kuber trust revoke` removes the current directory registration. ## Environment Assumptions kuber targets a specific self-hosted cluster and workstation setup. It is not intended to run unchanged against an arbitrary Kubernetes environment. The expected setup: - an account on the `kuber-server` management API - a working ClusterRole/Role (see [Server Deployment](#server-deployment)) - a registry the in-cluster BuildKit can push to Not required locally: - Docker - `kubectl` ## API Origin The v2 management API has one hard-coded origin: ```text https://kuber.astrxl.dev/api/v2 ``` ## Authentication `kuber login [username] [--persist]` opens an interactive BasicAuth prompt for the password (the optional username pre-fills the username field). Commands that need authentication, including `init`, log in this way when no session is available; `init` also retries login if the server reports an expired or unauthorized session. Login requires an interactive terminal when credentials are needed. ```bash kuber login dmgnr kuber login dmgnr --persist kuber whoami kuber logout ``` The default login is stored with mode `0600` under `$XDG_RUNTIME_DIR/kuber/session.json` and disappears with the user runtime directory. `--persist` instead uses `$XDG_CONFIG_HOME/kuber/session.json`, or `~/.config/kuber/session.json` when `XDG_CONFIG_HOME` is unset. Password input is never echoed. Runtime sessions last 24 hours and persistent sessions last 30 days; `logout` revokes the server-side session. `readSession` falls back to the persistent file when no runtime session exists. The default login is scoped to the current user and the authenticated identity is available to every v2 command. `login`, `logout`, `whoami`, and global `maintenance` are the only commands that run without a loaded project configuration. ### API Keys Create API keys for automation or other non-interactive clients. Keys belong to a user and carry explicit capabilities; use the smallest capability set the task needs. The `--workspace` option further restricts Kubernetes access to one workspace. Capabilities describe _what_ the key may do, while workspace scope describes _where_ its Kubernetes access applies; neither replaces the other. Valid capabilities are `kubernetes:read`, `kubernetes:write`, `kubernetes:exec`, `users:read`, `users:write`, `sessions:revoke`, and `platform:adopt`. For example, a deployment key that only reads and updates resources in the `website` workspace can be created with: ```bash kuber users keys create deployer --capabilities kubernetes:read,kubernetes:write --workspace website ``` The default expiry is 90 days. Set `--expires-days` to an integer from 1 to 365, or explicitly use `none` for a key that does not expire: ```bash kuber users keys create deployer --capabilities kubernetes:read --expires-days 30 kuber users keys create deployer --capabilities kubernetes:read --expires-days none ``` The raw token is printed once at creation. Copy it directly into a secret manager; it cannot be retrieved later. List keys (optionally filtered by owner) to find the key ID, then revoke a compromised or retired key: ```bash kuber users keys ls kuber users keys ls deployer kuber users keys revoke deployer ``` Revoking a key immediately prevents it from authenticating. Do not place API tokens in source control, command-line arguments, or committed configuration. ### CI Integration Create a dedicated automation user/key with only the capabilities required by the CI job, and set the token as a masked repository or organization secret (for example, `KUBER_API_TOKEN`). Expose the secret as an environment variable for the job step. Commands that use the v2 API can then authenticate with that token without an interactive `kuber login`: ```yaml - name: Deploy with kuber env: KUBER_API_TOKEN: ${{ secrets.KUBER_API_TOKEN }} run: kuber up ``` Use the CI provider's secret store, never commit the token or print it in logs. For example, a workspace-scoped key with `kubernetes:read,kubernetes:write` allows deployment operations in that workspace, but does not grant user administration, exec, or platform adoption. Add a capability only when the job needs that operation, and choose `--workspace` to limit its Kubernetes scope. ### Roles and Authorization The server grants capabilities through three roles: - `viewer` — read-only cluster access (`kubernetes:read`) - `operator` — `viewer` plus `kubernetes:write` and `kubernetes:exec` - `admin` — all capabilities, including user administration (`users:read`, `users:write`, `sessions:revoke`, `platform:adopt`) Administer users with the `users` command tree: ```bash kuber users ls kuber users add dmgnr --roles admin kuber users update dmgnr --roles operator kuber users update dmgnr --password kuber users disable dmgnr kuber users enable dmgnr kuber users delete dmgnr kuber users revoke dmgnr ``` `users add`, `update --password`, and `delete` prompt for password / written confirmation on an interactive terminal. Passwords are hashed with Argon2id on the server and never stored in plaintext. Updating a user's roles or password revokes all of that user's active sessions. Inspect server-side operations and the audit trail: ```bash kuber operations ls kuber operations get kuber audit ls ``` ## Server Deployment The repository's `compose.yml` owns the `kuber-system` namespace, the `kuber-server` image, its Deployment, Service, and Ingress. `.kuberrc.ts` extends the rendered manifests with the server's ServiceAccount and RBAC: - a namespaced `Role`/`RoleBinding` (`kuber-server-auth`) for the `kuber-system` Secrets, ConfigMaps, Pods, Jobs, and Leases the server itself reads and writes - a `ClusterRole`/`ClusterRoleBinding` (`kuber-server-manager`) granting the cross-namespace verbs it needs to manage user projects `compose.yml` runs the server as a non-root user (`runAsUser`/`runAsGroup` `1000`), drops all Linux capabilities, uses a read-only root filesystem with a `RuntimeDefault` seccomp profile, and backs `/data` with a Longhorn PVC (`kuber-build-data`). The PVC hosts the CAS, materialized workspaces, and resumable upload bytes, and is shared with BuildKit Jobs. Deploy the server with kuber itself: ```bash KUBER_BOOTSTRAP_PASSWORD='replace-me' kuber up ``` The server creates an `admin` user (from `KUBER_BOOTSTRAP_USERNAME`, default `dmgnr`) on first boot only if that user does not already exist. The bootstrap Secret is only rendered while `KUBER_BOOTSTRAP_PASSWORD` is set. After logging in successfully, reconcile without that variable and restart once so kuber removes the stale bootstrap Secret and the password leaves the pod environment: ```bash kuber up --no-b kuber restart kuber-server ``` See [Account Recovery](#account-recovery) for what to do if you are locked out. ### Security Constraints Because the server's ServiceAccount is scoped by the RBAC in `.kuberrc.ts`, it can only act on the resources kuber manages. The management layer additionally enforces ownership, so the server refuses to mutate resources that do not carry kuber's workspace labels. Deleting a resource requires its UID as an optimistic concurrency precondition, and workspace deletion also requires the namespace UID. Namespaces that are not owned by kuber, or owned by a different workspace, are never mutated (see [Workspaces and Migration](#workspaces-and-migration)). ## Workspaces and Migration Each project maps to a Kubernetes namespace derived from the current working directory. The CLI records a "workspace" on the server keyed by project name and borrows the namespace UID to guarantee it owns the namespace before reconciling. Adopting existing resources relabels them (Server-Side Apply) only when they are already managed by kuber and not owned by another workspace. `up` refuses to mutate a namespace when: - the namespace already exists but does not carry kuber's managed-by label (`external`), or - the namespace is labeled for a different workspace UID (`different-workspace`) In those cases the CLI prints a hint to `POST /workspaces//adopt` with the namespace UID to complete a safe, explicit adoption. The platform namespace (`kuber-system`) is adopted through the admin-only platform adoption route. Workspace state, revisions, operations, and audit events are stored as Secrets and ConfigMaps in `kuber-system` keyed by kuber's `kuber.astrxl.dev/type` label. Workspace updates are optimistic (If-Match on resource version) and immutable revisions are recorded so history survives. Expired sessions are cleaned up on an interval, and stale operations are marked failed on server startup recovery. ### Migration Behavior `kuber up` is idempotent and safe to re-run. Each run: 1. snapshots the workspace and uploads missing blobs to the server CAS, 2. ensures the workspace record (creating or updating it with an optimistic If-Match), 3. adopts the namespace and its kuber-managed resources, 4. reconciles managed Postgres and S3 claims, 5. renders manifests, plans the diff, applies desired resources, waits for rollout, and deletes stale resources. Because resource references are immutable digests, re-running `up` only restarts deployments whose image content actually changed. `start` re-resolves published digests without building. ## Registry Authentication Building and pushing images from inside the cluster typically requires credentials for the target registry. These are read from a Docker config file (`KUBER_REGISTRY_CONFIG`, default `/etc/kuber/registry/config.json`) and mounted as an image pull secret named by `KUBER_REGISTRY_SECRET`. Registry authentication is optional. If `KUBER_REGISTRY_SECRET` is unset, the server warns at startup and BuildKit uses anonymous registry access. This is intended for registries that allow anonymous push/pull. The same credentials are used when the server resolves a published image digest (`start` / `export`). The server supports both standard registry bearer-token (`WWW-Authenticate: Bearer`) and pre-emptive Basic auth when resolving digests. ### Build Registry Environment The server's registry behavior is driven by a few closely related environment variables: - `KUBER_BUILD_REGISTRY` (default `registry.neko-piranha.ts.net`): the registry the in-cluster BuildKit pushes built images to and `kuber` uses as the image namespace. It also supplies the registry host for authentication. - `KUBER_INTERNAL_REGISTRY_HOST`: the host of an internal registry (for example the in-cluster distribution service) used for the BuildKit cache image and, when set, the image direct push target. When unset, push and cache fall back to `KUBER_BUILD_REGISTRY`. - `KUBER_INTERNAL_REGISTRY_INSECURE`: set to `"true"` to push to the internal registry over plain HTTP instead of HTTPS. Only meaningful when `KUBER_INTERNAL_REGISTRY_HOST` is set. - `KUBER_PUSH_IMAGE_PREFIX` (default `kuber/`): a prefix applied to project images pushed to the internal registry when `KUBER_INTERNAL_REGISTRY_HOST` is configured. - `KUBER_REGISTRY_RESOLVE_ORIGIN`: an explicit origin used to resolve a published image digest (used by `start` / `export`). Useful when the digest must be resolved from a different endpoint than the build/push registry, such as an internal HTTP registry. - `KUBER_REGISTRY_CONFIG`: path to the Docker config file with registry credentials (see above). - `KUBER_REGISTRY_SECRET`: the image pull Secret mounted for BuildKit's registry access. - `KUBER_BUILDKIT_IMAGE`: override the BuildKit runner image used for builds. - `KUBER_BUILD_DATA_CLAIM`: the PVC claim backing builds. - `KUBER_BUILD_RECONCILE_MS` (default `30000`): the background build reconciliation interval in milliseconds. Values must be greater than zero and no greater than `2147483647` (the JavaScript timer maximum); invalid values use the default. - `KUBER_BUILD_RECONCILE_TIMEOUT_MS` (default `20000`, maximum `25000`): the deadline for one background build-reconciliation scan in milliseconds. The maximum leaves time within the 30-second reconciliation lease for normal renewal or release. The reconciler renews both its workspace and per-build leases every 10 seconds for the lifetime of a generation, including while an observation, log read, or store call is pending. On timeout its abort signal is cancelled and logged, but Kubernetes requests may be unabortable; the generation remains active and continues its lease heartbeat until that request settles, so a later scan cannot overlap it. See [Server Deployment](#server-deployment) for how `compose.yml` wires the internal registry variables for the kuber-server pod. ## Account Recovery If you lose your credentials and cannot log in: 1. Recreate the bootstrap admin by deploying with `KUBER_BOOTSTRAP_PASSWORD` set again: ```bash KUBER_BOOTSTRAP_PASSWORD='new-password' kuber up --no-b kuber restart kuber-server ``` 2. The server only creates the bootstrap user if the account does not already exist, so a fresh `kuber login ` with the new password works, or use the newly created admin to reset other accounts: ```bash kuber users update --password ``` 3. After recovering, reconcile without `KUBER_BOOTSTRAP_PASSWORD` and restart so the bootstrap Secret is removed and the password leaves the pod environment. Because user records and session hashes are stored as Secrets in `kuber-system`, recovery relies on cluster administrators being able to redeploy the server with bootstrap credentials. `users revoke ` forcibly logs a user out across all devices. ## Next.js Example [`example/`](example/) contains a documented deployment template for adding kuber to an existing Bun-powered Next.js project without initializing or bundling an application in this repository. It includes a standalone-output Dockerfile, `.dockerignore`, `compose.yml`, and the required Next.js configuration. ## Running During development: ```bash bun run index.ts up ``` Run the dedicated unit suite and type checks: ```bash bun run test bun run typecheck ``` Other useful commands: ```bash bun run index.ts ps bun run index.ts logs bun run index.ts logs -f bun run index.ts exec app sh bun run index.ts start bun run index.ts stop bun run index.ts restart bun run index.ts rollback bun run index.ts fuck app bun run index.ts db ls bun run index.ts s3 ls bun run index.ts s3 creds app bun run index.ts s3 ui app bun run index.ts login dmgnr bun run index.ts users ls bun run index.ts operations ls bun run index.ts audit ls ``` All commands accept `--config` to use a configuration file other than `.kuberrc.ts`: ```bash kuber --config production.kuberrc.ts up kuber up --config production.kuberrc.ts ``` `login`, `logout`, and `whoami` are context-free and do not require a configuration. ### Shell Completion Generate and load completions for your shell: ```bash source <(kuber complete zsh) source <(kuber complete bash) ``` For a permanent setup, write the generated script to a file and source it from your shell configuration. Fish and PowerShell are also supported through `kuber complete fish` and `kuber complete powershell`. ## Commands - `up [--no-b]`: build images if needed, ensure the workspace, reconcile managed Postgres/S3, render manifests, apply them, wait for rollout, and delete stale resources - `start`: like `up` but re-resolves the currently published image digests instead of building - `login [username] [--persist]`: authenticate with the kuber API - `logout`: revoke and remove the current API session - `whoami`: show the authenticated API user and roles - `users`: administer user accounts and roles - `operations`: inspect server-side reconciliation operations - `audit`: inspect the audit trail - `ps [-a]`: print an ANSI graph of the current project namespace, hiding stopped deployments by default - `logs [deployment] [-f]`: print (or follow) logs for one deployment or all managed deployments - `exec `: execute a command inside a running deployment pod over an interactive WebSocket - `restart [deployment]`: roll out a restart across managed deployments - `stop`: delete the matching name-scoped HPAs (so autoscaling cannot scale replicas back up) and scale managed deployments to zero - `rollback` (alias `fuck`) `[deployment]`: roll one deployment back to its previous release, or all managed deployments when no name is given - `down [-f]`: delete managed resources while keeping ingress, PVCs, managed databases, and managed S3 storage; `-f` also deletes those and the namespace - `db ls` / `db creds `: list or print credentials for managed Postgres claims - `s3 ls` / `s3 creds ` / `s3 ui `: list managed S3 claims, print their credentials, or print the Garage UI URL for a bucket - `export [-o file]`: render manifests to a YAML file without applying them Lifecycle commands (`restart`, `stop`, `rollback`, `down`) are idempotent: identical requests are deduplicated server-side and tracked as operations. When an API response advertises a newer kuber version, the CLI quietly starts a cross-platform install with the current runtime's package manager and inherited environment. The CLI waits for the result, bounded by 30 seconds, and reports whether the update succeeded or a new version remains available. It does not replay the command; rerun it yourself if needed. ### Initialize a project or add an app `kuber init` creates a project Compose file and adds its first app. In an empty directory, the default app uses `nginx:stable` as its image; initialization always adds a service. With an interactive terminal, the first prompt selects the app source. The selected source then determines which follow-up fields are shown: an image URI for `image`, a CNB buildpack URI for `auto`, or an embedded Dockerfile template for `template`. ```bash kuber init kuber init --project my-site --name web --path apps/web --source image --image nginx:stable ``` With an interactive terminal, `init` prompts for missing project and app details. If multiple app roots are detected, an AutoComplete prompt lets you choose the app path; pass `--path` to select one directly. A YAML Snippet prompt then edits the project, service, claims, resources, and replica count together; the replica count defaults to `1`. Blank optional values are omitted from the generated configuration. The `managedBy` documentation header is added when the Compose file is written. Initialization logs in automatically when needed and trusts the current directory for the selected project, so a first `kuber up` does not require a separate `kuber trust`. Use `--non-interactive` to disable prompts and provide the required values as flags. The project name is optional and defaults from the current project context. If project configuration already fixes a project name, `--project` must match it. `kuber add app` adds another service to an existing project without replacing existing services. Its syntax and the shared app options are: ```text kuber add app [--name ] [--path ] [--source image|auto|template] [--image ] [--template