feat: k3s migration — k8s manifests, Traefik Ingress, CI deploy change

- Remove drizzle stage from Dockerfile (migration moved to k8s Job) - Add k8s manifests: deployments, services, ingress, HPA, PDB, RBAC - Replace Swarm deploy in CI with kubectl create job + kubectl set image - Add k8s trusted origin to better-auth config - Document all decisions in k8s/README.md
2026-06-19 12:26:15 +00:00
parent c842d9efd1
commit 0f2e21ec9a
16 changed files with 371 additions and 22 deletions
@@ -0,0 +1,113 @@
+# Buzz Events — k3s Deployment
+
+## Architecture Decisions
+
+### Ingress
+- **k3s built-in Traefik**, not nginx.
+- Annotations `traefik.ingress.kubernetes.io/response-buffering: "false"` + `read-timeout: "200"` preserve SSE (Server-Sent Events).
+- Traefik Middleware `sse-optimizer` caps `maxRequestBodyBytes` at 100 MB (for slip image uploads).
+- Standard `networking.k8s.io/v1` Ingress resource (not Traefik CRD IngressRoute) for portability.
+- External port: **80/443** (standard). `m.dgnr.us` DNS should point at k3s node IP(s). Old port 8604 is retired.
+
+### Namespace
+- Everything lives in namespace `buzz`.
+- The namespace is created once manually (`kubectl create ns buzz`) and listed in kustomization.yaml.
+
+### Secrets
+- **`buzz-env`** Secret created once from `.env`:
+  ```
+  kubectl create secret generic buzz-env --from-env-file=.env -n buzz
+  ```
+- All Deployments and the migration Job reference it via `envFrom.secretRef`.
+- Update by deleting and recreating (or editing in place):
+  ```
+  kubectl delete secret buzz-env -n buzz && kubectl create secret generic buzz-env --from-env-file=.env -n buzz
+  ```
+
+### Database Migration
+- **k8s Job** (deploy-time), not in Dockerfile.
+- The `drizzle` stage was removed from `Dockerfile`. The build no longer runs `bun dr push`.
+- CI creates the `db-migrate` Job imperatively, waits for completion, then updates deployments.
+- The Job uses the frontend image (has drizzle config) and `command: ["bun", "dr", "push"]`.
+- Old job is deleted before creating the new one (`kubectl delete job ... --ignore-not-found`).
+
+### Service Discovery
+- Frontend Service: `app` (ClusterIP, port 3000).
+- Backend reaches frontend at `http://app:3000` (same namespace, k8s DNS resolves `<service>`).
+- Backend references in `backend/index.ts:231,279` using `http://app:3000` remain unchanged.
+- `lib/auth.ts` `trustedOrigins` includes `http://app.buzz.svc.cluster.local:3000` for completeness.
+
+### CI/CD
+- Gitea Actions builds and pushes images as before (same tags: `:frontend`, `:backend`, `:frontend-$SHA`, `:backend-$SHA`).
+- Deploy step in `build.yml`:
+  1. Delete old migration Job, create fresh one with `:frontend-$SHA`, wait for completion.
+  2. `kubectl set image deployment/app` and `deployment/backend` with the new SHA tags.
+- Kustomize is used for initial cluster bootstrap (`kubectl apply -k k8s/`). Subsequent updates use imperative `kubectl set image` because Kustomize's `images` field cannot differentiate two images that share the same image name (different tags). If image naming changes to separate repos (e.g., `buzz-frontend:tag` / `buzz-backend:tag`), switch to `kustomize edit set image` + `kubectl apply -k .`.
+
+### CI Authentication
+- **ServiceAccount `ci-deployer`** in namespace `buzz`.
+- RBAC Role grants: Deployments (get/list/watch/update/patch), Pods (get/list/watch), Jobs (full CRUD), Services/Ingresses/Middlewares (get/list/watch/create/update/patch).
+- Token for the runner: `kubectl create token ci-deployer -n buzz` (or use a long-lived token from a Secret tied to the SA).
+- The runner also needs `docker` login access to `mts.dgnr.us:5000` (same as today).
+
+### Replicated Stateful Services (external to buzz namespace)
+- **PostgreSQL**: CloudNativePG operator. `DATABASE_URL` in Secret points to the CNPG cluster's read-write service (e.g. `postgres://...@<cluster>-rw.<cnpg-ns>.svc:5432/buzz`).
+- **Redis**: redis-operator. `REDIS_URL` in Secret points to the Redis operator's endpoint.
+
+### Container Registry
+- Same private registry `mts.dgnr.us:5000` running on the k3s control plane node.
+- All nodes must be able to pull from it. If registry uses HTTP (no TLS), configure `containerd` registries mirror or set `system-default-registry` in k3s.
+
+### Resource Limits
+- **app**: 2 replicas, limits 1 CPU / 1 Gi memory, requests 500m / 512 Mi. Liveness probe on `/api/health` (60s interval). Readiness probe same.
+- **backend**: 1 replica, limits 1 CPU / 512 Mi memory, requests 250m / 256 Mi. No probes (no HTTP endpoint).
+- **db-migrate Job**: limits 1 CPU / 512 Mi, requests 250m / 256 Mi.
+
+### HPA / PDB
+- HPA scales `app` on CPU > 70%, min 2 / max 6 replicas (replaces `swarm.autoscale` label).
+- PDB ensures min 1 pod always available during voluntary disruptions (node drains, rollouts).
+
+### TLS
+- Not configured in manifests. Currently HTTP-only behind Traefik. Add `spec.tls` + cert-manager or k3s Traefik's built-in Let's Encrypt when ready.
+
+## First Deploy (manual bootstrap)
+```bash
+kubectl create ns buzz
+kubectl create secret generic buzz-env --from-env-file=.env -n buzz
+kubectl apply -k k8s/
+```
+
+## Subsequent Deploys (CI)
+```bash
+# In .github/workflows/build.yml
+TAG=${{ gitea.sha }}
+
+# DB migration
+kubectl delete job db-migrate -n buzz --ignore-not-found
+kubectl create job db-migrate \
+  --image=mts.dgnr.us:5000/astral/buzz:frontend-$TAG \
+  -n buzz \
+  --command -- bun dr push
+kubectl wait --for=condition=complete job/db-migrate -n buzz --timeout=120s
+
+# Roll app & backend
+kubectl set image deployment/app app=mts.dgnr.us:5000/astral/buzz:frontend-$TAG -n buzz
+kubectl set image deployment/backend backend=mts.dgnr.us:5000/astral/buzz:backend-$TAG -n buzz
+```
+
+## Files Changed from Swarm
+| File | Change |
+|------|--------|
+| `Dockerfile` | Removed `drizzle` stage (migration moved to k8s Job) |
+| `lib/auth.ts` | Added `app.buzz.svc.cluster.local:3000` to `trustedOrigins` |
+| `.github/workflows/build.yml` | Replaced `docker service update` with k8s deploy commands |
+| `nginx/nginx.conf` | Archived — replaced by Traefik Ingress |
+| `docker-compose.yml` | Archived — replaced by k8s manifests |
+
+## Key Distinctions from Swarm
+- No `overlay` network (k8s CNI handles multi-node networking).
+- No host port `8604` (Traefik on 80/443).
+- Service discovery via k8s DNS, not Swarm DNS.
+- Rolling updates via `kubectl set image`, not `docker service update`.
+- Health checks via k8s probes, not Docker HEALTHCHECK.
+- Autoscaling via HPA, not `swarm.autoscale` label.
@@ -0,0 +1,56 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: app
+  namespace: buzz
+  labels:
+    app: buzz
+    component: app
+spec:
+  replicas: 2
+  strategy:
+    type: RollingUpdate
+    rollingUpdate:
+      maxUnavailable: 1
+      maxSurge: 1
+  selector:
+    matchLabels:
+      app: buzz
+      component: app
+  template:
+    metadata:
+      labels:
+        app: buzz
+        component: app
+    spec:
+      terminationGracePeriodSeconds: 5
+      containers:
+        - name: app
+          image: mts.dgnr.us:5000/astral/buzz:frontend
+          ports:
+            - containerPort: 3000
+              protocol: TCP
+          envFrom:
+            - secretRef:
+                name: buzz-env
+          resources:
+            limits:
+              cpu: "1"
+              memory: 1Gi
+            requests:
+              cpu: 500m
+              memory: 512Mi
+          livenessProbe:
+            httpGet:
+              path: /api/health
+              port: 3000
+            initialDelaySeconds: 10
+            periodSeconds: 60
+            timeoutSeconds: 5
+            failureThreshold: 3
+          readinessProbe:
+            httpGet:
+              path: /api/health
+              port: 3000
+            initialDelaySeconds: 5
+            periodSeconds: 10
@@ -0,0 +1,42 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: backend
+  namespace: buzz
+  labels:
+    app: buzz
+    component: backend
+spec:
+  replicas: 1
+  strategy:
+    type: RollingUpdate
+    rollingUpdate:
+      maxUnavailable: 0
+      maxSurge: 1
+  selector:
+    matchLabels:
+      app: buzz
+      component: backend
+  template:
+    metadata:
+      labels:
+        app: buzz
+        component: backend
+    spec:
+      terminationGracePeriodSeconds: 5
+      containers:
+        - name: backend
+          image: mts.dgnr.us:5000/astral/buzz:backend
+          env:
+            - name: NO_AUTH_CHECK
+              value: "true"
+          envFrom:
+            - secretRef:
+                name: buzz-env
+          resources:
+            limits:
+              cpu: "1"
+              memory: 512Mi
+            requests:
+              cpu: 250m
+              memory: 256Mi
@@ -0,0 +1,19 @@
+apiVersion: autoscaling/v2
+kind: HorizontalPodAutoscaler
+metadata:
+  name: app
+  namespace: buzz
+spec:
+  scaleTargetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: app
+  minReplicas: 2
+  maxReplicas: 6
+  metrics:
+    - type: Resource
+      resource:
+        name: cpu
+        target:
+          type: Utilization
+          averageUtilization: 70
@@ -0,0 +1,22 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: app
+  namespace: buzz
+  annotations:
+    traefik.ingress.kubernetes.io/response-buffering: "false"
+    traefik.ingress.kubernetes.io/read-timeout: "200"
+    traefik.ingress.kubernetes.io/router.middlewares: buzz-sse-optimizer@kubernetescrd
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: m.dgnr.us
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: app
+                port:
+                  number: 3000
@@ -0,0 +1,17 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: buzz
+
+resources:
+  - namespace.yaml
+  - deployment-app.yaml
+  - service-app.yaml
+  - deployment-backend.yaml
+  - ingress.yaml
+  - middleware.yaml
+  - hpa.yaml
+  - pdb.yaml
+  - service-account.yaml
+  - role.yaml
+  - role-binding.yaml
@@ -0,0 +1,8 @@
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: sse-optimizer
+  namespace: buzz
+spec:
+  buffering:
+    maxRequestBodyBytes: 104857600
@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: buzz
@@ -0,0 +1,11 @@
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: app
+  namespace: buzz
+spec:
+  minAvailable: 1
+  selector:
+    matchLabels:
+      app: buzz
+      component: app
@@ -0,0 +1,13 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: ci-deployer
+  namespace: buzz
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: ci-deployer
+subjects:
+  - kind: ServiceAccount
+    name: ci-deployer
+    namespace: buzz
@@ -0,0 +1,30 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: ci-deployer
+  namespace: buzz
+rules:
+  - apiGroups: ["apps"]
+    resources: ["deployments"]
+    verbs: ["get", "list", "watch", "update", "patch"]
+  - apiGroups: [""]
+    resources: ["pods"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: [""]
+    resources: ["services"]
+    verbs: ["get", "list", "watch", "create", "update", "patch"]
+  - apiGroups: ["batch"]
+    resources: ["jobs"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+  - apiGroups: ["networking.k8s.io"]
+    resources: ["ingresses"]
+    verbs: ["get", "list", "watch", "create", "update", "patch"]
+  - apiGroups: ["traefik.io"]
+    resources: ["middlewares"]
+    verbs: ["get", "list", "watch", "create", "update", "patch"]
+  - apiGroups: ["autoscaling"]
+    resources: ["horizontalpodautoscalers"]
+    verbs: ["get", "list", "watch", "create", "update", "patch"]
+  - apiGroups: ["policy"]
+    resources: ["poddisruptionbudgets"]
+    verbs: ["get", "list", "watch", "create", "update", "patch"]
@@ -0,0 +1,5 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: ci-deployer
+  namespace: buzz
@@ -0,0 +1,18 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: app
+  namespace: buzz
+  labels:
+    app: buzz
+    component: app
+spec:
+  selector:
+    app: buzz
+    component: app
+  ports:
+    - port: 3000
+      targetPort: 3000
+      protocol: TCP
+      name: http
+  type: ClusterIP