Chapter 0418 min read

Install on k3s / k8s

Deploy CCHP on a single k3s node with 11 manifest files, covering how to save the files, fill in your domains and secrets, deploy and complete the setup wizard, and back up, upgrade, and uninstall.

This page deploys CCHP on a single k3s node. Docker Compose runs all roles in one container; here, control (the console and the management API), gateway (the LLM gateway), and worker (background jobs) each run as a separate workload with its own resource limits, health checks, and logs, and you can restart each one on its own. The databases are a single-instance PostgreSQL 18 (with TimescaleDB) and Dragonfly on the same node. Choose this page if you already manage your servers with k3s or want to limit resources by role; otherwise, Install with Docker Compose is simpler.

Components of a single-node k3s deployment

CCHP

What you need#

A Linux server (amd64 or arm64) with k3s installed and the Traefik bundled with k3s turned off. For a new k3s installation:
bash
```
curl -sfL https://get.k3s.io | sh -s - --disable=traefik
```
The ingress-nginx ingress controller, installed. The Ingress in the manifests uses ingress-nginx annotations to turn off buffering for streaming responses and to set the read and write timeouts to 3600 seconds.
Two domains, both with DNS pointing to this server: the console domain (cchp.example.com on this page) and the API domain (api.cchp.example.com on this page). Also have a TLS certificate and private key that cover both domains.
Memory: the memory limits of all workloads total 5 GiB. Also leave room for what k3s and the ingress controller themselves use.
Disk: the local-path storage bundled with k3s doesn’t limit volume sizes, and all volumes share the node’s disk. The PersistentVolumeClaims in the manifests total about 481 GiB, so give the node at least that much disk.
The server can reach ghcr.io, Docker Hub, docker.dragonflydb.io, and cch-plus.com (the model price list).

Run all the commands on this page on this server as the root user. k3s includes kubectl.

Step 1: Save the manifest files#

Create a directory named cchp-k3s and save the following 11 files in it:

bash

mkdir cchp-k3s

Namespace:

cchp-k3s/00-namespace.yamlyaml

apiVersion: v1
kind: Namespace
metadata:
  name: cchp

Database passwords and secrets. You replace the placeholder values in step 2:

cchp-k3s/01-secrets.yamlyaml

# Replace the placeholder values with random values in step 2
apiVersion: v1
kind: Secret
metadata:
  name: cchp-migration-secrets
  namespace: cchp
stringData:
  POSTGRES_PASSWORD: CHANGE_ME_OWNER_PASSWORD
  ATLAS_DATABASE_URL: postgres://hub:CHANGE_ME_OWNER_PASSWORD@postgres:5432/app?sslmode=disable
  CCHP_DATABASE_RUNTIME_PASSWORD: CHANGE_ME_RUNTIME_PASSWORD
---
apiVersion: v1
kind: Secret
metadata:
  name: cchp-secrets
  namespace: cchp
stringData:
  DATABASE_URL: postgres://cchp_runtime:CHANGE_ME_RUNTIME_PASSWORD@postgres:5432/app?sslmode=disable
  DRAGONFLY_URL: redis://dragonfly:6379/0
  CCH_TOKEN_PEPPER: CHANGE_ME_TOKEN_PEPPER
  CCH_MASTER_KEY: CHANGE_ME_MASTER_KEY
  DRAGONFLY_KEY_SECRET: CHANGE_ME_DRAGONFLY_KEY_SECRET

Shared configuration. You replace the three URLs in step 2:

cchp-k3s/02-config.yamlyaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: cchp-config
  namespace: cchp
data:
  APP_ENV: production
  CCHP_DATABASE_RUNTIME_ROLE: cchp_runtime
  CCH_PUBLIC_BASE_URL: https://cchp.example.com # Console URL
  CCH_CONTROL_PLANE_ALLOWED_ORIGINS: https://cchp.example.com # Same as the console URL
  CCH_GATEWAY_PUBLIC_URL: https://api.cchp.example.com # API URL
  CCH_SIEM_MODE: disabled # Don't deliver audit events to an external SIEM
  # Capacity of the database disk: all local-path volumes are on the same disk, so reading the shared volume is enough
  CCHP_DATABASE_STORAGE_SOURCE: filesystem
  CCHP_DATABASE_STORAGE_PATH: /var/lib/cchp/data/objectstore

The two volumes that control, gateway, and worker share:

cchp-k3s/03-volumes.yamlyaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cchp-objectstore
  namespace: cchp
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 20Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cchp-task-results
  namespace: cchp
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 50Gi

PostgreSQL:

cchp-k3s/04-postgres.yamlyaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: cchp
spec:
  serviceName: postgres
  selector:
    matchLabels:
      app.kubernetes.io/name: postgres
  template:
    metadata:
      labels:
        app.kubernetes.io/name: postgres
    spec:
      terminationGracePeriodSeconds: 60
      securityContext:
        runAsNonRoot: true
        runAsUser: 70
        runAsGroup: 70
        fsGroup: 70
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: postgres
          image: timescale/timescaledb:2.31.0-pg18
          args:
            - postgres
            - -c
            - shared_preload_libraries=timescaledb,pg_stat_statements
            - -c
            - pg_stat_statements.max=10000
            - -c
            - track_io_timing=on
            - -c
            - max_connections=200
          env:
            - name: POSTGRES_USER
              value: hub
            - name: POSTGRES_DB
              value: app
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: cchp-migration-secrets
                  key: POSTGRES_PASSWORD
            - name: POSTGRES_INITDB_ARGS
              value: --auth-host=scram-sha-256
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          ports:
            - name: postgres
              containerPort: 5432
          readinessProbe:
            exec:
              command: ["sh", "-c", "pg_isready -h 127.0.0.1 -U hub -d app"]
            periodSeconds: 5
          livenessProbe:
            tcpSocket:
              port: postgres
            initialDelaySeconds: 30
            failureThreshold: 6
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              memory: 2Gi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
            - name: run
              mountPath: /var/run/postgresql
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: run
          emptyDir: {}
        - name: tmp
          emptyDir: {}
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 20Gi
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: cchp
spec:
  selector:
    app.kubernetes.io/name: postgres
  ports:
    - port: 5432

Dragonfly:

cchp-k3s/05-dragonfly.yamlyaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: dragonfly
  namespace: cchp
spec:
  serviceName: dragonfly
  selector:
    matchLabels:
      app.kubernetes.io/name: dragonfly
  template:
    metadata:
      labels:
        app.kubernetes.io/name: dragonfly
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
        runAsGroup: 999
        fsGroup: 999
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: dragonfly
          image: docker.dragonflydb.io/dragonflydb/dragonfly:v2.0.2
          args: ["--proactor_threads=2", "--maxmemory=768mb", "--dir=/data"]
          ports:
            - name: redis
              containerPort: 6379
          readinessProbe:
            exec:
              command: ["redis-cli", "ping"]
            periodSeconds: 5
          livenessProbe:
            tcpSocket:
              port: redis
            initialDelaySeconds: 10
            failureThreshold: 6
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              memory: 1Gi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 2Gi
---
apiVersion: v1
kind: Service
metadata:
  name: dragonfly
  namespace: cchp
spec:
  selector:
    app.kubernetes.io/name: dragonfly
  ports:
    - port: 6379

Database migration. The init container wait-for-migrations of control, gateway, and worker waits for it to finish before starting each process:

cchp-k3s/06-migrate-job.yamlyaml

apiVersion: batch/v1
kind: Job
metadata:
  name: cchp-migrate
  namespace: cchp
spec:
  activeDeadlineSeconds: 44640
  ttlSecondsAfterFinished: 3600
  template:
    spec:
      restartPolicy: Never
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: migrate
          image: ghcr.io/cch-hq/cch-plus:latest
          imagePullPolicy: IfNotPresent
          command: ["/usr/local/bin/migrate"]
          args: ["apply"]
          env:
            - name: APP_ENV
              valueFrom:
                configMapKeyRef:
                  name: cchp-config
                  key: APP_ENV
            - name: CCHP_DATABASE_RUNTIME_ROLE
              valueFrom:
                configMapKeyRef:
                  name: cchp-config
                  key: CCHP_DATABASE_RUNTIME_ROLE
            - name: ATLAS_DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: cchp-migration-secrets
                  key: ATLAS_DATABASE_URL
            - name: CCHP_DATABASE_RUNTIME_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: cchp-migration-secrets
                  key: CCHP_DATABASE_RUNTIME_PASSWORD
            - name: CCH_MASTER_KEY
              valueFrom:
                secretKeyRef:
                  name: cchp-secrets
                  key: CCH_MASTER_KEY
            - name: CCH_SCRATCH_DIR
              value: /var/cache/cchp
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: "1"
              memory: 512Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: scratch
              mountPath: /var/cache/cchp
      volumes:
        - name: scratch
          emptyDir: {}

control:

cchp-k3s/07-control.yamlyaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: control
  namespace: cchp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: control
  template:
    metadata:
      labels:
        app: control
    spec:
      securityContext:
        runAsNonRoot: true
        fsGroup: 65532
        fsGroupChangePolicy: OnRootMismatch
        seccompProfile:
          type: RuntimeDefault
      initContainers:
        - name: wait-for-migrations
          image: ghcr.io/cch-hq/cch-plus:latest
          imagePullPolicy: IfNotPresent
          args: ["db", "wait", "--timeout=744m"]
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: cchp-secrets
                  key: DATABASE_URL
            - name: CCHP_DATABASE_RUNTIME_ROLE
              valueFrom:
                configMapKeyRef:
                  name: cchp-config
                  key: CCHP_DATABASE_RUNTIME_ROLE
          resources:
            requests:
              cpu: 10m
              memory: 64Mi
            limits:
              cpu: 200m
              memory: 256Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
      containers:
        - name: control
          image: ghcr.io/cch-hq/cch-plus:latest
          imagePullPolicy: IfNotPresent
          args: ["api", "--role=control"]
          envFrom:
            - configMapRef:
                name: cchp-config
            - secretRef:
                name: cchp-secrets
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: CCH_NODE_INTERNAL_URL
              value: http://$(POD_IP):8092
            - name: CCH_CONTROL_OBSERVABILITY_ADDR
              value: ":8085"
          ports:
            - name: http
              containerPort: 8080
            - name: node-internal
              containerPort: 8092
            - name: metrics
              containerPort: 8085
          livenessProbe:
            httpGet: { path: /livez, port: http }
            initialDelaySeconds: 30
          readinessProbe:
            httpGet: { path: /readyz, port: http }
            initialDelaySeconds: 5
            periodSeconds: 5
            failureThreshold: 2
          startupProbe:
            httpGet: { path: /livez, port: http }
            periodSeconds: 5
            failureThreshold: 30
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 512Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: objectstore
              mountPath: /var/lib/cchp/data/objectstore
            - name: task-results
              mountPath: /var/lib/cchp/data/task-results
            - name: scratch
              mountPath: /var/cache/cchp
      volumes:
        - name: objectstore
          persistentVolumeClaim:
            claimName: cchp-objectstore
        - name: task-results
          persistentVolumeClaim:
            claimName: cchp-task-results
        - name: scratch
          ephemeral:
            volumeClaimTemplate:
              spec:
                accessModes: ["ReadWriteOnce"]
                resources:
                  requests:
                    storage: 64Gi
---
apiVersion: v1
kind: Service
metadata:
  name: control
  namespace: cchp
spec:
  selector:
    app: control
  ports:
    - port: 8080

gateway:

cchp-k3s/08-gateway.yamlyaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: gateway
  namespace: cchp
spec:
  serviceName: gateway-hl
  replicas: 1
  selector:
    matchLabels:
      app: gateway
  template:
    metadata:
      labels:
        app: gateway
    spec:
      # Wait up to 900 seconds before stopping so that in-flight requests can finish
      terminationGracePeriodSeconds: 930
      securityContext:
        runAsNonRoot: true
        fsGroup: 65532
        fsGroupChangePolicy: OnRootMismatch
        seccompProfile:
          type: RuntimeDefault
      initContainers:
        - name: wait-for-migrations
          image: ghcr.io/cch-hq/cch-plus:latest
          imagePullPolicy: IfNotPresent
          args: ["db", "wait", "--timeout=744m"]
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: cchp-secrets
                  key: DATABASE_URL
            - name: CCHP_DATABASE_RUNTIME_ROLE
              valueFrom:
                configMapKeyRef:
                  name: cchp-config
                  key: CCHP_DATABASE_RUNTIME_ROLE
          resources:
            requests:
              cpu: 10m
              memory: 64Mi
            limits:
              cpu: 200m
              memory: 256Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
      containers:
        - name: gateway
          image: ghcr.io/cch-hq/cch-plus:latest
          imagePullPolicy: IfNotPresent
          args: ["api", "--role=gateway", "--addr=:8081"]
          envFrom:
            - configMapRef:
                name: cchp-config
            - secretRef:
                name: cchp-secrets
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: CCH_ARCHIVE_INTERNAL_URL
              value: http://$(POD_NAME).gateway-hl:8091
            - name: CCH_GATEWAY_OBSERVABILITY_ADDR
              value: ":8084"
            - name: CCH_REPLAY_OWNER_PROXY_NODES
              value: gateway-0=http://gateway-0.gateway-hl:8081
            - name: CCH_ALLOW_EMPTY_ROUTING_BOOTSTRAP
              value: "true"
          ports:
            - name: http
              containerPort: 8081
            - name: archive
              containerPort: 8091
            - name: metrics
              containerPort: 8084
          livenessProbe:
            httpGet: { path: /livez, port: http }
            initialDelaySeconds: 30
          readinessProbe:
            httpGet: { path: /readyz, port: http }
            initialDelaySeconds: 5
            periodSeconds: 5
            failureThreshold: 2
          startupProbe:
            httpGet: { path: /livez, port: http }
            periodSeconds: 5
            failureThreshold: 30
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              cpu: "2"
              memory: 1Gi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: objectstore
              mountPath: /var/lib/cchp/data/objectstore
            - name: task-results
              mountPath: /var/lib/cchp/data/task-results
            - name: archive
              mountPath: /var/lib/cchp/node
            - name: scratch
              mountPath: /var/cache/cchp
      volumes:
        - name: objectstore
          persistentVolumeClaim:
            claimName: cchp-objectstore
        - name: task-results
          persistentVolumeClaim:
            claimName: cchp-task-results
        - name: scratch
          ephemeral:
            volumeClaimTemplate:
              spec:
                accessModes: ["ReadWriteOnce"]
                resources:
                  requests:
                    storage: 9Gi
  volumeClaimTemplates:
    - metadata:
        name: archive
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 16Gi
---
apiVersion: v1
kind: Service
metadata:
  name: gateway
  namespace: cchp
spec:
  selector:
    app: gateway
  ports:
    - port: 8081
---
# Stable DNS name for each gateway Pod
apiVersion: v1
kind: Service
metadata:
  name: gateway-hl
  namespace: cchp
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    app: gateway
  ports:
    - name: http
      port: 8081
    - name: archive
      port: 8091

worker:

cchp-k3s/09-worker.yamlyaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: worker
  namespace: cchp
spec:
  # Only 1 replica can run
  replicas: 1
  selector:
    matchLabels:
      app: worker
  template:
    metadata:
      labels:
        app: worker
    spec:
      terminationGracePeriodSeconds: 45
      securityContext:
        runAsNonRoot: true
        fsGroup: 65532
        fsGroupChangePolicy: OnRootMismatch
        seccompProfile:
          type: RuntimeDefault
      initContainers:
        - name: wait-for-migrations
          image: ghcr.io/cch-hq/cch-plus:latest
          imagePullPolicy: IfNotPresent
          args: ["db", "wait", "--timeout=744m"]
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: cchp-secrets
                  key: DATABASE_URL
            - name: CCHP_DATABASE_RUNTIME_ROLE
              valueFrom:
                configMapKeyRef:
                  name: cchp-config
                  key: CCHP_DATABASE_RUNTIME_ROLE
          resources:
            requests:
              cpu: 10m
              memory: 64Mi
            limits:
              cpu: 200m
              memory: 256Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
      containers:
        - name: worker
          image: ghcr.io/cch-hq/cch-plus:latest
          imagePullPolicy: IfNotPresent
          args: ["api", "--role=worker"]
          envFrom:
            - configMapRef:
                name: cchp-config
            - secretRef:
                name: cchp-secrets
          env:
            - name: CCH_WORKER_OBSERVABILITY_ADDR
              value: ":8082"
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: CCH_NODE_INTERNAL_URL
              value: http://$(POD_IP):8093
          ports:
            - name: operations
              containerPort: 8082
            - name: node-internal
              containerPort: 8093
          livenessProbe:
            httpGet: { path: /healthz, port: operations }
            timeoutSeconds: 2
          readinessProbe:
            httpGet: { path: /readyz, port: operations }
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 2
          startupProbe:
            httpGet: { path: /healthz, port: operations }
            periodSeconds: 2
            failureThreshold: 30
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: "1"
              memory: 512Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: objectstore
              mountPath: /var/lib/cchp/data/objectstore
            - name: task-results
              mountPath: /var/lib/cchp/data/task-results
            - name: scratch
              mountPath: /var/cache/cchp
      volumes:
        - name: objectstore
          persistentVolumeClaim:
            claimName: cchp-objectstore
        - name: task-results
          persistentVolumeClaim:
            claimName: cchp-task-results
        - name: scratch
          ephemeral:
            volumeClaimTemplate:
              spec:
                accessModes: ["ReadWriteOnce"]
                resources:
                  requests:
                    storage: 300Gi

Ingress. The console domain routes to control, and the API domain routes only the paths /v, /v1, /v1beta, /v4, and /ws:

cchp-k3s/10-ingress.yamlyaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: cchp
  namespace: cchp
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - api.cchp.example.com # API domain
        - cchp.example.com # Console domain
      secretName: cchp-tls-cert
  rules:
    - host: api.cchp.example.com # API domain
      http:
        paths:
          - path: /v
            pathType: Prefix
            backend: &gateway
              service:
                name: gateway
                port:
                  number: 8081
          - path: /v1
            pathType: Prefix
            backend: *gateway
          - path: /v1beta
            pathType: Prefix
            backend: *gateway
          - path: /v4
            pathType: Prefix
            backend: *gateway
          - path: /ws
            pathType: Prefix
            backend: *gateway
    - host: cchp.example.com # Console domain
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: control
                port:
                  number: 8080
---
# CC Hub export file uploads: no request body size limit, forwarded while being received
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: cchp-cchub-import-upload
  namespace: cchp
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - cchp.example.com # Console domain
      secretName: cchp-tls-cert
  rules:
    - host: cchp.example.com # Console domain
      http:
        paths:
          - path: /api/admin/cchub-import/upload
            pathType: Exact
            backend:
              service:
                name: control
                port:
                  number: 8080

Step 2: Fill in your domains and secrets#

Replace the example domains with your own domains. First change the two variables on the first two lines, then run:

bash

API_DOMAIN=api.cchp.example.com   # Your API domain
CONSOLE_DOMAIN=cchp.example.com   # Your console domain
sed -i -e "s/api\.cchp\.example\.com/$API_DOMAIN/g" -e "s/cchp\.example\.com/$CONSOLE_DOMAIN/g" cchp-k3s/*.yaml
grep -n -e "$API_DOMAIN" -e "$CONSOLE_DOMAIN" cchp-k3s/*.yaml

Result: The output lists 3 lines in 02-config.yaml and 6 lines in 10-ingress.yaml, all with your domains.

Generate a random value for each placeholder value:

bash

for name in OWNER_PASSWORD RUNTIME_PASSWORD TOKEN_PEPPER MASTER_KEY DRAGONFLY_KEY_SECRET; do
  sed -i "s/CHANGE_ME_$name/$(openssl rand -hex 32)/g" cchp-k3s/01-secrets.yaml
done
grep -n CHANGE_ME cchp-k3s/01-secrets.yaml

Result: grep prints nothing. When the same placeholder value appears in two places, both places get the same value.

Warning: CCH_MASTER_KEY in 01-secrets.yaml is the master encryption key. If you lose or change it, data that was stored encrypted can no longer be decrypted. Back up this file to a safe location. Don’t regenerate its values after the first deployment: the database passwords are written only when PostgreSQL starts for the first time.

Always open the console at the console domain. Sign-in cookies, cross-site request checks, passkeys, and the callback URLs for third-party sign-in are all bound to CCH_PUBLIC_BASE_URL. For LAN testing only, you can turn on the reduced-security deployment and run with http URLs and LAN host names; see Reduced-security deployment (LAN testing).

Step 3: Deploy#

Create the namespace and the Secret for the TLS certificate, then apply all the files. Replace fullchain.pem and privkey.pem with your certificate and private key files:

bash

kubectl apply -f cchp-k3s/00-namespace.yaml
kubectl -n cchp create secret tls cchp-tls-cert --cert=fullchain.pem --key=privkey.pem
kubectl apply -f cchp-k3s/

If you issue the certificate with cert-manager, skip the second command. Instead, add cert-manager.io/cluster-issuer: <your ClusterIssuer name> under annotations of the cchp Ingress in 10-ingress.yaml. The other Ingress reuses the same certificate.

Result: kubectl lists the created resources line by line. Until PostgreSQL is ready, the migration Job fails and retries automatically a few times. This is expected. The control, gateway, and worker Pods stay at the init container wait-for-migrations until the migration completes.

Step 4: Confirm that the services are ready#

bash

kubectl -n cchp wait --for=condition=complete job/cchp-migrate --timeout=744m
kubectl -n cchp get pods
kubectl get --raw /api/v1/namespaces/cchp/services/gateway:8081/proxy/readyz
curl -s https://cchp.example.com/readyz

--timeout is the same as the maximum run time of the migration Job. On a fresh installation, the migration usually completes within a few minutes.

Result:

The first command prints job.batch/cchp-migrate condition met.
The control, gateway-0, worker, postgres-0, and dragonfly-0 Pods are all Running with READY at 1/1. The latest Pod of the migration Job is Completed, and Pods that failed before PostgreSQL was ready show Error.
Both /readyz requests return "status":"ready": the first checks the gateway directly, and the second checks control through the Ingress and TLS.

Step 5: Complete the setup wizard#

Get the one-time setup token from the control logs:
bash
```
kubectl -n cchp logs deploy/control | grep 'CCHP setup token'
```
Result: The output is one line, CCHP setup token: cchp_setup_.... If you can’t find it in the logs, see “Can’t find the setup token in the logs” under “Frequently asked questions”.
Open https://cchp.example.com in your browser. On the Enter the setup token page, paste the token into Setup token and click Verify and continue.
Complete the seven steps in order: Site, Administrator, Two-step, Email, Sign-in, Database, and Organization. You can skip Two-step and Email. For how to fill in each step, see Step 7: Complete the setup wizard in Install with Docker Compose. The steps are the same for all three installation methods.
- Site: Public URL shows https://cchp.example.com.
- Database (the page title is Check database observability): The migration Job has already granted the permissions for the built-in PostgreSQL, so the page shows Database observability is ready. Click Continue.
- Organization: Enter the Organization name and the Team name, choose Prepaid or Postpaid under Organization type, then click Create and continue. Creating the first organization completes the installation.

Result: The browser opens the console. There are no providers yet, so the gateway returns a no-route error for proxied requests. Later sections show how to create providers and LLM keys.

View logs#

bash

kubectl -n cchp logs deploy/control -f
kubectl -n cchp logs statefulset/gateway -f
kubectl -n cchp logs deploy/worker -f
kubectl -n cchp logs job/cchp-migrate

The migration Job is deleted automatically 1 hour after it completes, and its logs are deleted with it. control, gateway, and worker serve Prometheus metrics at /metrics on Pod ports 8085, 8084, and 8082 respectively. If you need monitoring, scrape each Pod.

Backup#

Content	Location	Recommendation
Secrets	`cchp-k3s/01-secrets.yaml`	Must be backed up. The master encryption key and the token hashing key are stored only here and in the cluster’s Secrets
Database	Volume `data-postgres-0`	Must be backed up. Export it with `pg_dump`
Node data such as request archives	Volume `archive-gateway-0`	Back it up separately from the database
Shared data	Volumes `cchp-objectstore` and `cchp-task-results`	Plugin packages, export files, audit archives, CC Hub uploads, and task result files

Each local-path volume is a directory on the node under /var/lib/rancher/k3s/storage/ whose name ends with the volume name.

Export the database:

bash

kubectl -n cchp exec postgres-0 -- pg_dump -U hub -d app -Fc > cchp-$(date +%F).dump

Result: A backup file named cchp- plus the date appears in the current directory. If pg_dump reports circular foreign-key constraints, you can ignore it. It’s a notice about TimescaleDB internal tables. For how to restore, see “Roll back to an earlier version”.

Upgrade#

latest is the latest stable release. Before you upgrade, export the database as described in “Backup”.

bash

k3s crictl pull ghcr.io/cch-hq/cch-plus:latest
kubectl -n cchp scale deployment/control deployment/worker --replicas=0
kubectl -n cchp delete job cchp-migrate --ignore-not-found
for f in cchp-k3s/0[0-6]-*.yaml; do kubectl apply -f "$f"; done
kubectl -n cchp wait --for=condition=complete job/cchp-migrate --timeout=744m
kubectl apply -f cchp-k3s/09-worker.yaml
kubectl -n cchp rollout status deployment/worker
kubectl apply -f cchp-k3s/
kubectl -n cchp rollout restart statefulset/gateway
kubectl -n cchp rollout status deployment/control
kubectl -n cchp rollout status statefulset/gateway

What each command does:

The manifests pull the CCHP image only when it isn’t already on the node (imagePullPolicy: IfNotPresent), so restarting a Pod never switches to a new version by surprise. That’s why you first get the new latest with k3s crictl pull.
Stop control and worker, and delete the previous migration Job: a Job can’t be modified, and it doesn’t run again after it completes.
Apply files 00 through 06 to run only the migration with the new version. Meanwhile, the gateway keeps handling requests with the old version.
After the migration completes, start the new version of the worker first. Then apply all the files to start control and restart the gateway. The gateway waits up to 900 seconds before stopping so that in-flight requests can finish, so the last step can take more than 10 minutes.

Result: Current version in System Management → About shows the new version number. The console is unavailable from the moment control stops until it’s ready again.

If the manifest files change in a new version, first update the files in cchp-k3s/ from this page, then run the commands above.

To run a specific version, replace latest in the files with the version number, for example 1.0.11, then run all the commands above except the first. If that version isn’t on the node, the Pods pull it automatically when they start:

bash

sed -i 's/cch-plus:[^ ]*/cch-plus:1.0.11/' cchp-k3s/*.yaml

Roll back to an earlier version#

Migrations only run forward. Switching back to the old image alone doesn’t undo the changes the new version made to the database, and the old version’s wait-for-migrations may exit with an error. To roll back, rebuild the database from the backup you took before the upgrade:

bash

kubectl -n cchp scale deployment/control deployment/worker statefulset/gateway --replicas=0
sed -i 's/cch-plus:[^ ]*/cch-plus:1.0.11/' cchp-k3s/*.yaml
kubectl -n cchp exec postgres-0 -- psql -U hub -d postgres -c 'DROP DATABASE app WITH (FORCE)' -c 'CREATE DATABASE app'
kubectl -n cchp exec postgres-0 -- psql -U hub -d app -c 'CREATE EXTENSION IF NOT EXISTS timescaledb' -c 'SELECT timescaledb_pre_restore()'
kubectl -n cchp exec -i postgres-0 -- pg_restore -U hub -d app < cchp-YYYY-MM-DD.dump
kubectl -n cchp exec postgres-0 -- psql -U hub -d app -c 'SELECT timescaledb_post_restore()'
kubectl -n cchp delete job cchp-migrate --ignore-not-found
kubectl apply -f cchp-k3s/

Replace 1.0.11 with the version you were running when you took the backup, and replace the file name with your backup file. timescaledb_pre_restore() and timescaledb_post_restore() are the pre-restore and post-restore steps that TimescaleDB requires.

Result: pg_restore prints no errors; the migration Job confirms that all the migrations for this version have run, and control, gateway, and worker resume service.

Uninstall#

Warning: Deleting the cchp namespace also deletes all the Secrets and volumes in it. The local-path reclaim policy is Delete, so the database, request archives, and uploaded files are permanently deleted. Before you uninstall, export the database as described in “Backup” and keep 01-secrets.yaml.

bash

kubectl delete namespace cchp

Result: kubectl get namespace cchp reports that the namespace can’t be found. The ingress controller and k3s itself are not affected.

Frequently asked questions#

Pods stay at Init:0/1#

The control, gateway, and worker Pods stay at the init container wait-for-migrations until the migration completes. Check the migration Job with kubectl -n cchp get job cchp-migrate and kubectl -n cchp logs job/cchp-migrate. If the logs show password authentication failed, the database passwords in 01-secrets.yaml were regenerated after the first deployment. Switch back to the original file. After you fix it, run kubectl -n cchp delete job cchp-migrate and kubectl apply -f cchp-k3s/.

A process exits right after it starts#

Check the logs of the previous start with kubectl -n cchp logs deploy/control --previous. If they show CCH_MASTER_KEY hex decode, 01-secrets.yaml still has placeholder values that weren’t replaced. Replace them as described in step 2, then run kubectl apply -f cchp-k3s/01-secrets.yaml and kubectl -n cchp rollout restart deployment/control deployment/worker statefulset/gateway.

The setup wizard shows “You opened a different address”#

The address open in the browser is different from CCH_PUBLIC_BASE_URL. Open the console URL in 02-config.yaml, or change this file and then run kubectl apply -f cchp-k3s/02-config.yaml and kubectl -n cchp rollout restart deployment/control deployment/worker statefulset/gateway.

Can’t find the setup token in the logs#

Print the current token:

bash

kubectl -n cchp exec deploy/control -- gateway identity setup-token

If the token was already used but the installation didn’t finish, run kubectl -n cchp rollout restart deployment/control, and control prints a new token in its logs. Once an administrator has been created, the command fails with installation already has users. Sign in directly.

The domain doesn’t open, or it returns 404#

Make sure that Traefik is turned off and that ingress-nginx is running: the output of kubectl get ingressclass should include nginx. The Traefik bundled with k3s doesn’t handle Ingresses whose ingressClassName is nginx.

Installation overview
Install with Docker Compose: the steps of the setup wizard
Install from binaries

What you need#

Step 1: Save the manifest files#

Step 2: Fill in your domains and secrets#

Step 3: Deploy#

Step 4: Confirm that the services are ready#

Step 5: Complete the setup wizard#

View logs#

Backup#

Upgrade#

Roll back to an earlier version#

Uninstall#

Frequently asked questions#

Pods stay at Init:0/1#

A process exits right after it starts#

The setup wizard shows “You opened a different address”#

Can’t find the setup token in the logs#

The domain doesn’t open, or it returns 404#

Related pages#