# Kubernetes

Flowplan on Kubernetes with the Helm chart: from one pod with SQLite on a volume to several pods with high availability against PostgreSQL, Redis and S3.

## Two ways to run it (since 0.10.0)

| | One pod | Several pods (cluster mode) |
| --- | --- | --- |
| For | up to about 20 people active at the same time | more people, updates without downtime, losing a node |
| Data | SQLite and files on a volume (PVC) | PostgreSQL, files in S3, Redis between the pods |
| Updates | short downtime (`Recreate`) | rolling, pod by pod |
| Example values | [`values-small.yaml`](https://github.com/yannicschuller/flowplan/blob/main/deploy/helm/flowplan/values-small.yaml) | [`values-ha.yaml`](https://github.com/yannicschuller/flowplan/blob/main/deploy/helm/flowplan/values-ha.yaml) |

How many pods and how much CPU and memory: see [Sizing](/en/sizing). What the pods share and how they work together: see [Cluster mode](/en/cluster).

## Requirements (since 0.10.0)

- **Kubernetes 1.23** or newer, **Helm 3**, an ingress controller (ingress-nginx is set up by default) and a certificate for your address (e.g. cert-manager).
- **One pod:** a block or local volume (`ReadWriteOnce`) – no NFS, SQLite can break on it.
- **Several pods:** **PostgreSQL 16** or newer, **Redis** (tested with 7, no persistence needed) and an **S3 bucket** (MinIO, Garage, AWS …). The chart does not install any of them – use managed services or their own charts or operators. The PostgreSQL role should own the database but not be a superuser ([Role without superuser rights](/en/postgresql#role-without-superuser-rights)); PostgreSQL's `max_connections` must stay above `replicas × (cluster.poolSize + 3)`.

> [!IMPORTANT]
> The chart takes no secret as a value. Passwords, keys and connection strings go into Secrets you create; the chart only names them. Only the setup token can be generated by the chart.

## Install with Helm (since 0.10.0)

The chart lives in the repository under [`deploy/helm/flowplan`](https://github.com/yannicschuller/flowplan/tree/main/deploy/helm/flowplan); its README describes every value.

::: tabs

== One pod

```bash
git clone https://github.com/yannicschuller/flowplan.git && cd flowplan
kubectl create namespace flowplan
kubectl -n flowplan create secret generic flowplan-env \
  --from-literal=FLOWPLAN_ENCRYPTION_KEY="$(openssl rand -base64 32)"
cp deploy/helm/flowplan/values-small.yaml my-values.yaml   # address, host, SMTP, S3
helm install flowplan deploy/helm/flowplan -n flowplan -f my-values.yaml
```

The chart creates the Secret `flowplan-setup` with a random setup token, keeps it on upgrades and does not delete it on uninstall.

== Several pods

```bash
kubectl create namespace flowplan
kubectl -n flowplan create secret generic flowplan-cluster \
  --from-literal=DATABASE_URL='postgres://flowplan:<password>@postgres:5432/flowplan' \
  --from-literal=REDIS_URL='redis://:<password>@redis-master:6379/0'
kubectl -n flowplan create secret generic flowplan-s3 \
  --from-literal=S3_ACCESS_KEY_ID='<key-id>' \
  --from-literal=S3_SECRET_ACCESS_KEY='<secret>'
# The keys every pod shares (push keys: npx web-push generate-vapid-keys)
kubectl -n flowplan create secret generic flowplan-keys \
  --from-literal=FLOWPLAN_ENCRYPTION_KEY="$(openssl rand -base64 32)" \
  --from-literal=WEB_PUSH_PUBLIC_KEY='<public VAPID key>' \
  --from-literal=WEB_PUSH_PRIVATE_KEY='<private VAPID key>'
kubectl -n flowplan create secret generic flowplan-env \
  --from-literal=OIDC_CLIENT_SECRET='<client secret>' \
  --from-literal=FLOWPLAN_METRICS_TOKEN="$(openssl rand -hex 24)"
kubectl -n flowplan create secret generic flowplan-setup \
  --from-literal=FLOWPLAN_SETUP_TOKEN="$(openssl rand -hex 20)"
cp deploy/helm/flowplan/values-ha.yaml my-values.yaml   # address, hosts, bucket, replicas
helm install flowplan deploy/helm/flowplan -n flowplan -f my-values.yaml
```

Before the pods start, a Job brings the database schema up to this version. A pod is ready once its schema and data steps are done.

:::

Then:

```bash
helm test flowplan -n flowplan
echo "https://flowplan.example.org/register?setup=$(kubectl -n flowplan get secret flowplan-setup \
  -o jsonpath='{.data.FLOWPLAN_SETUP_TOKEN}' | base64 -d)"
```

Open the link and create the first account – it administers the instance. Keep a copy of `FLOWPLAN_ENCRYPTION_KEY` apart from the backups: without it, encrypted files and backups cannot be read.

### What the chart sets up

- **Probes:** one pod on `/api/health`; several pods with startup and liveness on `/api/health/live` and readiness on `/api/health/ready` – ready only once the schema and the data steps are done and the database and the bucket answer. Redis does not count by default (`cluster.readyRequire: database,objectStorage`): if Redis fails, every pod stays in the rotation and editing goes on.
- **Ingress:** for ingress-nginx, no buffering, read and send timeouts of an hour for the live connections, and uploads up to `ingress.bodySize` (110 MB). Other controllers: see the chart's README.
- **Security:** the pod runs as user 1000 without privileges on a read-only root file system; `/tmp`, `/app/.next/cache` and `/home/node` are `emptyDir`s.
- **Resources:** per pod 1 CPU and 2 GiB requested, 4 GiB at most (from [Sizing](/en/sizing)); no CPU limit, as throttling slows live editing.
- **Several pods:** rolling updates without an unavailable pod, a PodDisruptionBudget, pods spread over nodes, a cache per pod (`cluster.cacheSizeLimit`) that keeps itself below 70 % of its limit.
- **Optional:** a `ServiceMonitor` for the Prometheus Operator (`metrics.serviceMonitor.enabled`) and a `NetworkPolicy` (`networkPolicy.enabled`).

Not part of the chart: PostgreSQL, Redis or MinIO as subcharts, a HorizontalPodAutoscaler and separate worker pods – every pod serves requests and takes jobs from the queues, the leader the timed ones.

## Moving from one SQLite container (since 0.10.0)

You bring the data into PostgreSQL with the migration tool and the files into the bucket with the S3 mirror of the single container – both without losing anything; only the copy itself needs a short downtime.

::: steps

1. **Same version.** Bring the single pod to the Flowplan version the cluster will run (`helm upgrade` with the new chart or `image.tag`) and let it start once.
2. **Files into the bucket.** Turn on S3 in the single pod (`s3.bucket`, `s3.endpoint`, the keys in `s3.existingSecret` or `existingSecret`) and wait until every file is there: **Administration → Storage & backup**, or `objectStorage.pending` at `0` in [`/api/health`](/en/operations#health-check). Use the same bucket and prefix in the cluster later.
3. **Keep the keys.** Without `FLOWPLAN_ENCRYPTION_KEY` and push keys, the single container made them itself. Take them from its data folder into the Secret `flowplan-keys`: the content of `secrets.key` as `FLOWPLAN_SECRETS_KEY` (in addition to a `FLOWPLAN_ENCRYPTION_KEY`), `publicKey`/`privateKey` of `web-push-keys.json` as `WEB_PUSH_PUBLIC_KEY`/`WEB_PUSH_PRIVATE_KEY`:

   ```bash
   kubectl -n flowplan exec deploy/flowplan -- cat /app/data/secrets.key
   kubectl -n flowplan exec deploy/flowplan -- cat /app/data/web-push-keys.json
   ```

4. **Prepare the target:** an empty PostgreSQL database and a role that owns it ([Requirements](/en/postgresql#requirements)), and the Secret `flowplan-cluster` as above.
5. **Back up:** **Administration → Storage & backup → Whole instance → Download**.
6. **Stop Flowplan:** `kubectl -n flowplan scale deploy/flowplan --replicas=0`. The volume stays.
7. **Copy and verify** with a pod on the same volume (`flowplan-data` is the chart's PVC):

   ```yaml
   apiVersion: v1
   kind: Pod
   metadata:
     name: flowplan-migrate
   spec:
     restartPolicy: Never
     securityContext: { runAsUser: 1000, runAsGroup: 1000, fsGroup: 1000 }
     containers:
       - name: migrate
         image: ghcr.io/yannicschuller/flowplan:0.10.0
         command: ["node", "/app/scripts/db-migrate.mjs"]
         args: ["--from", "sqlite:/app/data/flowplan.sqlite", "--to", "$(DATABASE_URL)"]
         env:
           - name: DATABASE_URL
             valueFrom: { secretKeyRef: { name: flowplan-cluster, key: DATABASE_URL } }
         volumeMounts: [{ name: data, mountPath: /app/data }]
     volumes:
       - name: data
         persistentVolumeClaim: { claimName: flowplan-data }
   ```

   `kubectl -n flowplan apply -f migrate.yaml`, then `kubectl -n flowplan logs -f flowplan-migrate`. The last line must read `verified: every table has the same rows`. Delete the pod afterwards. Details and options of the tool: [Switching the database](/en/database-migration).
8. **Switch to cluster mode:** `helm upgrade flowplan deploy/helm/flowplan -n flowplan -f my-values.yaml` with `cluster.enabled: true` and the same bucket. The PVC is kept – delete it only once the cluster runs and its first backup exists.
9. **Check:** sign in, open a page with images, search, and under **Administration → Monitoring → System** see the pods, the leader and **PostgreSQL**.

:::

## Updates (since 0.10.0)

```bash
git pull
helm upgrade flowplan deploy/helm/flowplan -n flowplan -f my-values.yaml
```

With several pods, the schema Job runs first (it only adds, never removes), then the pods are replaced one by one: a new pod only gets requests once it is ready, an old one stops only after that. Browsers of a stopping pod reconnect to another one by themselves. If the Job fails, nothing is replaced – `kubectl -n flowplan logs job/flowplan-schema` says why. Going back with `helm rollback` works: an older version runs on the newer schema.

A pod that stops gets `terminationGracePeriodSeconds` (30): open requests and running jobs finish, uploads still under way reach the bucket, the lead goes to another pod. For long imports or exports, raise the value.

## Backups (since 0.10.0)

- **One pod:** as with Docker – scheduled backups in the administration, with S3 also Litestream ([Storage, S3 and backups](/en/storage-and-backups)). Back up the volume as well, if your platform can.
- **Several pods:** PostgreSQL holds all data: the scheduled backups in the administration are `pg_dump` archives in the bucket ([Backups with pg_dump](/en/postgresql#backups-with-pg-dump)); add your provider's backups or point-in-time recovery. The files are in the bucket – turn on versioning there or replicate it. Redis needs no backup.
- **Always:** keep the Secrets (above all `FLOWPLAN_ENCRYPTION_KEY` and the push keys) apart from the backups.
- **Restoring** a backup replaces every table: scale to **one pod** first (`cluster.replicas: 1`), restore, then scale up again.

## Scaling (since 0.10.0)

Set `cluster.replicas` and run `helm upgrade` – one pod per about 25 people active at the same time, plus a spare. More pods need more database connections: each pod opens up to `cluster.poolSize` plus three of its own. The PodDisruptionBudget (`cluster.podDisruptionBudget.minAvailable`) keeps pods running while nodes are drained; keep it below `replicas`.

What to watch: response times and CPU per pod under **Administration → Monitoring**, "waiting" connections under **System** (raise `cluster.poolSize` or add pods), and with Prometheus the metrics of each pod (`instance_id` label; the installation's figures only from the leader).

## Limits (since 0.10.0)

- **Restores and instance moves with one pod** – see Backups.
- **Imports** are staged in the pod's cache (`cluster.cacheSizeLimit`, 5 GiB) and in the bucket; for imports of several gigabytes raise the limit and `ingress.bodySize`.
- **Redis:** one address only, no Sentinel or Redis Cluster. While Redis is away, changes between pods arrive with the browser's next check (within about 20 seconds) instead of at once, and rate limits count per pod.
- **Meeting notes:** a recording is transcribed by the pod that received its end; if that pod stops meanwhile, start the processing again in the block.
- **Not tested on every platform:** the chart is checked against the Kubernetes API; report what your controller needs.

## Troubleshooting (since 0.10.0)

| Symptom | Cause and fix |
| --- | --- |
| Pods stay "not ready" | Ask the pod what is missing: `kubectl -n flowplan exec deploy/flowplan -- node -e "fetch('http://127.0.0.1:3000/api/health/ready',{headers:{authorization:'Bearer '+process.env.FLOWPLAN_METRICS_TOKEN}}).then(r=>r.text()).then(console.log)"` – `opening` names a failed data step or an older schema, `database` and `objectStorage` their errors. |
| The schema Job fails | `kubectl -n flowplan logs job/flowplan-schema`: usually the address, the password or missing rights of the role. |
| Pods do not start: "Cluster mode cannot start" | A Secret or key is missing; the log lists every problem at once. |
| Live changes come late or the connection drops every few seconds | The ingress buffers or cuts the stream: with ingress-nginx keep `ingress.nginxDefaults`; for others see the chart's README. |
| Uploads fail with 503 | The bucket does not answer – a file is only stored once it is in the bucket. Check the endpoint, keys and network policy. |
| Pods are evicted ("ephemeral-storage") | The cache or `/tmp` is too small for large files: raise `cluster.cacheSizeLimit` or `scratch.tmpSizeLimit`. |
| "remaining connection slots are reserved" | Too many connections: lower `cluster.poolSize` or raise `max_connections`. |
| "Shutdown: not finished in time" in the log | Raise `terminationGracePeriodSeconds`. |
| Two-factor sign-in or push works only sometimes | The pods have different keys: `cluster.keys.existingSecret` must be the same Secret for all. |
