Skip to content
flowplanDocumentationCommunityMarketplaceStart page

Scaling

High availability

Keep Flowplan running when a server fails or is updated: active/passive with a standby server, or active/active in cluster mode – with a complete example at Hetzner.

11 min read

On this page
  1. Two ways
  2. What both need
  3. Active/passive
  4. With PostgreSQL
  5. With SQLite and Litestream
  6. Active/active: cluster mode
  7. Updates without downtime
  8. Example: Hetzner Cloud
  9. Servers and network
  10. Object Storage
  11. PostgreSQL
  12. Redis
  13. Load balancer
  14. Setting up the app servers
  15. Backups
  16. Updates
  17. Next

Two ways

Active/passiveActive/active (cluster mode)
What runstwo servers, Flowplan runs on only one at a timetwo or more instances at once behind a load balancer
A server failsyou start Flowplan on the other one (minutes)the other instances keep working (seconds)
Updatesa short interruption while it restartsno interruption, one instance after the other
DatabaseSQLite with Litestream, or PostgreSQLPostgreSQL
Other servicesS3 (or a volume you can move)PostgreSQL, Redis and S3
More peopleno, one instance carries everyoneyes, about 25–30 active people per instance (Sizing)

For most teams active/passive is the better start: little new to run, and an outage costs minutes instead of hours. The cluster pays off when updates should not interrupt at all, or when one instance is no longer enough.

What both need

  • Files on more than one server: an S3 bucket (Storage, S3 and backups). Required in a cluster, the simplest way for active/passive; otherwise a volume you can attach to the other server (for SQLite never a network share such as NFS).
  • The same keys as variables on every server: FLOWPLAN_ENCRYPTION_KEY (or FLOWPLAN_SECRETS_KEY) as well as WEB_PUSH_PUBLIC_KEY and WEB_PUSH_PRIVATE_KEY. Without them each instance creates its own in its data folder, and two-factor sign-in, stored secrets and push subscriptions only match one server. How to create them or take them over from an existing data folder: Cluster mode.
  • One address: a load balancer or a floating IP in front, the same APP_URL everywhere and FLOWPLAN_TRUSTED_PROXIES matching the number of proxies.
  • The same version: the same image tag (FLOWPLAN_VERSION) on every server.

Active/passive

Two servers with the same compose.yaml and the same .env. Flowplan runs on the active one; on the passive one the container is stopped. In front sits a load balancer that checks /api/health/ready and sends traffic only to the running server – then nothing about the address changes when you switch. With a floating IP instead of a load balancer, you move the IP to the other server when you switch.

Warning

Without cluster mode only one instance may run at a time. Two instances run timed jobs twice (reminders, digests, backups, workflows), and live changes do not reach the other one. With SQLite and Litestream, two running instances spoil the copy in the bucket.

With PostgreSQL

Both servers point DATABASE_URL at the same database and S3_* at the same bucket. If a second instance starts by mistake, Flowplan notices through a lock in the database: it still starts, logs a loud warning every ten minutes, and Administration → Monitoring shows on every instance, until only one runs: Since 0.10.0

"A second Flowplan instance is running against the same database without cluster mode (instance, since time). Timed jobs such as reminders, digests, backups and workflows then run twice, and live changes do not reach the other instance. Switch cluster mode on for every instance (FLOWPLAN_CLUSTER=1 with Redis and S3) or stop one of the instances. During a rolling restart this is normal for a moment."

Switching over when the active server fails:

  1. Make sure the old server is really down: stop the container or power the server off in your provider's console. A server that is merely unreachable may otherwise keep running.
  2. On the passive server, docker compose up -d.
  3. Wait until /api/health/ready answers 200; the load balancer then sends traffic there by itself (with a floating IP: move the IP now).
  4. Check under Administration → Monitoring that no warning about a second instance shows.

Switching back works the same way. A stopped server may start again only once the other one is stopped.

With SQLite and Litestream

The database is a file on the active server; Litestream copies every change into the bucket continuously (under <prefix>/db/, every second), and the files are there too. The passive server has the same S3_* variables and keys and an empty data folder. The warning about a second instance does not apply here – each instance would have a file of its own.

  1. Stop the old server – two Litestream processes must never write into the same folder of the bucket at once.
  2. No old database may be on the passive server: Litestream restores only when flowplan.sqlite is missing. Empty a data folder from an earlier stint first (or take a new volume).
  3. docker compose up -d: Litestream fetches the database from the bucket, Flowplan the files on first access.
  4. Switch the address as above; afterwards empty the old server's data folder before it becomes the passive server.

At most the changes of the last second before the failure, which Litestream had not copied yet, are lost. The steps match way C of moving an instance. Without S3 a block volume works too, which you detach from the failed server and attach to the passive one (at Hetzner: Volumes, within one location only) – never a network share.

Switching automatically (e.g. with keepalived) only with a safe way to power off the old server: after a network glitch both run otherwise.

Active/active: cluster mode

In cluster mode all instances run at once. They share live editing, presence, rate limits and the parallel AI answers through Redis, all data is in PostgreSQL, all files in the bucket; any instance answers imports and print views. No sticky sessions are needed. When an instance fails, the load balancer takes it out after the next health check, and another one takes over the timed jobs within seconds.

Two single points remain: PostgreSQL (see Failover and replicas and below) and Redis. When Redis fails, everyone keeps working; only live changes between instances arrive late. Set FLOWPLAN_READY_REQUIRE=database,objectStorage for that – otherwise every instance reports "not ready" and the load balancer has no target left.

Updates without downtime

One instance after the other: stop it, start the new image, wait until it is ready, then the next one. On SIGTERM an instance answers 503 on /api/health/ready at once, finishes open requests and hands on live connections and the lead (stop_grace_period: 30s). The new version only extends the schema, so old instances keep working with it.

So that browsers do not mix versions, every image carries a deployment id (v<version>-<commit>, fixed at build time). The app requests its files with ?dpl=<id>, so caches on the way serve nothing stale, and when a page of the old version reaches a new instance, the browser reloads the whole page at the next navigation. Should an instance still lack a file of the other version, Flowplan's own build check heals the page. Building the image yourself, set the id with FLOWPLAN_DEPLOYMENT_ID (--build-arg FLOWPLAN_DEPLOYMENT_ID=…) – the same for every instance of one image – or switch it off with off. Since 0.10.0

Example: Hetzner Cloud

A cluster with two Flowplan servers behind the Hetzner Load Balancer, Redis and PostgreSQL in a private network, and files in Hetzner Object Storage. The files for it are in the repository under deploy/compose/hetzner (opens in a new tab). Names, plans and menus change – the Hetzner Console is what counts.

Internet ── HTTPS ──▶ Load Balancer (public IP, certificate)
                          │  private network 10.0.0.0/16
              ┌───────────┴───────────┐
          app-1 :3000             app-2 :3000      Flowplan (FLOWPLAN_CLUSTER=1)
              └─────┬───────────┬─────┘
               redis :6379   postgres :5432        private network only
              Object Storage (S3, over HTTPS)

Servers and network

All servers in the same location (e.g. Falkenstein, fsn1), with Ubuntu 24.04 and your SSH key.

ServerSizeNote
app-1, app-22 vCPU, 4 GB RAM each (the smallest type of that size, Arm or x86)each carries about 25–30 active people; more app servers for more people
redisthe smallest typestores nothing permanently
postgres2 vCPU, 4 GB RAM; from about 60 active people 4 vCPU, 8 GB, ideally with dedicated vCPUsor one of the options below
  • Network: under Networks, create a private network (e.g. 10.0.0.0/16) and attach every server and the load balancer. Flowplan, Redis and PostgreSQL listen on their private address only.
  • Placement group: put app-1 and app-2 into a placement group of type spread, so they do not run on the same physical host.
  • Firewall: one Cloud Firewall for every server: inbound only TCP 22 from your address. The app servers need no public ports – the load balancer reaches them through the private network. The Cloud Firewall only applies to the public interface, so protect Redis and PostgreSQL with a password as well and bind them to the private address (Docker bypasses ufw).

Object Storage

Under Object Storage, create a private bucket in the same location and create a key under S3 credentials (the secret is shown only once). Endpoint and region follow the location, e.g. S3_ENDPOINT=https://fsn1.your-objectstorage.com and S3_REGION=fsn1 – use the endpoint the console shows for your bucket. With FLOWPLAN_ENCRYPTION_KEY, files and backups are stored there encrypted.

PostgreSQL

OptionThe database server failsEffort
One PostgreSQL server with backups (postgres.yaml in the example)Flowplan is down until the server runs again or is rebuilt from the backuplow: one server, backups as below
Patroni on three small servers with etcd, HAProxy locally on every app serveranother server takes over within secondshigh: run and monitor Patroni, etcd and HAProxy
Managed service of a provider with a data centre nearbythe provider switches overlow, but the database sits outside the private network: only with TLS (sslmode=require) and access only from your servers' addresses

If Hetzner offers no managed PostgreSQL in your location, pick a provider nearby – every query travels that distance. How Flowplan handles a failover and which address belongs in DATABASE_URL is under Failover and replicas. With Patroni, DATABASE_URL points to 127.0.0.1:5000, where HAProxy offers only the primary.

Redis

A small server with redis.yaml from the example: a password, no persistence, on the private address only. A managed Redis with a fixed address works too (rediss:// with TLS); Redis Sentinel and Redis Cluster do not. With FLOWPLAN_READY_REQUIRE=database,objectStorage (as in .env.example), Flowplan stays ready while Redis is briefly away.

Load balancer

Under Load Balancers, create a load balancer in the same location (the smallest type is enough) and attach it to the private network:

SettingValue
Targetsapp-1 and app-2, through the private network
ServiceHTTPS, port 443 → destination port 3000; a certificate managed by Hetzner (Let's Encrypt; the DNS zone must be at Hetzner for that) or one you upload
HTTP → HTTPSswitch the redirect on
Health checkHTTP, port 3000, path /api/health/ready, every 5–10 seconds, timeout 5 seconds, status 200
Sticky sessionsoff – not needed
Proxy protocoloff
Algorithmround robin or least connections

The load balancer sets X-Forwarded-For, which is why .env has FLOWPLAN_TRUSTED_PROXIES=1. Live connections are server-sent events: the load balancer has to pass them through unbuffered, and Flowplan sends a heartbeat regularly so they are not closed for silence. After setting up, check that a second person's changes appear at once and that a large import (up to 100 MB) goes through. Finally point your domain's A and AAAA records at the load balancer's addresses.

Setting up the app servers

  1. Install Docker following the official guide (opens in a new tab).
  2. Get the example files: git clone https://github.com/yannicschuller/flowplan.git and copy what is in deploy/compose/hetzner to /opt/flowplan.
  3. Create the keys once – openssl rand -base64 32 for FLOWPLAN_ENCRYPTION_KEY, npx web-push generate-vapid-keys for the push keys – and keep a copy apart from the backups.
  4. cp .env.example .env and fill it in: domain, database, Redis, bucket, keys, setup token. .env is the same on both app servers except for PRIVATE_IP and FLOWPLAN_INSTANCE_ID.
  5. docker compose up -d on app-1 first, once it is ready on app-2; then create the first account at https://<domain>/register?setup=<token>.

Backups

  • Switch on the scheduled backup under Administration → Storage & backup: the leading instance writes a pg_dump archive into the bucket under <prefix>/backups/ daily (or hourly), encrypted with FLOWPLAN_ENCRYPTION_KEY (Backups with pg_dump).
  • A second copy outside the bucket, e.g. via SFTP to a Hetzner Storage Box – a fault in the bucket then does not hit both.
  • The database server additionally with Hetzner's backups (images of the disk); with Patroni, or to restore to any point in time, WAL archiving (e.g. pgBackRest) into a bucket of its own.
  • The app servers hold nothing that is not in the bucket or the database – a new server with the same .env replaces an old one.

Updates

  1. Read the release notes and set FLOWPLAN_VERSION in .env on both servers to the new version.
  2. On app-1: docker compose pull && docker compose up -d. The old instance answers 503, the load balancer takes it out, and the new one brings the schema up to date.
  3. Wait until the load balancer shows app-1 as healthy again (or curl http://<private IP>:3000/api/health/ready).
  4. The same on app-2.

To be on the safe side, remove the server as a target in the load balancer before step 2 and add it back afterwards. The deployment id makes sure browsers do not mix both versions in the meantime.

Next

Was this helpful?
YesThanks!
No – tell us what's missing

Found a mistake or something missing?Edit on GitHub