Storage and operations
Troubleshooting in operation
What to check when a running instance restarts without an error, Litestream reports errors or two containers share one data folder – and what belongs in a bug report.
5 min read
On this page
More on this topic: Operations and troubleshooting · Monitoring and maintenance · Storage, S3 and backups · Coolify and reverse proxies
Problems during setup (sign-in, permissions, proxy) are in the troubleshooting table.
Restarts without an error
Flowplan restarts, the log shows no error, and the users only notice a short interruption. Start with Flowplan's own record, then ask Docker and the kernel.
In Flowplan
Administration → Monitoring → Restarts Since 0.10.0 lists every restart of the last 30 days with its reason (clean, crash, likely out of memory, killed without trace), the memory at the last heartbeat and, for an unexpected one, the last 200 lines of the server and Litestream log before the end, with Download as .txt. What the reasons mean and where their limits are is under Restarts.
With Docker
# Why the container ended last time, and how often it restarted
docker inspect --format 'OOMKilled={{.State.OOMKilled}} ExitCode={{.State.ExitCode}} RestartCount={{.RestartCount}} FinishedAt={{.State.FinishedAt}}' <container>
# The last health checks with their output
docker inspect --format '{{json .State.Health}}' <container>
# What happened to the container in the last 24 hours (die, oom, kill, health_status)
docker events --since 24h --until "$(date +%s)" --filter container=<container>
# Memory and CPU right now, against the limit
docker stats --no-stream <container>
# Whether the kernel ended a process for lack of memory
dmesg -T | grep -i oom| You see | Means |
|---|---|
OOMKilled=true, ExitCode=137 | The container hit its own memory limit and the kernel ended it. Raise the limit or find what grows (below, Litestream). |
ExitCode=137 without OOM | SIGKILL from outside: docker kill, kill -9, or a stop that took longer than the grace period. |
ExitCode=143 or 0 after docker stop/an update | A normal stop (SIGTERM). |
Health.Status unhealthy, FailingStreak rising | /api/health failed several times in a row; Health.Log holds the answers. Docker alone does not restart an unhealthy container, an orchestrator may. |
RestartCount rising | The restart policy brings the container back; docker events shows when. |
Out of memory: Killed process … (litestream) or (node) in dmesg | The kernel ended the process. Memory cgroup out of memory means the container's limit, a line without cgroup (constraint=CONSTRAINT_NONE) means the whole server was out of memory. |
The host runs out of memory as a whole. Without a memory limit on the container, the kernel's global OOM killer picks the largest process on the server – it can be Litestream, the container's first process. OOMKilled stays false, because the container's own limit was not the cause; and once Docker has restarted the container, docker inspect shows the new, running container (ExitCode=0), not how the old one ended – docker events or the restart entry tell that. Flowplan lists the restart as "killed without trace" or, if the server had 5 % or less of its memory available at the last heartbeat, as "likely out of memory". Only dmesg tells for sure. Give the container a memory limit, or leave the server enough room for Flowplan, Litestream and everything else on it; see Sizing.
Litestream
Litestream replicates the SQLite database to the bucket (only with S3 and without DATABASE_URL). Its lines in the log carry level=…; the saved copy is /app/data/logs/litestream.log.
| Log line | Means |
|---|---|
sync error … rename …ltx.tmp …: no such file or directory | Two Litestream processes work on the same database – two containers on one volume (below). |
compaction failed … level=1 … file checksum mismatch, every 30 seconds | One of Litestream's change files (LTX) is corrupt, locally or in the bucket – usually left behind by such an overlap. The database itself is fine. |
While compaction is stuck, Litestream no longer cleans up its change files: they pile up locally and in the bucket, and Litestream's memory keeps growing – up to the point where the kernel ends it (see the host running out of memory above). A restore from the bucket through the broken file fails too.
Flowplan repairs this by itself. Since 0.10.0 Every 15 minutes it checks the bucket; a stuck replica gives a warning in Administration → Monitoring, a message to the administrators and, after a scheduled backup, a repair on the next restart (Restart now does it right away). FLOWPLAN_LITESTREAM_AUTOHEAL=off keeps only the warning.
The repair is a marker file /app/data/litestream-reset, read on the next start before Litestream replicates:
| Content | Does |
|---|---|
empty (touch) | Resets only the local state (.flowplan.sqlite-litestream); Litestream carries on from the bucket. |
remote as its first line Since 0.10.0 | Also deletes <prefix>/db/ in the bucket; Litestream starts afresh with a full snapshot. Written by the self-healing and by Rebuild the replica now. |
Never the database, uploads or backups are touched. The steps, what the log shows and what to do without a running container are under Repair the Litestream state.
Two containers on one data folder
With SQLite, only one container may run on a data folder. Every container holds a lock on /app/data/.flowplan.lock Since 0.10.0; a second one that finds it taken serves anyway, starts no Litestream and writes a block to the log:
==========================================================================
Flowplan: ANOTHER CONTAINER IS USING THE DATA FOLDER /app/data.
It holds /app/data/.flowplan.lock. Two Flowplan containers on one SQLite database must not overlap:
…
==========================================================================Administration → Monitoring shows the warning "A second Flowplan container (host) has been using the same data folder since time …" on both containers. A few seconds during an update point to a rollout with overlap; a warning that stays points to a second resource, replica or docker run on the same volume. Find the other container (docker ps, the host name in the warning) and stop it. How to roll out without overlap, also in Coolify, is under Roll out SQLite without overlap.
What to put in a bug report
Open an issue (opens in a new tab) with:
- the version line, the first line Flowplan writes to the log, e.g.
Flowplan 0.10.0 (release, build abc1234, built …, Node.js …)(orversionfrom/api/healthwith the token); - for a restart, the entry's .txt from Administration → Monitoring → Restarts;
- the output of the
docker inspectlines above, and thedmesglines around the time if there are any; - for Litestream, the first error lines in the log and whether S3 and
S3_PREFIXare set (not their values); - how the instance runs: Docker, Compose, Coolify, Kubernetes; SQLite or PostgreSQL; memory limit.
Never post secrets: no keys, tokens, passwords, .env files or a full docker inspect without --format – it lists every environment variable with its value. Flowplan redacts its saved log files (what is redacted), but read every line before you post it; docker logs is not redacted.