Cascade with Podman Quadlet

Two ways to run it. Both want Podman 5.0 or newer and a root install, so they start at boot with nobody logged in. Both follow ghcr.io/hotloop-io/hotloop-cascade:stable, run it read-only and nonroot with every capability dropped, and let systemd restart it. Updates and how they roll back are at the end.

One box, SQLite

hotloop-cascade.container and hotloop-cascade-data.volume. The database is a file in the volume, next to the key that seals stored credentials.

sudo cp hotloop-cascade.container hotloop-cascade-data.volume /etc/containers/systemd/
sudo install -D -m 600 /dev/null /etc/hotloop-cascade/cascade.env
sudo systemctl daemon-reload
sudo systemctl start hotloop-cascade

Open http://this-box:8080. The first person to open it makes the first account, so do that before you share the address.

PostgreSQL, a server and workers

postgres/. The database, the server and, if you want them, more workers. Postgres is the queue, so the workers take work from the same place. The database sits on a network of its own and publishes no port.

Three secrets first, made once. The commands are at the top of postgres/hotloop-cascade.container. The key one matters most: lose it and every stored credential is unreadable. Keep a copy somewhere that is not this box.

cd postgres
sudo cp hotloop-cascade.network hotloop-cascade-db.container hotloop-cascade-db.volume \
  hotloop-cascade-data.volume hotloop-cascade.container /etc/containers/systemd/
sudo install -D -m 600 /dev/null /etc/hotloop-cascade/cascade.env
sudo systemctl daemon-reload
sudo systemctl start hotloop-cascade

More workers: hotloop-cascade-worker.container. Workers share the data volume, so they have to be on the same box. Past one box, use the Helm chart.

Settings

/etc/hotloop-cascade/cascade.env, one NAME=value per line, and it has to exist even if it is empty (the install line makes it). The ones you will want:

Setting	What it does
`HOTLOOP_CASCADE_PUBLIC_URL`	The address people reach it at. Webhook and OAuth callback links are built from it. Until you set it, it is `http://localhost:8080`, which only works from the box itself.
`HOTLOOP_CASCADE_METRICS_TOKEN`	Requires `Authorization: Bearer` on `/metrics`.
`HOTLOOP_CASCADE_FILES_DIR`	Turns on the file steps, in this directory. Mount it into the container too.

The units set only the data directory, the ports and the key, and a value set in a unit beats one in the env file, so leave those alone. After a change: sudo systemctl restart hotloop-cascade.

Checking it

sudo podman healthcheck run hotloop-cascade
curl -s http://127.0.0.1:8081/readyz
sudo podman exec hotloop-cascade /usr/local/bin/hotloop-cascade workflows import - < workflow.yaml

Health and metrics are on port 8081, published on the loopback only. If something else already uses 8080 or 8081, change the left side of the PublishPort= lines.

Upgrading

The app units follow :stable and carry AutoUpdate=registry. Turn Podman’s timer on once per box:

sudo systemctl enable --now podman-auto-update.timer

When it fires, Podman pulls :stable, and if the digest changed it restarts the unit on the new image. The unit is not counted as started until the container passes its health check (Notify=healthy; the check is the binary asking its own /readyz, as a JSON array because the image has no shell). If the new image never gets there, Podman puts the previous one back under :stable and starts that. The wait is up to 15 minutes, because a new image migrates the database on its way up.

To hold updates until you say, leave the timer off and run sudo podman auto-update when you are ready. To let them happen only in a window, install update-window.conf (its header says how).

Two things a rollback can’t undo. Migrations only go forward, so take a backup before you turn updates on. A release that changes a table in a way the old one can’t read says so in its release notes. And the database: its image is pinned to an exact tag and carries no update label, on purpose. A new server image that can’t start is undone by putting the old image back, and a new database image can start, change the data directory, and only then turn out to be a mistake. Within PostgreSQL 18 that is an edit to the tag, after the release notes and a backup.

In the PostgreSQL flavour the server and each worker are separate units, so an update that can’t start rolls back that one unit and not the one beside it.

What was tried

The units were run for real on Podman 6.1.3, rootless under systemd, with a local registry standing in for ghcr.io and the shipped units unedited except for host ports. Standalone and PostgreSQL both: a first start that waits for a healthy container; an update to a new digest, which came back healthy with the data intact and, in the PostgreSQL flavour, the database container untouched; and an update to an image that exits at start, which podman auto-update reported as rolled back, with the previous digest running healthy again and the earlier workflow still running. As a root install on Podman 4.9.3 the earlier version of the units started and restarted under systemd; the update path was not run there, and Notify=healthy is not honored by 4.9 Quadlet, which is why the units also carry --sdnotify=healthy.

Then both flavours again as a root install, which is how the units say to run them, on Podman 5.8.7 under systemd in a throwaway privileged container, with the same local registry standing in for ghcr.io as a mirror and the units unedited. Standalone: the start waited for a healthy container, a killed container came back healthy, an update to a good image came up healthy on the new digest, an update to an image that exits at start was reported rolled back with the good one running again, and a workflow ran after each. PostgreSQL, with the server and one extra worker: systemctl start on the server held until the database and then the server were healthy, a killed server came back, an update to a good image moved the server and the worker and left the database container alone, an update to an image that exits at start came back rolled back with both on the good image again and the database still untouched, and a workflow ran before, between and after. One thing to know when you read that output: the server and the worker share one image, and when the first of them rolled back the other showed true instead, with a restart, and it ended up on the good image anyway. Which one rolls back first changed from run to run.

Not run: a real Raspberry Pi, and a Quadlet auto-update against ghcr.io itself.