The :stable auto-update

Every box tracks :stable and Podman updates it. That is the rule for all four products, and it is why no unit here pins a release number: a box pinned to 4.17.0 is exactly as old as the day somebody installed it, and nobody goes back to a plant floor to bump a tag. release.yml moves :stable on every tag, and .github/scripts/release_guard.sh refuses a release whose units don’t follow it.

Turn the timer on once per box:

sudo systemctl enable --now podman-auto-update.timer

What an update does

Podman pulls :stable when the timer fires, and if the digest changed it restarts the app’s unit on the new image. Only the app’s: the database container isn’t touched, so an update doesn’t bounce the historian. The unit has a healthcheck that needs no shell (the image has none): hotloop healthcheck asks the app’s own /readyz, which answers only while the database does. The unit stays in “activating” until it passes, so Podman can tell an update that came back from one that didn’t, and if the new image never comes up it puts the previous one back under :stable and starts that.

Starting the new image runs its migrations first, on the database that is already there, and a big database takes a while. The unit gives the app 15 minutes to answer before it calls the update failed, and failures during the first five minutes don’t count, because a migration is not a failure.

Two things were found by running it, not by reading about it, and the units carry the fix for each. A plain HealthCmd string is run through /bin/sh -c, and the image has no shell, so the check could never pass: it is a JSON array. And older Quadlet reads Notify=healthy as plain true and passes --sdnotify=conmon, which calls the app up the moment its container starts: an update that cannot start was reported as a success and nothing was rolled back. The units say --sdnotify=healthy themselves.

Why there is no pod

The app and the database used to share a pod. That cost the rollback, and it was found by running it, not by reading the manual. For a container in a pod, Podman’s auto-update restarts the pod’s unit, and the pod’s unit comes up fine whatever is wrong with the app inside it. A release that couldn’t start was reported as UPDATED true, the app crash-looped on it (twenty restarts in the first minutes), and :stable on the box was the broken image. It also restarted the database on every app update. With a network instead, the app is its own unit, Podman restarts that, and the unit stays “activating” until the app answers /readyz: the same release came back as UPDATED rolled back.

What a rollback can’t undo

A migration. When a release migrates the database and then fails its health check, Podman puts the previous release back, and it starts on the schema the failed one left. That is safe, and it is held safe three ways (MIGRATIONS.md has the detail):

• Migrations only add. A table, a column with a default, an index. Never a drop, a rename or a type change, so the previous release finds everything it knew where it was and never looks at what was added. A test (TestMigrationsAreAdditive) fails the build on one that doesn’t.
• The previous release is run on it before a release ships. test/prevrelease.sh migrates a database with the new code, then starts the last published image on it and runs login, devices, tags, a write through the gate and history. It is a step in the release checklist.
• An app refuses a schema it can’t be trusted on. The database records which release applied each migration. A rolled-back release one step behind starts, and says so in its log. One two or more releases behind does not start and says why, naming both versions. This is the case where a box fails an update, rolls back, fails another, and the database has now moved twice. The app exits at once with the message in journalctl -u hotloop, and the unit’s Restart=always keeps trying and keeps being refused. What fixes it is a release that knows the schema: the newest one, which is what :stable is (sudo podman auto-update, or pull it and restart). Or restore a backup taken by the release you are on (BACKUP.md).

So a rollback no longer has a release-notes caveat. What it still can’t undo is data: the failed release may have written rows the old one ignores. Take a backup before you turn updates on; it is the one thing between a bad release and a restore.

Why the database does not follow :stable

The app is labeled for auto-update. The database container is not, and its Image= is an exact tag. The reason is that the app’s updates can be undone and the database’s can’t.

• A PostgreSQL major version (16 to 17) is not a restart. PostgreSQL’s versioning policy says the data directory can’t be kept backward compatible across majors, and a dump and reload, or pg_upgrade, is required (postgresql.org/support/versioning). A tag that floated across majors would do it to a plant’s historian during an unrelated restart. Pinning the major is the minimum, and not an “endpoint pin” in the sense the rule above is about.
• A minor release (16.14 to 16.15) needs no dump and restore. The same page adds that some “might require additional steps so always read the release notes first”. An unattended box doesn’t read release notes.
• TimescaleDB (the Gateway and Edge) is the other half. The app runs ALTER EXTENSION timescaledb UPDATE itself at startup after a newer image, and an extension update is a one-way change to the database: putting the older image back afterward doesn’t undo it. So a floating Timescale tag would make the app’s reversible update drag an irreversible one along behind it.
• A new database image that starts and runs is not a healthy one. The healthcheck can tell a container that won’t start from one that will, and that is all it can tell. Whether a historian survived an engine upgrade is not something pg_isready answers.

So the database tag moves with a HotLoop release that was tested against it. release_guard.sh holds each hotloop-db.container equal to its chart’s database, so a Quadlet box and a chart install of the same product run the same one, and refuses an AutoUpdate= on it. The cost is that PostgreSQL’s own patch releases reach a box when a release ships them, or when you change the tag by hand. Within the same major that is an edit to hotloop-db.container, a daemon-reload and a restart of the pod, after reading the release notes and taking a backup. A major version is a restore into a new database, the way BACKUP.md describes moving between versions.

Holding updates until you say

The default is no hold: updates happen when the timer fires. A site that wants a person to decide has two ways, neither of which needs the app to be reachable.

• Hold. Leave podman-auto-update.timer off. Nothing updates. When you are ready, run sudo podman auto-update: that is the approval, with the same healthcheck and the same rollback. Podman updates every labeled container on the machine when it runs, not just these, and can’t be told to skip one, so a box that has to run the timer for other containers can’t also hold these.

• A window. update-window.conf is a drop-in for the timer that says when it may fire, such as Sunday at 03:00, instead of the default of daily at midnight:

  sudo mkdir -p /etc/systemd/system/podman-auto-update.timer.d
  sudo cp update-window.conf /etc/systemd/system/podman-auto-update.timer.d/
  sudo systemctl daemon-reload && sudo systemctl restart podman-auto-update.timer

The Edge Relay is different on one count: a Gateway can approve its update from the fleet screen, because a relay already listens for commands from the Gateway it forwards to. The three products have no such channel to reach them, and building one would put a “restart this box” button on a web UI for the sake of a feature that sudo podman auto-update already is. See EDGE-RELAY.md for the relay’s hold, and edge-relay-hold.conf.

What the Gateway sees

GET /api/status on any of the three says its version, and its imageDigest when the unit said (it does, before every start). An Edge or Gateway that has edge publish on is a Sparkplug node like a relay, and puts the same two in every birth, so a Gateway that has it in its fleet list shows the version and digest and logs the change when an auto-update moves it (FLEET.md). IoT has no upstream, so its digest is on its own /api/status and nowhere else.

Under Helm

A pod can’t replace itself, so there is no unattended update for the charts: helm upgrade is the update, and the chart’s image.tag defaults to its appVersion, which is the release.