HotLoop core 4.18.10 release notes

Written against main

This page follows main at c118b2f, and it differs from the copy in HotLoop core 4.18.10, the latest release. Parts of it may be newer than what you have installed. The release notes say what moved.

The release where HotLoop stops being one image. IoT, Edge, the Gateway and the Edge Relay all ship from this tag, in lockstep, each with its own image and its own chart. On top of that: devices that announce themselves over MQTT, history written out to the plant’s own database, a database you pick (SQLite, Postgres or TimescaleDB), an Edge that keeps working with the WAN dead, and boxes that follow :stable and roll a bad update back on their own. Coming from 4.17.0 the Gateway is a plain helm upgrade. Read “Upgrading from 4.17.0” below before you touch a running plant, because a couple of defaults moved.

There is no 4.18.0, 4.18.1, 4.18.2, 4.18.3, 4.18.4, 4.18.5, 4.18.6, 4.18.7 or 4.18.9 to install, and 4.18.8 is not one to run. I tagged all ten. Nine of them published nothing, no image, no chart and no release. 4.18.8 passed every test and then I cancelled its run by mistake while it was publishing, so it left images on ghcr.io with no chart and no release (more on that below). The first three died in the Test job. CI ran Docker on a small shared runner and my lab runs Podman on a big box, and the integration services’ script had three things only the lab forgave. The network-discovery image was built without naming its Containerfile. InfluxDB 3’s token file went in readable by root alone, which Docker leaves it as and Podman quietly hands to the container’s user. SQL Server died at boot when the runner was busy starting everything else, so it starts on its own first now and gets tried again if it dies. The port 465 test had no relay to run against, so it skipped, and a skip fails the release. Two reloads of the runtime at once could race on a worker, and reloads take turns now. The fourth, 4.18.3, got past all of that, started every service, and ran the runner out of disk five minutes into the tests: a hosted runner starts a job with 14 GB free, the service images are 10.5 GB of it, and Go took the rest. The job clears the toolchains it never uses before it pulls anything now, and prints where the disk went whether it passes or not. The fifth, 4.18.4, had the room and died in the Test job anyway, on two tests that only fail on a starved box: a test broker that counted a client which had already hung up, and a browser test that builds a dashboard. So the rehearsal of the release stopped being a quiet box. The Test job runs on two CPUs, the runner’s RAM and the runner’s disk, with something else burning CPU beside it, and a release isn’t tagged until that has passed. What it turned up is fixed here: a discovered device’s value could be stamped older than its own “nothing yet” reading and the cache dropped it, which is a bug in the product and not in the test, the dashboard stops redrawing a gauge that hasn’t changed, and the test Chrome runs without its GPU, because software GL was eating both CPUs. And the images and charts are built with Podman and Buildah now, where the release workflow and the test services still talked to a Docker engine. Dockerfile and Dockerfile.edge-relay are Containerfile and Containerfile.edge-relay. The sixth, 4.18.5, passed all of that here and died on the runner before a single test ran: the suite mounts Go’s module cache from the runner’s home directory, a fresh runner doesn’t have that directory yet, and Podman won’t mount a folder that isn’t there where Docker quietly made one. The suite makes it first now. The seventh, 4.18.6, ran every test and lost one: ntfy refused an alarm notification that carried the right token. That’s a race inside ntfy itself. It files a request under its sender’s address and briefly forgets who the sender is while it checks a password, so a token publish that lands beside a password request from the same address reads as nobody, and the lockdown turns it away. HotLoop sent the right thing. The test ntfy gives its user its own lane now, and if your phones log in to ntfy with a password from the same address HotLoop publishes from, give that user a tier too. The eighth, 4.18.7, lost the one test that had been flaky all week, and it was a real bug. The broker built into HotLoop checked a client’s SUBSCRIBE packet id against the messages it was sending that same client. MQTT numbers those two apart and both start at 1, so a listener that subscribed while it was being sent QoS 1 traffic got refused, the client library called the refusal a grant, and the device behind that subscription was never discovered. The check is gone, a raw packet test proves it, and a broker that refuses a subscription is logged now instead of shrugged off. The ninth, 4.18.8, passed every test. I cancelled its run by hand, because the same broker had the mirror image of that bug and I wanted both fixes in one release, and I thought it was still testing. It wasn’t. It was publishing. By then the Gateway, Edge Relay, IoT and Edge images were already pushed as 4.18.8, and the Gateway, Edge Relay and IoT ones had moved :stable and :latest with them. No chart and no release followed, and 4.18.10 replaces it, so a box that follows :stable may have run 4.18.8 until it updated. The mirror image was this: when a client published with the same packet id as a QoS 1 or 2 message the broker still owed it, the broker threw its own message away and never resent it after a reconnect. The broker keeps the two directions in separate tables now, with a test for each QoS, and that fix is in this release. The tenth, 4.18.9, lost one browser test to a real bug: a dashboard still loading wrote its address over the one the person had just moved to, so a reload put them back on the dashboard they had left. A dashboard only takes the address now if nobody has moved it since it started loading. 4.18.10 is everything 4.18.0 was going to be, plus what changed on main since 4.18.3 was tagged.

The full notes are in the README: What changed in 4.18.10.

In short

• Built with Podman. The images and the charts come out of Podman and Buildah, and Dockerfile and Dockerfile.edge-relay are Containerfile and Containerfile.edge-relay. Same four images, same tags, same architectures.
• Since 4.18.4 was tagged. A discovered device’s value could be lost behind its own “nothing yet” reading and is not now. A UPnP device whose description failed to read is read again in seconds, not after 15 minutes. OK on the card editor works on the first click, Save in the entity drawer stays put, the dashboard stops redrawing what hasn’t changed, and a counter two processes hammer no longer refuses the one that keeps losing.
• The license changed. All four products are under HotLoop’s Business Source License 1.1 now. Home and lab use is free, so is use by schools, nonprofits and government, and business use goes through EmberNET. Each version turns into GPL-3.0 or later four years after it first goes out. It is not open source until then, and I’m not going to pretend it is. Read LICENSE before you put it on a plant.
• Since 4.18.3 was tagged. UniFi on a wall board and a phone: the network map zooms and pinches, WAN latency and port traffic get traces, and one button makes the usual alarms. An agent can ask over MCP for the network’s map (get_network_topology) and what the console just said (get_network_events). The logbook shows UniFi events under the switch they are about. One MQTT device could get two clients with the same id, and the broker kicked each in turn; that is fixed. Alarm messages print a value the way the value card does, the alarms card can be pointed at devices or equipment, and a trend of a tag that holds still is a line across the window.
• Four products, four images, four charts, one tag. hotloop-iot and hotloop-edge are new on ghcr.io and in the chart repository. The Gateway keeps ghcr.io/hotloop-io/hotloop and the chart hotloop, so nobody running it changes anything. A product is a build, not a setting: the OPC UA server isn’t switched off in the IoT image, it isn’t in it. See docs/PRODUCTS.md.
• MQTT discovery. Point ESPHome, Zigbee2MQTT, Tasmota or a Shelly at the broker and it shows up, reading. It can’t command anything until an admin accepts it and arms each point by hand. Network discovery (mDNS, SSDP, DHCP) listens and sends nothing unless you list a network to probe.
• Historian targets. Every reading HotLoop keeps also goes to the plant’s own PostgreSQL, TimescaleDB, SQL Server, InfluxDB or MySQL/MariaDB, the same rows with the same quality.
• The database is a setting. SQLite, PostgreSQL or TimescaleDB per product, from the same binary. A Pi doesn’t need a 300 MB database image.
• Edge keeps the plant running with the WAN dead, and its store and forward queue survives the ugly cases: a boot during an outage, a link that dies without a reset, a replay that has to outrun the plant.
• Every box follows :stable and updates itself, and a release that can’t start is rolled back. Migrations only add, and an app refuses to start on a schema two releases ahead of it, so that rollback is always safe.
• The Edge Relay takes a pushed config live, with no restart and no lost Sparkplug session, and discovers devices on a broker like the Gateway does.
• Every image carries every third-party license, and a GPL module stops the build.
• The sun. Sunrise, sunset and the sun’s height, worked out on the box from a latitude and a longitude.

API changes

• New routes: /api/discovery, /api/network-discovery, /api/mqtt/brokers, /api/broker/logins, /api/historian/targets, GET /api/entities/{entity_id}/image and POST /api/fleet/{id}/update. All in docs/API.md.
• GET /api/status says product and features, and the image digest. A route for a feature the product you’re talking to doesn’t have answers 404 with the product that does, instead of pretending. Every route is still there in every product.
• New permission fleet.update, admin only. The operator role didn’t change.

Upgrading from 4.17.0

• helm upgrade <release> hotloop/hotloop --version 4.18.10 --reset-then-reuse-values. I rendered the 4.17.0 and 4.18.10 Gateway charts side by side: the same eight objects, the same selectors, the same volume claims. Plain --reuse-values renders too this time, but it replays 4.17.0’s values and drops every new default, so use the reset form or your values file.
• Migrations 0016 to 0024 run on start (MQTT discovery, broker logins, MQTT sources, historian targets, network finds, two more for what the MQTT sources speak, the table dashboards are saved in, and a no_data column on tags that is empty for every tag that already exists). All additive. pg_dump first, like always.
• edgePublish.queueMaxBytes defaults to 64 MiB now, not unbounded. Past it the oldest readings are dropped and counted. An unbounded queue filled the volume in a long outage, which is worse, but it is a changed default: size it from your own reading rate (about 180 bytes a reading) or set it to 0 to keep the old behavior. An upgrade with --reuse-values keeps whatever you had.
• Broker logins. Logins you make in Settings, MQTT carry topic rules from a profile. Logins from HOTLOOP_BROKER_USERS are untouched and can do anything, exactly as before. A broker with no HOTLOOP_BROKER_USERS starts now, and only HotLoop and the logins made in it get in.
• Going back to 4.17.0 works: the schema only grew, and 4.17.0 runs on it. test/prevrelease.sh proves that against the published 4.17.0 image before every release. From here on the app also counts how far ahead its database is and refuses by name at two releases, so a rollback is never a surprise.
• Edge Relay: upgrade it with --reset-then-reuse-values too. Quadlet users take the new edge-relay.container, which follows :stable instead of pinning 4.17.0, and the healthcheck that makes a bad update roll back.
• Quadlet units for IoT, Edge and the Gateway are new. 4.17.0 only had the relay’s. They are under deploy/quadlet/, one directory per product.

What isn’t proven

• Edge and IoT as live Quadlet units under systemd, except IoT’s SQLite unit, which was run. The rest are generated from the Gateway’s, checked against them, and not started. The Gateway’s units were run for real, rootless under systemd, including an update that migrates and then can’t start and gets rolled back. A real Pi under systemd as root: not yet.
• Moving a site up a product as a helm upgrade on a real cluster, and going back down one. The Quadlet move, IoT to Edge to Gateway on one database volume, has been run.
• Discovery against real devices is proven against recordings of real installs, not against devices on a bench here. ESPHome’s mqtt: component doesn’t build for a host, so there is no live ESPHome in CI.
• The 32-bit arm images on real hardware. A Pi 3 under load: not yet.