Fleet management

Added in 4.7.0, building on 4.6.0’s Edge Relay edition. internal/fleet lets the Gateway track registered edge relays, whether they’re alive, and, for an authorized operator, ask one to rebirth, restart, or adopt a newly pushed config. Off by default (HOTLOOP_FLEET_ENABLED): a deployment with no edge relays has no broker to point this at.

What it is not

Not the same thing as the MQTT driver’s Sparkplug host mode. internal/driver/mqtt.go polls tag values out of whatever Sparkplug nodes a configured Device connects it to, one Device row is one broker connection, built for tag consumption. Fleet management owns its own, separate MQTT connection and answers a different question: is a registered edge relay alive, not what value is on one of its tags. Bolting this onto the tag-polling driver would have meant either forcing an operator to configure a compatible Device just to get fleet visibility, or reaching into that package’s private per-node bookkeeping, which was never built to be shared.

Not automatic. An edge relay showing up on the broker does not make it a fleet member. Membership is declared, a row in fleet_members, created through POST /api/fleet, the same distinction this app already draws between a tag a driver discovers and a tag someone configures. An NBIRTH from a group/node nobody registered is logged at debug and otherwise ignored.

How it fits together

Two data paths, both from internal/fleet.Manager’s own MQTT connection:

• Presence, from NBIRTH/NDEATH. Outbound-only from the relay’s side, it dials the broker, same as it always does to publish readings, so this works through NAT with no inbound connection to the relay required.
• Detail, from an optional HTTP poll of the relay’s own /status (see docs/EDGE-RELAY.md), only for members with a statusUrl registered. Fanned out concurrently with a short per-member timeout, the same shape internal/api/sites.go already uses for polling remote sites, an unreachable relay is marked reachable: false, never reported as a healthy zero.

Commands travel the other way over Sparkplug NCMD, extending what a relay already handles:

• Rebirth (Node Control/Rebirth), already handled by every edge relay’s internal/edgepublish.Publisher since 4.5.0. A benign, idempotent nudge.
• Restart (Node Control/Reboot, the Sparkplug spec’s own name for this, not one this app invented), the relay publishes its own graceful NDEATH and exits; its supervisor (a Quadlet unit’s Restart=always, or a Deployment’s default restartPolicy: Always) brings it back up with whatever is on disk at that moment. No new shutdown path, this is the exact same clean-exit route the relay’s own SIGTERM handling already takes.

Because commands ride the same MQTT connection the relay already dials out on, sending one needs no inbound connection to the relay either.

A relay that shows up online can be commanded, full stop. Every relay subscribes to its NCMD topic, and waits for the broker to confirm it, before it publishes a birth, on every connection, so “Online” is never ahead of “reachable.” The manager holds itself to the same rule on its side: it does not report itself started until its NBIRTH/NDEATH subscriptions are confirmed, so a relay that births the instant the Gateway comes up is not missed. If the broker’s ACL refuses either side’s subscription, that side refuses to start and says so, instead of running half-deaf.

Config push

Pushing what a relay polls doesn’t ride NCMD, a Sparkplug metric isn’t sized for a whole device file, it goes to a retained, plain MQTT topic per relay: hotloop/edge-relay-config/<group>/<node>. Retained delivery is what makes this work with no separate “catch me up” request: whatever was last published there is handed to any client that subscribes, immediately, including a relay reconnecting after being offline, or starting up for the very first time.

POST /api/fleet/{id}/config (fleet.config, admin-only, the same narrowing as fleet.restart) parses and validates the device file before publishing anything, using the identical rules internal/edgerelay applies on the relay’s own side, so a malformed push is a rejected API call, not a relay quietly stuck with something broken. The relay validates again on receipt, this app cannot assume nothing except the Gateway can ever publish to that topic, only that it should refuse to publish something broken itself.

A device file can carry credentials, and a push publishes it. A device’s config block holds whatever its driver logs in with, such as an OPC UA username and password (4.15.0), and the file is published as it is, retained. Anyone who can subscribe to hotloop/edge-relay-config/# can read it, so the broker needs TLS and an ACL on that topic before a file with credentials is pushed. The Gateway’s API hides device credentials from every role as of 4.15.1, but a pushed file is not behind the API and is not changed by that. See EDGE-RELAY.md and AUTH.md.

A push applies live, and it never restarts anything. Up to 4.17 a push only staged the file, and making it count took a second action, a POST /api/fleet/{id}/restart, which dropped the relay’s broker session and every device session it held to change one tag. Now:

• A relay that’s running validates the file, saves it, and applies it in place: new devices connect, removed ones disconnect, changed ones reconnect, and every device that didn’t change keeps its session. The Sparkplug session and the forward queue stay up the whole time. The node rebirths for its new metric set and nulls the metrics it dropped first, so a host doesn’t keep showing a removed device’s last good value. EDGE-RELAY.md has exactly what goes out on the wire and why.
• A file the relay refuses (it won’t parse, it won’t validate, it can’t be saved) changes nothing on the relay, and the relay’s /status says so under config.rejected. GET /api/fleet shows that for every member with a statusUrl, next to the digest of the file it’s running, so “I pushed it” and “it’s running it” are two things you can check, not one you assume. The Gateway validates before it publishes too, so in practice this is a relay on a newer or older release disagreeing about the rules, or something other than the Gateway publishing to the topic.
• A relay that’s offline gets the retained push the moment it connects. At boot it waits a couple of seconds for it before loading its local file (see cmd/edge-relay/main.go’s applyPushedConfig), so it comes up on the pushed list instead of starting on the old one and switching.

A pushed file lands on the relay’s data volume, not over its local device file, and wins over the local one until the push is cleared. The relay’s EDGE-RELAY.md has the precedence rules and the 4.16.0 bug that made them necessary.

Taking a push back is DELETE /api/fleet/{id}/config (fleet.config, same as pushing). It publishes an empty retained message to the member’s config topic, which is how MQTT deletes a retained value, so no later boot applies the push. A relay connected at the time drops its pushed file and goes back to its local one live, the same way a push applies, unless its local file doesn’t load, in which case it keeps the pushed one and says so. A relay that was offline never hears the clear (an empty message isn’t retained), so send it again once the relay is back.

What each relay runs, and updates

Every relay puts its version and the digest of its image in each birth, and so does an Edge or a Gateway with edge publish on, so a box registered here as a member shows the same (string metrics named Node Info/Version and Node Info/ImageDigest, with no alias, so a Sparkplug host that doesn’t know them ignores them). GET /api/fleet shows them as release on each member: version, imageDigest and since, when the Gateway first heard that build. A relay from before this reports none, and the Gateway says nothing about it rather than invent an “unknown build”.

When a member’s build changes, the Gateway writes it to the system log as a config event on that member, with the old and new version and digest in its detail ({"kind":"fleet.release",...}). The first build it ever hears is logged too, as the baseline the next one is measured against. It survives a Gateway restart, because the last record is read back from the log: a restart never invents an update. This is the audit trail of every update a relay takes, by whatever route (Podman’s timer, an approval, helm upgrade, somebody on the box), because the Gateway doesn’t cause an update and can only report it. A moving tag is how :stable updates, so a changed digest under the same version is logged too, as “a different image of the same version”.

POST /api/fleet/{id}/update (fleet.update, admin-only) approves the update a relay holds for. It only does anything to a relay set to hold updates; any other relay logs that it ignored it. The request names no image. What this does and doesn’t do for each install type is in EDGE-RELAY.md.

Command authorization

Two layers, deliberately not one:

• At the API, an operator needs fleet.rebirth to request a rebirth; restarting, approving an update or pushing a config each need fleet.restart/fleet.update/fleet.config, granted only to admin. A restart is more disruptive to a running poll cycle than a rebirth nudge, and pushing what a relay polls is treated the same way, the same narrowing alarms.ack and automations.trigger already model for other “do it now” actions.
• At the broker, this app has no way to enforce anything, NCMD publishing and the config topic must both be restricted by the broker’s own ACLs to the Gateway’s own MQTT credentials. Without that, anything that can reach the broker and knows a relay’s group/node can publish the same command or config this feature does. Configure this at deployment time; it is not optional just because it lives outside this app’s own code.

API

Route	Permission	Details
GET /api/fleet	fleet.read	Every registered member, with live presence and, where configured, a polled /status
POST /api/fleet	fleet.write	Create or update a member (a body with an existing id updates it)
DELETE /api/fleet/{id}	fleet.write	Remove a registered member
POST /api/fleet/{id}/rebirth	fleet.rebirth	Publish Node Control/Rebirth
POST /api/fleet/{id}/restart	fleet.restart	Publish Node Control/Reboot
POST /api/fleet/{id}/update	fleet.update	Approve the image update a relay is holding for (does nothing to a relay that isn’t)
POST /api/fleet/{id}/config	fleet.config	Push a new device file (raw JSON body, validated before publishing); a running relay applies it live
DELETE /api/fleet/{id}/config	fleet.config	Clear the push, so the relay goes back to its local device file, live if it’s connected

Configuration

Variable	Default	Details
HOTLOOP_FLEET_ENABLED	false	The feature is off until this is set
HOTLOOP_FLEET_BROKER	n/a	Required when enabled. Bare host, host:port, or a full URL
HOTLOOP_FLEET_USER / _PASSWORD	n/a	Broker credentials
HOTLOOP_FLEET_TLS	false
HOTLOOP_FLEET_TLS_SKIP_VERIFY	false	Per-deployment opt-in; never set this by default

What has and has not been verified

Tested against a real Mosquitto broker and a real internal/edgepublish.Publisher standing in for an edge relay, not a hand-rolled stand-in, the actual code every edge relay runs: presence tracking on a real NBIRTH/NDEATH, a real rebirth round trip (the sent NCMD observed by the relay’s own already-tested command handler, and the resulting second birth observed back by this package’s own listener), a real restart round trip against the RestartRequested channel, and /status polling against both a real httptest server and a deliberately unreachable address. Config push is tested the same way for the push itself (a real retained publish, received by a real Publisher.ConfigReceived), and separately end to end through the actual compiled cmd/edge-relay binary, run as a real subprocess against a real broker: a config retained before the process even starts is what it boots with, proving applyPushedConfig’s boot-ordering claim rather than just its two pieces in isolation.

Not yet run against a certified Sparkplug host application publishing NCMD to this app, nor against a real broker’s ACL configuration restricting who may publish it, the authorization boundary described above is a deployment requirement this app cannot verify from the inside, only document.