Edge publish and store-and-forward

Added in 4.5.0. Turns this gateway into a Sparkplug B edge node, publishing everything it reads to a central Gateway over MQTT, the opposite direction from the existing MQTT driver, which consumes Sparkplug as a primary host. internal/edgepublish is the implementation; this is what it means to run it.

Turning this on runs it inside the same Gateway process, still connected to TimescaleDB, useful when a site wants the full Gateway’s UI and history locally as well as an uplink to a central one. For hardware that should not carry a database at all, see docs/EDGE-RELAY.md: cmd/edge-relay, added in 4.6.0, is this exact publisher and forward queue with no Gateway process around them, a separate, smaller binary built for a Pi or an industrial PC. Fleet management for many edge relays landed in 4.7.0 (tracking, from their births and deaths) and 4.8.0 (config push); see docs/FLEET.md.

Turning it on

export HOTLOOP_EDGE_PUBLISH_ENABLED=true
export HOTLOOP_EDGE_PUBLISH_BROKER=mqtt.example.local:1883
export HOTLOOP_EDGE_PUBLISH_GROUP=Plant1
export HOTLOOP_EDGE_PUBLISH_NODE=press-line-3   # defaults to the pod hostname
export HOTLOOP_EDGE_PUBLISH_QUEUE_PATH=/data/forward.db

Every reading that reaches the tag bus (the same stream the live UI and the alarm evaluator read) is published as Sparkplug B NDATA. While the broker is reachable and confirming, that happens straight from memory and the disk is not touched. When the broker stops confirming, the readings go to the forward queue on disk and come out of it, in order, when it comes back.

What it actually does

Subscribed before it is born. On every connection, first and every reconnect, the node subscribes to its NCMD topic (and its config push topic) and waits for the broker to acknowledge both before it publishes anything. Sparkplug B 3.0 requires this (“Prior to sending an NBIRTH message, the MQTT client associated with the Edge Node must subscribe to receive NCMD messages”), and for a good reason: a host is allowed to answer a birth with an immediate rebirth or reboot request, and a node that births first can miss it. Start() does not return until the subscriptions are in place. If the broker refuses one (an ACL that does not let this node read its own NCMD topic), Start() fails with an error that says so; a node nobody can command should not come up pretending otherwise. A reconnect that cannot resubscribe does not birth on that session; readings keep going to the forward queue.

A broker that isn’t there doesn’t stop the boot. Refused is a config problem and fails Start(). Unreachable is an outage: Start() logs it, returns, and the node keeps trying in the background (one second, doubling to a minute) with every reading going to the forward queue. It used to fail on that too, and the Gateway and the Edge exit when edge publish fails to start, so a box rebooted during a WAN outage crash-looped with no screens, no alarms, no automations and nothing buffered until the WAN came back.

NBIRTH once per connection. Every known tag, announced by name with a stable alias, and its current value. A birth happens on first connect, on every reconnect (a Sparkplug session starts over, its sequence numbering has no meaning across a reconnect), when a tag nobody has seen before shows up in a batch, and on request, see below.

NDATA for everything after, by alias only, no name. seq starts at 0 with the birth and increments (mod 256) with every message until the next one.

NDEATH two ways: gracefully on Stop(), and via the MQTT connection’s own last-will if the process dies without the chance to. Both carry the same bdSeq value, matched to the birth they end. A host tells an old death certificate, left over from a previous session, from the current one by that number.

bdSeq counts sessions, and the count survives a restart. Every MQTT CONNECT gets the next number: 0 on a queue’s first session, one more each time, round to 0 after 255 (Sparkplug B 3.0, Operational Behavior, Edge Node Session Establishment: tck-id-message-flow-edge-node-birth-publish-will-message-payload-bdSeq), and the NBIRTH on that connection carries the same number as the will it registered (tck-id-message-flow-edge-node-birth-publish-nbirth-payload-bdSeq). It lives in the forward queue’s file, so a restarted node carries on counting instead of starting over at 1 and handing the host a will it had already seen, which a host is entitled to ignore as stale. A reconnect is a new session, so it gets a new number too; the old code kept one for the life of the process because paho reuses the will it was created with, and the node now builds a fresh client, and so a fresh will, for every connection attempt. The metric is INT64, as the spec asks (it went out as UInt64 before; the value rides in the same field, so a host reading it by value doesn’t notice).

NCMD → Node Control/Rebirth. A host asking for a rebirth gets a fresh NBIRTH, same bdSeq. A rebirth is not a new session, so it does not get a new one (tck-id-operational-behavior-data-commands-rebirth-action-3).

When the uplink is down

The disk is for outages, not for every poll. While the uplink is up and the broker is confirming, readings sit in memory only until the drainer has sent them and the broker has confirmed them, and nothing is written to disk. The moment there is no session, a confirmation does not come, or memory would pass 100,000 readings, whatever is in memory goes to the disk queue (internal/forward, bbolt, pure Go, no cgo) in order, everything after it goes behind it, and the drainer reads from the disk, oldest first, taking a reading off it only once the broker has confirmed it. The node goes back to memory only when the disk is empty, so there is never a moment when newer readings can pass older ones.

I built this twice, and the first version was wrong for the hardware. It wrote every reading to disk before sending it, so a kill lost nothing the node had accepted. It also wrote about 5 MB a second at 2,000 readings a second, which is 450 GB a day, from about 160 bbolt commits a second at eight pages each. That kills an SD card and is a silly amount of wear on any flash. The disk cost in steady state now is nothing (the numbers are below), and what I gave up for it is stated plainly in the next section.

A hard kill with the uplink up loses what was in memory. A SIGKILL, a power cut or a kernel panic, while the link is up and the broker is confirming, takes the readings that had not yet been confirmed. That is about one poll: at 2,000 readings a second and a confirmation round trip of a few milliseconds, a few hundred readings at the very most, and the same number of polls the host would have seen a moment later. They are not counted as dropped, because nothing knows they existed. A graceful stop (Stop, a SIGTERM, a restart) writes what is in memory to the disk first and loses nothing. A cut link loses nothing either way, because the first thing the node does when the link stops confirming is move memory to the disk. The one case this design does not cover is the process dying with the link up, and it is not covered on purpose.

HOTLOOP_EDGE_PUBLISH_QUEUE_MAX_BYTES bounds the outage queue, and the default is 64 MiB, about two and three quarter minutes of outage at 2,000 readings a second (the rig below runs it there). Past it the oldest readings go first and are counted. Set it to 0 and the queue is unbounded: the process warns about that at startup, because a very long outage on an unbounded queue fills the volume instead of dropping old readings the way the historian’s own in-memory queue already does.

A reading the drainer gets out within a couple of seconds of it being queued goes out as the tag’s current value, in one NDATA with the rest of its poll. One that waited longer, or that was left behind by a process that died, goes out flagged is_historical, so a host doesn’t let it overwrite what the tag is doing now. On reconnect the queue drains before anything newer is published, oldest first, because there is only one way out of it.

A replay is packed: up to 200 historical readings share one NDATA, each metric carrying its own timestamp and is_historical. The spec allows it (“Multiple value changes for the same metric MAY be included in the same Sparkplug NDATA message … as long as they have different timestamps”, Sparkplug B 3.0, Operational Behavior, Data Publish), so a message never holds one tag twice at the same millisecond and starts the next message instead. I packed them because one reading per message was the bottleneck, and profiling it said so: 57% of the replay’s CPU was the write syscall paho makes once per packet, and 200,000 queued readings took 58.5 seconds, 3,418 a second, against a plant producing 2,000 a second. A replay slower than the plant is a backlog that never ends. Packed, the same 200,000 readings take 2.9 seconds, 68,237 a second, measured while the node was still on QoS 1. Live readings still go out one poll to an NDATA.

QoS 0, as the spec says, and a round trip to know it arrived. NBIRTH and NDATA go out at QoS 0 and not retained (tck-id-topics-nbirth-mqtt, tck-id-topics-ndata-mqtt, Sparkplug B 3.0, Topics and Messages, Edge Node). The NDEATH will is QoS 1 and not retained (tck-id-payloads-ndeath-will-message-qos, Payloads), and so is the graceful NDEATH Stop publishes. The catch with QoS 0 is that there is no acknowledgement: a publish “completes” when the bytes reach the local socket, and that works fine into a dead link, so treating it as delivery would take readings off the disk exactly when they matter. Instead the node sends a request behind each chunk (a SUBSCRIBE to its own NCMD topic, which it has to be allowed to do anyway) and keeps the chunk until the broker answers it. A connection is one ordered byte stream and a broker handles a client’s packets in the order it reads them, so the answer means it read everything before it. That is the assurance a PUBACK gave, one round trip per chunk instead of one acknowledgement per message. A test writes into a silent link and checks nothing leaves the queue, memory or disk; without the round trip, all 30 readings in it did.

A connection that stops answering is given up, not waited out. If the broker doesn’t confirm a chunk, or paho can’t take a message, the node closes that connection itself and starts a new session (next bdSeq, new birth) with everything unconfirmed kept (moved to the disk, if it was in memory). It has to, because a link that dies without a reset takes writes until the kernel’s send buffer is full, and then paho’s one writer goroutine is stuck in a write. That is also the goroutine that sends the keepalive PING, so the keepalive never fires and the node sits in the dead connection for as long as the outage lasts: a six minute test cut found it had not noticed until the link came back. Worse, the pass that had been writing woke up on the new connection and kept sending the rest of its old chunk before the new session’s NBIRTH, with the old session’s sequence numbers. Every pass now belongs to the session it started in and gets an error, not the wire, once that session has ended.

Delivery is at-least-once. A chunk the broker took but never acknowledged is sent again, as history, and a kill between delivery and the node noting it does the same. A host consuming this must tolerate a duplicate; it must never assume this queue already deduplicated for it.

What it promises, and the numbers behind it

• Exact order. Every reading of every tag reaches the host in the order it was read, live and replayed together, across a SIGKILL and a restart.
• A bounded file. The bound is on the queue’s file, the thing that fills the volume, not on the data in it. bbolt’s file is the data plus page headers, branch pages, a freelist and growth slack; under the old bound (bbolt’s own count of leaf bytes) the file came out at twice the data. Anything under 1 MiB runs at 1 MiB, with a warning at startup.
• Nothing lost in an outage. Once the link stops confirming, every reading is on the disk and stays there until the broker has confirmed it, so every reading comes back out through an outage, a dead connection, a graceful stop, and a SIGKILL or a power cut in the middle of the outage. The exception is the one above: a hard kill or power cut with the link up loses about one poll, the readings still in memory.
• Loud, counted drops past the bound. The oldest readings go first, and each one goes into a dropped count that lives in the queue file, so a restart doesn’t wipe the evidence. It’s in GET /api/diagnostics under edgePublish (and in the relay’s /status under uplink), next to what’s queued, the file’s size and whether a replay is running. The log says it once a minute while it’s happening, never once per poll. The count can run a little ahead of the loss and never behind it: readings the host got just before a cut, whose confirmation never came, are still on the disk when the bound takes them, and are counted though the host has them.
• A reading that has only been polled can be lost to a hard kill with the link up. Between the driver handing a reading over and the broker confirming it, it exists only in the process, and a SIGKILL or a power cut takes it. That is the price of not writing every poll to disk, and I chose it over 450 GB a day. Kill it when nothing is in flight and the answer is exact: every reading the plant handed out was delivered or counted.

All of it is measured by test/storeforward: a real HotLoop Edge polling 20 HTTP devices with 25 tags each every 250 ms (2000 readings a second), publishing to a real Mosquitto. Each device answers every poll with a counter one higher than the last, so a host-side judge knows exactly which reading went missing, which came back twice and which came out of order. The uplink is cut by pulling the broker off the network: the Edge’s socket goes silent, no reset, no hang-up. To kill with nothing in flight the plant pauses for a few seconds before the SIGKILL (“quiesced”); the other kills land wherever polling happens to be (“instant”). The runs on 2026-10-12, with a 64 MiB bound. “Before” is 15419f1, the build before this work. “Write-ahead” is 165411f, which wrote every reading to disk first and which I took back out. “Now” is memory while the link is up and disk during an outage:

Details	Before	Write-ahead	Now
Steady state, uplink up, 2,000 readings/s	2,018 readings/s, 0 bytes/s to disk, 9% of a CPU	2,031 readings/s, 5.4 MB/s to disk (450 GB a day), 14% of a CPU	10 minutes: 2,002 readings/s, 0 bytes written to disk (0 bytes a day), 0 bbolt pages written, queue file 32,768 bytes before and after, 9% of a CPU
Replay of a 5 minute outage, at 2,000 readings/s still arriving	829,745 readings in 4 m 11 s, 3,308 a second	194,247 to 308,582 a second	244,335 readings in 824 ms, 296,523 a second
Accounting after a 5 minute outage	lost 441,255, counted 441,255	lost 474,978, counted 474,978	lost 599,165, counted 599,165, all the oldest, all from the bound
Order, aliases, sequence gaps, live inside a replay	0, 0, 0, 0	0, 0, 0, 0	0, 0, 0, 0
Quiesced SIGKILL, uplink up	lost 0	lost 0	lost 0
Quiesced SIGKILL 90 s into a silent 5 minute cut, past the bound	not run	lost 528,167, counted 528,667	lost 520,661, counted 521,161 (the 500 extra were delivered, never confirmed)
Ten instant SIGKILLs, uplink up	lost 0 (an earlier run: 50 in four)	lost 0 (an earlier run: 250)	lost 0 in this run; a kill can take about a poll, see above

The cost is the first row. Write-ahead did about 160 commits a second (one per poll to put it on disk, one per chunk to take it off) at roughly eight 4 KB pages each, which is 5 MB/s and 450 GB a day, and an SD card does not survive that. In steady state the node now does not write to the queue at all: the Edge’s write_bytes counter in /proc did not move in ten minutes. The ten instant kills lost nothing in that run, and that is luck, not a promise: the window is the readings in memory when the process dies, about a poll.

An earlier build found the other half of what this work was for: QoS 0 and a connection that stops answering. See “A connection that stops answering is given up” above. The five minute cut before that fix had the node sitting in a dead connection for the whole outage and then sending a stale chunk ahead of the new session’s birth; 100 readings arrived out of order.

An earlier build found the other half of what this work was for: QoS 0 and a connection that stops answering. See “A connection that stops answering is given up” above. The five minute cut before that fix had the node sitting in a dead connection for the whole outage and then sending a stale chunk ahead of the new session’s birth; 100 readings arrived out of order.

At that load a 64 MiB queue holds about two and three quarter minutes of outage. Size the bound from your own rate: about 180 bytes a reading on disk. The replay is no longer a limit: it runs at more than a hundred times the rate readings arrive.

The judge used to report 4,000 to 6,000 duplicates in a run with no outage. I captured the traffic to find out whose they were: every one was an NBIRTH repeating the last value of a tag an earlier message had already carried, with the reading’s own timestamp, and not one was an NDATA sent twice. A birth has to carry the current value of every metric the node has (that is what a birth is for), and the node births again each time a new tag shows up, so the later births repeat the earlier ones. The publisher was right and the judge was wrong. It now counts those separately as snapshots and keeps “duplicate” for a reading published again as data, which is the only kind a host could mistake for a new one. In the steady run and the ten-kill run, received now equals handed out exactly: 1,296,275 and 916,350.

To run it yourself, build both binaries for Linux and run test/storeforward/run.sh in a Podman lab (its header says how). It only creates and removes containers named sfp-*.

What has and has not been verified

Tested against Eclipse Mosquitto and against this codebase’s own primary-host-mode MQTT driver, a real, decoding MQTT client asserts on the actual protobuf frames this package publishes, including birth aliasing, sequence numbers matching a birth’s own reset, buffering across a simulated process restart with the readings’ order and is_historical flag intact, a requested rebirth keeping bdSeq, and a graceful death carrying one. The subscribe-before-birth rule is tested against a broker deliberately slowed down to acknowledge the subscription, on the first connect and after a dropped connection, and against one that refuses it outright. The uplink going away is tested through a proxy that refuses connections and one that swallows them without a word: the node starts with the broker unreachable and replays once it’s back, live readings wait behind a replay, and nothing from a dead session is resent onto the next one.

Not yet run against a certified Sparkplug host application, Ignition’s MQTT Engine, Chariot, or similar. docs/PROTOCOLS.md draws exactly this line for S7comm and EtherNet/IP: tested is a real claim, verified against real infrastructure is a further one, and this feature has only the first so far.

Configuration

Variable	Default	Details
HOTLOOP_EDGE_PUBLISH_ENABLED	false	The feature is off until this is set
HOTLOOP_EDGE_PUBLISH_BROKER	n/a	Bare host, host:port, or a full URL
HOTLOOP_EDGE_PUBLISH_GROUP	n/a	Sparkplug group id
HOTLOOP_EDGE_PUBLISH_NODE	pod hostname	Sparkplug node id
HOTLOOP_EDGE_PUBLISH_USER / _PASSWORD	n/a	Broker credentials
HOTLOOP_EDGE_PUBLISH_TLS	false
HOTLOOP_EDGE_PUBLISH_TLS_SKIP_VERIFY	false	Per-deployment opt-in; never set this by default
HOTLOOP_EDGE_PUBLISH_QUEUE_PATH	/data/forward.db	The forward queue’s file
HOTLOOP_EDGE_PUBLISH_QUEUE_MAX_BYTES	67108864 (64 MiB)	The queue file never grows past this; the oldest readings are evicted, and counted, to stay under it. 0 is unbounded, with a warning. Anything else under 1 MiB runs at 1 MiB, with a warning

In the Helm chart

The Gateway chart has an edgePublish: block that sets all of these. It’s off by default, and off renders exactly what the chart rendered before the block existed. Don’t set these through extraEnv any more.

edgePublish:
  enabled: true
  broker: tcp://central.example.local:1883
  group: Plant1
  node: ""               # empty uses the release name
  username: ""
  password: ""           # goes into a chart-managed Secret
  existingSecret: ""     # or name your own Secret (key: secretKey)
  tls: false
  tlsSkipVerify: false
  queueMaxBytes: 67108864   # 64 MiB; 0 is unbounded
  persistence:
    enabled: true
    size: 1Gi

A few things the chart decides for you:

• The queue file is /data/edge-publish/forward.db, on its own PVC. It isn’t a value, because anywhere else in the pod is the read-only root filesystem. With persistence.enabled: false it goes on an emptyDir, which keeps an outage’s readings through a container restart but not a rescheduled pod.
• The node id defaults to the release name, not the pod host name. A Deployment pod gets a new host name on every rollout, and the host would see a new edge node every time, with the buffered readings replayed under it.
• The Deployment switches to Recreate and replicaCount must be 1. The queue file takes a lock: under a rolling update the new pod can’t open it, never goes ready, and the rollout hangs. And two replicas would be two publishers on one node id.

The Edge Relay chart (charts/hotloop-edge-relay) has its own edgePublish: block along the same lines, minus enabled: the relay always publishes.