Store and forward

Mark a wire durable and, while the node on the other end keeps up, it is an ordinary wire and writes nothing to disk. The moment that node stops taking messages, the wire spills to the same SQLite file before the sender moves on, and a pump delivers them oldest first, one at a time, until the target takes each one. So the historian goes down, the readings pile up on disk instead of blocking the flow or getting dropped, and they go in, in order, when it comes back. The price of writing nothing while it’s healthy is that a kill -9 then loses what was in memory, about one poll’s worth, and I’d rather say that here than have you find out.

{ "id": "count", "type": "function", "ew_durable": ["influx"], "wires": [["influx", "reply"]] }

You don’t have to write the key by hand. Double-click the wire in the editor and tick Durable.

It survives a restart, keeps order, honors a disk quota (256 MiB by default, and the overflow policy of the node on the other end decides what gives when it’s full), and shows its depth as hotloop_flow_durable_wire_depth. It’s at-least-once, not exactly-once, and I’d rather tell you that here than have you find out in a postmortem: the exact words, and how influxdb out stays correct anyway, are in docs/durable.

The proof is docs/durable/store-and-forward.sh: a real InfluxDB 2.9.1 in Podman under twenty readings a second, killed for ten minutes, brought back, with the runtime killed with kill -9 once mid-outage and once mid-drain. 15,982 readings accepted, 15,982 rows stored, none missing, none doubled. A healthy durable wire writes nothing of its own, and with context flushed every thirty seconds the whole process wrote 1.7 MiB an hour at twenty messages a second (0.04 GB a day). The tables and the commands are in docs/durable.

A message the target refuses forever

By default a wire keeps retrying a message its target fails, with a pause that grows to five seconds, and nothing behind that message moves. For a historian that’s down that’s exactly right. For one malformed reading the historian will never accept it’s a wire stuck for good, so there’s a way out, and it’s off until you ask for it.

{ "id": "count", "type": "function", "ew_durable": ["influx"], "ew_durable_attempts": { "influx": 20 }, "wires": [["influx", "reply"]] }

Give the wire a number of attempts, in the editor’s wire dialog or as above, and after the target has refused one message that many times from the queue the wire raises it to the flow as an error from the node that refused it, with the message attached the way any node error carries it, and moves on to the next. A Catch node on that flow gets it and can park it, alert on it or write it somewhere. With no Catch the error still shows in the debug sidebar and the message is gone, so wire one. The first refusal, in memory, reaches the Catch as it always did, so a Catch sees the same message twice: once when it first failed and once when the wire gave up. The count is per message and in memory, so a restart starts it over. hotloop_flow_durable_wire_given_up_total counts them, beside the depth. A limit that isn’t a whole number of 1 or more is reported and the wire carries on retrying, because ignoring it quietly would leave a wire stuck that somebody had told to stop.