Every deploy is a record

“What changed, who changed it, put it back” is the first thing anybody asks after a line stops. Three .bak files answer none of it. They don’t know who, they roll over after three saves, and the one you need is always the fourth.

So every deploy writes an immutable record under data/deployments/: who deployed it (from their token, never from anything the client claims), from where, when, the note they left, the revision it replaced, the flow file byte for byte, and the credentials as they stood after the deploy, encrypted exactly like the credential file. Records are append-only. The log keeps the last 100 by default (history.retain, and 0 keeps all of them), and it never prunes the newest, because the newest is what’s running.

curl -s -H "Authorization: Bearer $TOKEN" http://localhost:1880/deployments
curl -s -H "Authorization: Bearer $TOKEN" http://localhost:1880/deployments/42/flows > flows-42.json

A note goes in with the deploy, as "note" in the wrapped document ({"rev": ..., "flows": [...], "note": "..."}) or as a HotLoop-Flow-Deployment-Note header. The document form is there because a browser won’t put anything outside Latin-1 in a header, and people write notes in their own language.

Two things it does that you’d only miss once they bit you:

• A hand edit gets caught. If the flow file on disk isn’t the last thing the log recorded, because somebody edited it while the process was down, startup records it as a baseline entry that says so. Same on the first start with a flow file already there. Otherwise a hand edit quietly becomes a past nobody wrote down.
• A log that can’t be written doesn’t stop the line. The flow file is already saved by then, so refusing the deploy would leave the disk and the running flows disagreeing until the next restart. The deploy goes ahead, and the response and the log both say, in words, that this one has no record.

The API never hands out the credentials in a record, encrypted or not. They’re kept for one job, putting them back, and a history endpoint isn’t a way to walk off with them.

The diff reads like a review

A line diff of a flow file is honest and useless. Drag a node twenty pixels and it’s a change. Change a threshold and it’s a change. The reviewer can’t tell which one is about to stop a press. So Flow diffs node by node:

1 changed, 2 moved

Tab "Line 3" (t1)
  ~ switch "Pressure check" (sw1)
      rules[1].v: "7.5" -> "8.0"
      wire added: port 1 -> change "set alarm" (alarm1)

Moved on the canvas only
  mqtt in "press 01" (in1)
  debug "ok" (ok1)

Added, removed and changed, property by property with the old and new value, which wires moved by output port, and layout kept apart from logic: moving, resizing or hiding a label is “moved on the canvas”, never a change, so a tidy-up reads as a tidy-up. A credential reads as “set” or “changed”, never with its value. Edits inside a subflow show up on the subflow, and dragging one of its ports is layout too.

Same answer everywhere, from one engine package:

• GET /deployments/{from}/diff/{to} compares two deployments.

• POST /flows/diff compares what’s live with a document nobody has deployed yet, which is what you look at before pressing deploy.

• hotloop-flow diff old.json new.json on the command line. -json for a program. Exits 1 when something differs, like diff.

• Inside git. Make it git’s diff driver for flow files and git diff, git log -p and git show print the semantic diff instead of a JSON hunk:

  git config diff.hotloop-flow.command 'hotloop-flow diff'
  echo 'flows.json diff=hotloop-flow' >> .gitattributes

  Or as a difftool: git config difftool.hotloop-flow.cmd 'hotloop-flow diff "$LOCAL" "$REMOTE"', then git difftool -t hotloop-flow.

A flow diffed against itself is no change at all, for any document the parser accepts. CI fuzzes that on every pull request, because a diff that invents a change is worse than no diff.

Put it back

curl -s -X POST -H "Authorization: Bearer $TOKEN" \
  -H "HotLoop-Flow-Deployment-Rev: $CURRENT_REV" \
  -d '{"note":"42 broke the label printer"}' \
  http://localhost:1880/deployments/41/rollback

A rollback deploys an old record again as a new record. History stays append-only: deployment 41 isn’t touched, and the rollback is its own entry that says “rollback to deployment 41”, who did it, and why. The flow file comes back byte for byte, so it’s the same revision it was then.

The credentials come back too. Every record carries them as they stood, so a node you deleted returns with its broker password instead of returning and failing to log in. That’s tested against a real Mosquitto that refuses anonymous clients: delete the subscriber, deploy, roll back, and it logs in again and messages flow. CI runs that broker on every pull request.

It refuses three ways, before anything is written. A stale HotLoop-Flow-Deployment-Rev gets the same 409 a deploy would, because a rollback racing somebody’s deploy is still a race. A deployment that retention already removed is a 404. And a record whose credentials can’t be decrypted with the current credential secret is a 422, because flows that can’t log in to anything are a rollback that only looks like it worked.

In the editor

Deploy doesn’t deploy. It opens a review: the engine’s diff against what’s running, drawn on the canvas and listed in words, with a box for the note. Green is added, amber is changed, blue is only moved, and red ghosts sit where removed nodes and wires used to be. A tab with anything changed on it gets a dot. The second Deploy button is the one that deploys, and the note goes in the log. Edit anything while the review is open and it closes, because a review of something you’ve since changed is a lie about what you’re about to push.

The history panel lists every deployment with who, when and the note. Changes opens that deployment on a read-only canvas with what it changed drawn on it, over the top of your working copy, which stays exactly as you left it. Roll back asks why, warns you if you’re about to throw away unsaved edits, and deploys it.

A deploy conflict shows what the other person did. It used to be a browser confirm() asking you to overwrite theirs or lose yours without saying what theirs was. Now it says who deployed, when, their note, and their change node by node, then lets you load theirs, overwrite with yours, or keep editing.

Copy, cut and paste work like you’d expect, and they speak the flow file. Select nodes, Ctrl+C, and what’s on your clipboard is the same JSON every other flow tool reads, so you can paste it into a chat, a ticket, or another box. Paste puts it where your pointer is, with new ids every time, so pasting twice gives you two copies and never a collision. A wire to something you didn’t copy is cut rather than left pointing at nothing, and a configuration node the nodes use comes along. If that configuration node is already in the flow you are pasting into, the copy shares it instead of opening a second connection to something that counts them. Credentials never go on the clipboard. A password in a paste buffer is how it ends up in a bug report.

Import and Export do the same for a selection, one flow, or everything, with a box to copy from, a Download button, and a file picker on the way in. Garbage gets a reason (“the id a appears twice”, “entry 1 has no id”) and the dialog stays open. A node type this runtime doesn’t have comes in gray and dashed, and the import tells you which ones, because a silently dropped node is worse than a loud one.

Subflows are edited on a canvas of their own. Press + subflow, and it opens with its input and output ports on it. Wire them to the nodes inside like anything else. Subflow properties sets the name, the color, how many outputs it has, and the properties an instance can set. Instances on flows follow when the shape changes, and the palette lists every subflow so you can drop one. Double-click an instance to give it its own name and its own value for each property. A subflow can’t be placed inside itself or inside one that holds it, and one still in use can’t be deleted. The editor says so rather than letting the next deploy find out.

Selection to subflow. Select nodes and press Selection to subflow. They move into a subflow of their own and an instance takes their place. Every wire that crossed the edge of the selection becomes a port: the one way in is the subflow’s input, and each output of a selected node that went somewhere else is an output of the subflow, so the flow does what it did. A subflow has one input, so if the selection is wired into two different nodes it says so and changes nothing, because feeding both from both would be a different flow. Select a group and it comes along with everything in it. A configuration node that belongs to the flow and that only the selection uses goes into the subflow with it, along with whatever it points at. One that something left behind also uses goes global instead, so the two ends keep sharing the one connection, because a second broker connection on a box that counts them is the kind of surprise I do not ship. Environment variables the moved nodes read (typed as env, as ${NAME} or $(NAME) in a string, or env.get) and that the flow declares become properties of the subflow that forward to the flow, so the value still comes from where it did and the instance can override it. Ctrl+Z undoes the lot.

Flows. Double-click a tab, or press Flow properties, to rename it or delete it. Deleting asks, takes every node on the flow with it, and cuts the wires and links other flows had into it so the file isn’t left pointing at nothing. It takes the deleted nodes out of any Catch, Status or Complete scope that named them, too. A Catch or Status whose whole list was deleted keeps its last id, since an empty list means the whole flow and I won’t flip a node from watching one thing to watching everything behind your back. Ctrl+Z brings it back until you deploy. The last flow can’t be deleted, because a runtime with no flow has nowhere to put a node.

Configuration nodes are in the palette under configuration. They sit on no canvas, so you double-click the type, fill in the dialog, and it’s added to the list under configuration in use, where a double-click edits it and the cross deletes it. One a node still uses won’t delete, and says how many do. That’s where “Add one from the palette first” in a node’s dialog sends you.

Deleting a wire. Alt-click it. A plain click does nothing, so brushing past a wire while you select can’t break a flow. Ctrl+Z puts it back.

Durable wires. Double-click a wire. Tick Durable and it’s drawn heavier and blue, and the node it leaves gets ew_durable in the file. The mark is on the pair of nodes, so it holds for every wire between them. It’s wiring to the runtime, so marking one never restarts the node.

Groups. Ctrl+G wraps the selection, Ctrl+Shift+G takes it apart. Double-click the title bar to name and color it. Drag a group and everything in it goes with it. Drag a node into one and it joins, drag it out and it leaves. Groups nest. Deleting a group deletes what’s in it, and Ctrl+Z puts it all back.

Search (Ctrl+F, or the button) looks in every flow and subflow at once: names, types, ids, and every setting, including a line in the middle of a function body. A hit takes you to the right tab with the node selected and in the middle of the view. Configuration nodes show up too, and their hit opens their dialog, since they have no place on a canvas.

Debug filtering. Narrow the sidebar to one flow, to the nodes you’ve got selected, to a word, or to just the errors. It applies to what’s already in the log, not only to what arrives next, so narrowing it never makes the line you were looking at disappear.

All of it is clicked through in a real browser on every pull request, against the real binary with the editor embedded. That includes every node dialog the runtime offers: the list comes from GET /nodes, so a node added next month is in the test next month without anyone remembering to add it. Playwright is a dev dependency of web/e2e only and never reaches the bundle. Writing those tests found two things that had never worked in Chromium: double-clicking a node to open its dialog, and dragging a wire onto a node. The canvas holds the pointer for the whole drag, so the event’s target was always the canvas and a wire let go over a node never found the node. Both about as embarrassing as bugs get in a visual editor. Both fixed.

Flows as code

The repository is where flows get reviewed, so the repository should be able to deploy them. Keep the flow file in git, set up the diff driver above so the pull request reads like a review, and let the merge deploy it:

hotloop-flow export -o flows.json          # the live file, byte for byte, to commit
hotloop-flow deploy -file flows.json -note "PR #12: line 3 pressure check"

deploy prints the diff against what’s running, the same text as everywhere else, then deploys against the revision that diff was made from. If somebody deploys in between, that’s a 409 and a red pipeline, not a quiet overwrite. -expect-rev pins it to the revision a reviewed plan was made against, and -dry-run prints the diff and deploys nothing. A file with nothing to change deploys nothing and adds nothing to the history, so a merge that didn’t touch the flows doesn’t litter the log. A node that fails to start deploys and still fails the job, because the rest of the flow is running and somebody needs to look. export -deployment 41 gets any deployment back out of the history.

A pipeline never holds an admin password. It gets an API token:

hotloop-flow token -name ci

prints a token, once, for the CI secret store, and the hash to give the runtime: HOTLOOP_FLOW_DEPLOY_TOKEN_HASH for a deploy token, or auth.tokens in the config file for anything else. The runtime only ever holds the hash, same as a password. A deploy token can read the flows and deploy, which includes rolling back, and nothing else: no settings, no inject, no editor. Both commands read the token from HOTLOOP_FLOW_TOKEN (or -token-file) and the instance from HOTLOOP_FLOW_URL (or -url), never a token on the command line, where it would land in the process list and every CI log that echoes the command. The history shows the token as the deployer, token:ci, with the note the pipeline passed.

Credentials never travel this way. They aren’t in the flow file, so a deploy from git keeps every broker password exactly where it was.

A git mirror, if you want one

The deployment log lives on the box. A repository is where everybody else already looks: it has a web view, blame, backups and people who know how to read it. Point Flow at one and every deployment becomes a commit of the flow file on a branch only Flow writes to, authored by whoever deployed it, at the time they deployed it, with their note as the message:

history:
  git:
    url: https://git.plant.example/line3/flows.git
    branch: main              # one nobody else pushes to
    path: flows.json
    emailDomain: plant.example  # dana deploys, the commit is from dana@plant.example

The password is an access token from the git server, and it goes in HOTLOOP_FLOW_GIT_PASSWORD from a Secret, never the file (HOTLOOP_FLOW_GIT_URL and HOTLOOP_FLOW_GIT_USERNAME work too). A URL with a password in it is refused at startup, so the URL can always be logged. http:// and https:// only: the image has no git binary for file:// and no keys for ssh://, and a mirror that fails on every push is worse than one that refuses to start. Rollbacks and startup baselines say what they are in the commit, and every commit carries a Flow-Deployment: N trailer, which is how a restarted process, or one whose clone got wiped, picks up exactly where the repository says it got to, without a duplicate.

It never holds up a deploy. Pushing happens on its own goroutine. A git server that is down, slow or refusing the push costs a deploy nothing: the commits wait in a local clone under data/git-mirror and go out on the next push that works, retried every minute. GET /deployments says how far the mirror got and the last error, and hotloop_flow_git_mirror_behind_deployments is the number to alert on. Tested against git’s own smart-HTTP server behind basic auth: three deploys make three commits by three authors, a push the server refuses waits and catches up when it stops refusing, and a deploy with a git server that takes ten seconds to say no returns in milliseconds.

It’s pure Go (go-git), so the binary stays static. It costs about 3.8 MB of binary, which is why it’s measured and not assumed, and why the 40 MiB ceiling in CI still has room.