The plant floor

Written against main

This page follows main at f1f417f, and it differs from the copy in HotLoop Flow 2.2.0, the latest release. Parts of it may be newer than what you have installed. The release notes say what moved.

scan could find a PLC and nothing in the palette could talk to it, which is a funny thing to ship in a product for plants. Modbus TCP, S7, EtherNet/IP, OPC UA and Sparkplug B are in now. Each one is a config node that holds the connection to the device and action nodes that point at it, same shape as the InfluxDB node, and the editor builds every dialog from the descriptor like everything else.

They sit on open source libraries: simonvetter/modbus v1.6.4 (MIT), gopcua/opcua v0.9.1 (MIT), robinson/gos7 (BSD-3-Clause, no tagged releases, so it’s pinned to the latest commit, and carried in third_party/gos7 with one line changed, third_party/gos7/HOTLOOP_PATCH.txt says which) and danomagnum/gologix v0.44.0-beta (MIT). Nothing is copied out of HotLoop’s drivers, which are a different license and stay that way.

Modbus TCP is modbus read and modbus write on a hotloop-flow-modbus server. Holding registers, input registers, coils and discrete inputs. Registers read as uint16, int16, uint32, int32, float32, uint64, int64 or float64, with the byte order and word order as switches, because the two devices on your line will not agree on either and the manual will not say. A quantity is values, not registers, so two float32 is four registers on the wire. Addresses are zero based as they go on the wire, and the dialog and the error both say out loud that register 40001 in a manual is address 0.

S7 is s7 read and s7 write on a hotloop-flow-s7 server, over ISO on TCP. gos7 is a Go implementation of the protocol and has none of Snap7’s C library in it, which matters because Snap7 is LGPL and I’m not linking that into a product I ship. You write the address the way Siemens does: DB1.DBX0.0, DB1.DBW4, M10.2, MW20, I0.0, QB4, with E and A for the German spellings. The size letter has to agree with the data type, so DB1.DBW0 read as a float32 is refused before it goes anywhere, with the address you meant. A quantity of 3 on a bit is three bits on from the one you named. A string is an S7 STRING: the write reads its maximum length first and refuses text that won’t fit instead of writing past it. A bit is written with the protocol’s own bit write, which changes that bit and leaves the other seven in the byte alone. An S7-1200 or S7-1500 says no to all of it until PUT/GET is permitted and the data block’s Optimized block access is off, and the errors say so, because that is the thing everybody trips on.

EtherNet/IP is ethernetip read and ethernetip write on a hotloop-flow-ethernetip controller, tag access on ControlLogix, CompactLogix and Micro800. You name the tag the way Logix spells it, with members (Pump1.Speed), array elements (Samples[4]) and bits (Status.3), and the data type it holds. A quantity above 1 moves that many array elements from the one named. A tag read as the wrong type is refused and the error says to use the tag’s own. The path to the CPU is 1,0 unless you change it, and empty for a controller you reach directly. PLC-5, SLC and MicroLogix speak PCCC and not CIP tags, and they’re out. Yes, the library is tagged beta. It’s the newest release there is, and I’d rather tell you than leave it out.

OPC UA is opcua read, opcua write and opcua subscribe on a hotloop-flow-opcua server. Security mode None, Sign or SignAndEncrypt with the Basic256Sha256 or Aes128_Sha256_RsaOaep policy, anonymous or a username and password. A client certificate comes from a tls-config (it needs an application URI in the subject alternative name), and without one the node makes a throwaway certificate each time the flow starts and logs a warning, because the server then has to be told to trust a new one every time. If you set a server thumbprint the node refuses any other certificate. If you don’t, it accepts whatever the server shows, which encrypts the conversation and proves nothing about who you’re talking to. Pin it.

Sparkplug B is sparkplug out, sparkplug in and sparkplug command on a hotloop-flow-sparkplug connection, which is either an edge node or a host application, because each has a will message and an MQTT connection has one. As an edge node, the connection is born when it connects, publishes each sparkplug out node as the edge node’s own metrics or as one device’s, dies with its will if it’s cut off (with a new bdSeq in the next birth, so a host can tell the sessions apart), says NDEATH before a clean disconnect, and answers a rebirth request. Every metric has to be declared on the node, because a birth lists everything before any data is sent. sparkplug in on an edge node is how a flow acts on a command. As a host application it publishes STATE the way the spec asks, with the will, the retain flag and the timestamp that has to match, then follows every edge node it can see: names and aliases from the births, datatypes from the births, sequence numbers checked, a death marking what the birth listed as stale, and a rebirth request when a gap isn’t filled inside the reorder timeout or data shows up from a node it never saw born. sparkplug command sends an NCMD or a DCMD, typed from the birth and by alias when the birth gave one. MQTT 3.1.1 through paho.mqtt.golang. The wire format is written by hand from the Eclipse schema and the 3.0 specification, and tested against that schema compiled with Google’s protobuf, which is a different implementation from mine. The schema is EPL-2.0, so it isn’t in this repository: CI fetches it pinned to a commit and a hash. It was also run against a real Mosquitto, and by hand against pysparkplug, in both directions, edge node and host.

An edge node can name a primary host application. It subscribes to that host’s STATE and says nothing, no NBIRTH and no DBIRTH, until the host says it is online. A death certificate older than the last online STATE is some old session’s and gets ignored. A current one gets an NDEATH, a disconnect and a start over. A list of other brokers is where it starts over: one at a time, round and round, moving on when a broker can’t be reached, when the host is offline there, and, with more than one broker, when the host never published there at all. With one broker an offline host just keeps the edge node waiting, connected, until it isn’t. The edge node is on one broker at a time, as the spec says. The array types, DataSet, Template and a metric’s PropertySet are carried both ways: arrays in the packed little endian form the spec gives, a DataSet as {columns, types, rows}, a Template as a definition or an instance with its parameters and members, and properties on any metric as {"engUnit": "degC", "Quality": {"type": "Int32", "value": 192}}. A Template instance whose definition isn’t in the NBIRTH is refused at birth with what to declare. The Eclipse schema reads every one of them off the wire, and its own bytes read back the other way. Not done: a metric’s metadata (files and multi-part bytes), a host application that talks to more than one broker, and MQTT 5. It hasn’t met Ignition’s MQTT Engine.

When the device goes quiet. This is what a plant floor does all day, so it is tested the ugly way, against the real simulators with a line in between that swallows the traffic (a hung PLC) or slams the door (a switched-off one). A read or write that gets no answer fails inside the timeout, says so in words (no answer from 10.20.1.15:502 within 3s. The device is silent. Check that it is powered...), drops the connection and opens a fresh one on the next message. An OPC UA subscription notices on its own, turns its badge red, raises one error for a Catch node per outage and not one per retry, waits 1 second and then longer up to 30, and picks the subscription back up when the server returns. No redeploy. Errors name the thing to change: the Modbus exception code and what it usually means, the OPC UA status code and which setting it points at, the S7 refusal that means PUT/GET is off, a refused connection versus a silent one. An OPC UA server that takes the connection and never answers endpoint discovery costs the node that asked its timeout and nobody else anything: the other nodes on that server, and a flow that is being stopped, don’t queue behind it.

HotLoop is hotloop in, hotloop state, hotloop script and hotloop recipe on a hotloop-flow-hotloop connection, which is a URL and an account. Flow runs next to HotLoop and calls its HTTP API from the outside, nothing else, and a flow that never picks a HotLoop never makes a call. The connection signs in with POST /api/auth/login, sends the bearer token HotLoop hands back, and signs in again when HotLoop says the session is gone. An operator account is enough for all four nodes and for the gated write below. Leave the username empty only for a HotLoop that runs with no login at all.

hotloop in follows HotLoop’s live feed (GET /api/stream) and sends a message for each entity that changes state: msg.topic is the entity id, msg.payload the new state, and msg.hotloop the entity id, the state it had before, its attributes, its quality and its timestamps. It follows the entity ids and patterns you list (sensor.tank_*, or *). The picture HotLoop sends when the node connects isn’t a change, so it isn’t sent, unless you turn on “Send the current state when it connects”. An attribute that moved without the state changing isn’t a change either. If the link drops, the badge goes red, one error reaches a Catch node for the outage and not one per retry, the wait grows from 1 second to 30, and when it is back an entity that changed in the meantime is sent and one that didn’t isn’t. HotLoop sends every state as text, so “Numbers as numbers” turns 12.5 into a number and leaves on and unknown alone.

hotloop state reads one entity when a message arrives (GET /api/states/{entity_id}) and sends the message on with the same fields. An entity that doesn’t exist, or is disabled, raises an error with HotLoop’s own words, which say which.

hotloop script runs a script (POST /api/scripts/{id}/run) with the fields you give it, and hotloop recipe applies a recipe (POST /api/recipes/{id}/apply) with its parameters and, if you want one, an older version on purpose. Both are HotLoop’s own: it checks every field and parameter and never moves a value to fit, runs every write through its write gate, and keeps the record. Flow sends the reason flow:<node id>, with your own words after it if the node has a reason, and HotLoop records that beside the run or the application, in its write audit and in its logbook, under the account Flow signed in as. A script waits for its run to end unless you turn that off, and the message goes on with the run, trace included, only if it succeeded. A recipe goes on only when every target was written, with each value’s verification. Anything else arrives at a Catch node with HotLoop’s reason and its status: a script that is busy, a field out of range, a run that failed at step 2, a recipe refused whole because one target isn’t armed, or one that stopped halfway, which says how many targets were written. Nothing that was written is undone, because that is HotLoop’s rule too.

Writes change equipment, and there is no ceremony, unless HotLoop is there. I looked for a confirmation pattern in Flow to follow and there isn’t one for equipment. The guards that exist bound what a node may reach (discovery.allowedCIDRs for scan, the file and secret scopes), and nothing asks “are you sure” before a message does something. So the write nodes don’t either. A write node is a deliberate thing someone placed on a canvas and put through the same deploy record as everything else, and the message arriving at its output means the device acknowledged it.

When HotLoop is on the network, the equipment is its to guard. Every equipment write node (modbus write, opcua write, s7 write and ethernetip write) has Write through HotLoop, empty by default, and HotLoop tag, the tag in HotLoop this write is for, written device/tag. With a HotLoop picked, the value goes to POST /api/write and not to the PLC. HotLoop’s write gate decides, in its order: that writes are switched on, that the tag is armed, that the value is in the tag’s range. HotLoop’s own address, data type and scale for that tag apply, so the address and data type on the Flow node are not used. The node’s equipment connection stays picked, so clearing the HotLoop puts the node back exactly as it was. A write the gate refuses never reaches the equipment and arrives at a Catch node with the gate’s words (HotLoop's gate refused the write to plc-1/sp: value is outside the tag's configured range: 120 is outside 0..95 (HTTP 422)). A HotLoop that can’t be reached fails the write too. It never falls back to writing directly, because that would be a write the gate never saw. The gate takes one bool, number or text per write, so a list of registers is refused, with a message saying so.

HotLoop is tested against the real thing. The published HotLoop IoT 4.18.10 image runs in CI with a login and writes on, next to the Modbus server below. The tests make a device, tags, a script, a recipe and an operator account through HotLoop’s own API, then check both ends: what HotLoop said, and what the simulated PLC holds afterwards. A write through the gate lands in the register. One outside the tag’s range, one to a tag that isn’t armed, one to a tag HotLoop doesn’t have, and one with HotLoop unreachable all fail with the gate’s words and leave the register where it was. The script and the recipe run and write through the gate, a field out of range is refused, hotloop state reads what HotLoop holds, and hotloop in sends the change a gated write made, with the state it had before. HotLoop’s write audit and logbook were checked for the account and the reason. Signing in again after HotLoop forgets the session is tested, and so is a refused login. The reconnect and the diffing of the feed are tested against a small stand-in for HotLoop that answers in the documented shapes, because dropping a real HotLoop’s connection on cue isn’t something I can do. Not done: HotLoop records the account Flow signs in as, and flow:<node id> goes in as the reason, because the API gives a client no way to name its own actor. The connection trusts the system’s certificate roots and nothing else, so an https HotLoop with a private CA isn’t covered. No node calls a HotLoop service directly, like switch.turn_on: a script or a recipe does that, with HotLoop’s checks.

Tested against Microsoft’s OPC PLC (opc-plc 2.15.11) with anonymous and username logins across None, Sign and SignAndEncrypt, and against pymodbus through oitc/modbus-server 2.3.0, with real exception codes coming back. S7 is tested against the server in python-snap7 3.2.1 (MIT), written in pure Python and a different implementation of the protocol from gos7. EtherNet/IP is tested against ab_server from libplctag 2.7.2, the simulator libplctag runs its own tests on, and the values Flow wrote were read back with libplctag’s own client as well. Both are built from testdata/plant, run as separate containers, and nothing from either is linked into Flow (libplctag is MPL-2.0 or LGPL). CI starts all four and fails if a test skips instead of running.

Neither the S7 nor the EtherNet/IP nodes have met a Siemens or Rockwell box yet. The python-snap7 server doesn’t do a bit write the way a CPU does (it overwrites the whole byte), so the bit write is checked on the wire, request by request, and not by reading the byte back. ab_server matches tag names with their case and Logix doesn’t, and gologix sends them lower case, so the simulator’s tags are lower case.

Aes256_Sha256_RsaPss isn’t in the OPC UA dropdown, and the fault is in gopcua. v0.9.1 pads RSA-OAEP-SHA256 with 130 bytes where the number is 66 (two hash lengths of 32, plus 2), so the secure channel request is cut into blocks the .NET stack under OPC PLC doesn’t expect, and it answers BadSecurityChecksFailed. I changed that one constant in a copy of gopcua and the same client connected to the same OPC PLC in Sign and in SignAndEncrypt. The fix is two open pull requests on gopcua (#915 and #931) and no release has it, so the policy stays out, and a test fails the day a gopcua update carries it. Basic256 and Basic128Rsa15 are deprecated and the simulator doesn’t offer them, so they’re out too. A setting I can’t test doesn’t get a dropdown entry.

Not done: no Modbus RTU, no Modbus over TLS, no OPC UA events, history or method calls, no browse, no S7 timers or counters, no PCCC controllers, and 64 bit integers past 2^53 come out as a string of their exact digits because a flow’s numbers can’t hold them.