HotLoop Flow 2.16.3 release notes
HotLoop Flow 2.16.3 stops the CPU a dashboard chart cost from growing the longer Flow runs. On a plant running about 13,700 nodes at about 3,000 messages a second, 2.16.2 started at about 130 millicores and climbed to about 420 within two hours. The cause was the chart history the dashboard keeps for a page that opens later. Nothing in your flows, settings or chart changes.
What changed
Section titled “What changed”- A chart point no longer copies the whole chart. Every point a chart took rebuilt that chart’s list of points and counted every point to apply the per-line limit. The cost of one point grew with the points already held, so it climbed for the first hour, until the chart held everything it keeps, and stayed there. On a chart with four lines of 4,800 points one point cost about a millisecond and three megabytes of garbage, and at the plant’s 96 chart points a second that was about 260 MB a second of garbage and a collection cycle every three seconds. Each line is now its own list that sheds old points by re-slicing, so a point costs the same on an empty chart as on a full one.
- What a page gets when it opens is the same. The points arrive in the order the lines first appeared, a line that has gone quiet still ages out of what is sent, and the limits per line (
removeOlder,removeOlderPoints, and 5,000 a line) are unchanged. - It is skipped unless that variable is set. A unit test in the dashboard package fails if a point on a full 16,000 point chart allocates more than 4 KB; it measured 191 KB before this release.
What it fixes, with numbers
Section titled “What it fixes, with numbers”On the plant’s own pod, 2.16.2 went from 0.11 collection cycles a second over its first 90 minutes to 0.37 a second, and from about 130 to about 420 millicores of CPU. Cost of one point on a full chart, measured on one machine:
| Details | 2.16.2 | 2.16.3 |
|---|---|---|
| One line of 4,800 points | 157 microseconds, 535 KB | 0.14 microseconds, 217 bytes |
| Four lines of 4,800 points | 1.1 milliseconds, 2.9 MB | 0.11 microseconds, 106 bytes |
New settings
Section titled “New settings”None.
Upgrade
Section titled “Upgrade”A plain helm upgrade to the 2.16.3 image and chart, or the new tag in your Quadlet unit. Nothing happens on first start and nothing needs migrating. A flow file loads and saves byte for byte as before. This release builds the amd64 image. The 2.16 and 2 tags keep the arm64 image they already hold from 2.16.1, and arm64 comes back in 2.16.4. If you are on arm64, stay on 2.16.1 until then.
Image: ghcr.io/hotloop-io/hotloop-flow:2.16.3, linux/amd64. Chart: https://hotloop.io/hotloop-flow/. The README at this tag has the rest, with pictures: https://github.com/HotLoop-io/hotloop-flow/blob/v2.16.3/README.md