k3s and Helm

The chart deploys two workloads: the gateway (a Deployment, it holds no durable state of its own) and TimescaleDB (a StatefulSet with a volume). That is the default. database.type picks another database: plain PostgreSQL, or SQLite, which is no second workload at all (SQLite, below).

Install

Through the EmberNET dashboard’s App Store, or directly:

helm install hotloop ./charts/hotloop \
  --namespace hotloop --create-namespace

Or from the chart repository, which is also the URL the App Store points at:

helm repo add hotloop https://hotloop.io/hotloop
helm install hotloop hotloop/hotloop \
  --namespace hotloop --create-namespace

That second form works from 4.16.0 on, the first release published as hotloop. The last one before it was industrial-iot 4.3.1, under the old name, and upgrading from it is its own procedure: see “Upgrading from 4.3.1” in the README’s 4.16.0 notes. The old repository address, https://embernet-ai.github.io/industrial-iot/, is a 404.

A bare install works. The chart generates a database password and an MCP token into the release Secret and reuses them across upgrades, so nothing needs supplying and an upgrade does not rotate the password out from under a running database.

Reaching it needs no ingress and no subdomain. The dashboard discovers the app by its store labels and proxies it at the in-cluster address:

<release>.<namespace>.svc.cluster.local:8080

To look at it directly:

kubectl -n hotloop port-forward svc/hotloop 8080:8080

Values worth setting

Auth

Since 4.4.0. See docs/AUTH.md for the full model; this is the chart side of it.

auth:
  enabled: true    # the default; requires a login
  insecure: false  # set true only for a deployment truly isolated behind the
                    # dashboard's own NetworkPolicy -- see AUTH.md

With enabled: true (the default) the chart generates an admin account into the release Secret on first install, the same way it generates the database password and the MCP token, and reuses it across upgrades:

kubectl get secret hotloop -n hotloop \
  -o jsonpath='{.data.admin-password}' | base64 -d

Setting auth.admin.password explicitly overrides the generated one. Neither this nor a Helm upgrade ever overwrites a password an operator has since changed through the UI, see EnsureAdminUser’s comment in internal/store/users.go.

Safety

The values that decide whether this pod can move machinery.

safety:
  allowWrites: false      # the master switch -- off by default since 4.4.0
  allowMcpWrites: false   # a narrowing of it, for agents
  confirmWrites: true     # the UI asks before writing

Writes are off by default. A tag is read-only until its own writable flag is set regardless, a value outside the tag’s configured range is refused (never clamped: sending 250 when somebody typed 500 is quietly doing something else), and every attempt, including every refusal, is recorded with the actor and the source. But a fresh install shouldn’t need all three of those to line up before it’s safe, so the master switch itself starts closed.

To turn writes on once the deployment has been reviewed:

safety:
  allowWrites: true
  allowMcpWrites: false  # let operators write while agents may not

Historian

historian:
  retentionDays: 90     # 0 keeps everything until the volume fills
  useTimescale: true

logbook:
  retentionDays: 90     # entity state changes; 0 keeps everything (see LOGBOOK.md)

database:
  postgresql:
    persistence:
      size: 20Gi
      storageClass: longhorn

Sizing is driven by tag count, scan rate and deadbands. A thousand tags at one second with no deadbands is roughly 86 million rows a day; the same plant with sensible deadbands is usually an order of magnitude less. Set deadbands before buying storage.

useTimescale: false forces the native-partitioning path, which is what running against a plain PostgreSQL looks like. Useful for reproducing an edge deployment.

When the TimescaleDB image changes. The image carries every older version’s library, so a database made on an older image keeps running on the new one, but on the old extension version, and it stays there until someone runs ALTER EXTENSION timescaledb UPDATE. The Gateway does that on start: it compares the installed version with the one the server ships and, if they differ, runs the update as the first statement of a fresh connection (TimescaleDB refuses it in a session that has already loaded the old version). The log says timescaledb extension updated from=... to=.... That needs a superuser, which the bundled database’s user is. Against an external or managed database where it isn’t, you get a warning with the command to run, and history keeps working on the old version in the meantime. Only happens with useTimescale on and migrations running on start (HOTLOOP_DB_MIGRATE, on by default).

Polling

runtime:
  defaultPollIntervalMs: 1000
  minPollIntervalMs: 100      # a floor applied to every device
  timeoutMs: 10000
  maxConcurrentPolls: 32

minPollIntervalMs is enforced rather than trusted. A 10 ms poll against a serial gateway is a denial of service with extra steps.

OPC UA client

opcuaClient:
  persistence:
    enabled: true      # off by default
    size: 64Mi

A device that connects to a secured OPC UA server, or that logs in with a user name and password the server needs encrypted, keeps a certificate and a trust store of approved server certificates (HOTLOOP_OPCUA_CLIENT_PKI_DIR, /data/opcua-client-pki). Turn this on before adding such a device: the root filesystem is read-only, so without the volume the device reports an error that says so instead of connecting with certificates that would be lost at the next restart. It is off by default because it adds a PersistentVolumeClaim to an existing install, which stays Pending on a cluster with no default StorageClass. Anonymous devices on an unsecured channel do not use it. Approving a server’s certificate, and everything else about these devices, is in PROTOCOLS.md.

Networking

networkPolicy:
  enabled: true
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: dashboard-system
      ports:
        - protocol: TCP
          port: 8080

Narrow ingress to the dashboard’s namespace once it is known. Egress is deliberately unrestricted: this app exists to reach equipment at addresses nobody can enumerate in advance, and an egress policy that silently blocks a poll looks like a broken driver rather than a firewall rule.

Edge deployments

The app image is built for amd64 and arm64, and TimescaleDB 2.30.1-pg16 is multi-arch, so the historian is not what stops an edge install.

resources:
  requests: {cpu: 100m, memory: 128Mi}
  limits:   {cpu: 500m, memory: 256Mi}

database:
  postgresql:
    persistence: {size: 5Gi}
    resources:
      requests: {cpu: 100m, memory: 256Mi}
      limits:   {cpu: 500m, memory: 512Mi}

historian:
  retentionDays: 30

nodeSelector:
  kubernetes.io/arch: arm64

Publishing upstream over Sparkplug

An edge Gateway can also publish everything it reads to a central one, as a Sparkplug B edge node. See EDGE-PUBLISH.md for what it publishes and how the forward queue behaves; this is the chart side.

edgePublish:
  enabled: true
  broker: tcp://central.example.local:1883
  group: Plant1
  node: press-line-3          # empty uses the release name
  username: press-line-3
  existingSecret: central-mqtt # key "password", or set secretKey
  tls: false
  queueMaxBytes: 536870912     # 512 MiB; the default is 64 MiB, and 0 is unbounded
  persistence:
    size: 1Gi

What turning it on changes, and why:

• A 1Gi PVC for the forward queue, mounted at /data/edge-publish. That’s where readings pile up while the uplink is down, so it has to outlive the pod. persistence.enabled: false puts it on an emptyDir instead, which survives a container restart and nothing else.
• The Deployment rolls with Recreate. The queue is one file with a lock on it. A rolling update starts the new pod first, the new pod can’t open the queue, it never goes ready, and the rollout sits there forever. Recreate stops the old one first; you lose a few seconds of polling per upgrade.
• replicaCount has to be 1, and the chart refuses to render otherwise. Two replicas would be two publishers claiming one Sparkplug node.
• The node id defaults to the release name, not the pod’s host name. The host name changes on every rollout, and a Sparkplug host would see a brand new edge node each time.

The password never goes in a plaintext default. Leave password and existingSecret empty for an anonymous broker and no Secret is created; set password and the chart puts it in <release>-edge-publish; set existingSecret to use one you manage.

External database

database:
  postgresql:
    enabled: false
    auth:
      existingSecret: my-db-credentials
      secretKey: password
  external:
    host: timescale.plant.local
    port: 5432
    database: industrial
    username: industrial
  sslMode: require

The app detects TimescaleDB and promotes tag_history to a hypertable when the extension is present. Against a plain PostgreSQL it falls back to native range partitioning instead of failing.

The decision is recorded and sticky. A database created as a hypertable refuses to start against a server without the extension rather than silently writing somewhere else, if you see that error, you have pointed the app at a different database than it was built on.

SQLite

The database is a setting, not an image. Every product runs on SQLite, Postgres or TimescaleDB out of the same binary (IoT skips TimescaleDB, see below):

Variable	Default	What it does
HOTLOOP_DB_TYPE	postgres	postgres or sqlite. TimescaleDB is postgres with the extension on the server
HOTLOOP_DB_PATH	/data/hotloop.db	The SQLite file. Only SQLite reads it

SQLite is the pure Go one (modernc.org/sqlite), compiled into every image, so there’s no second container, no 300 MB Postgres image on an SD card, and the arm builds stay static. Put the file on a volume that outlives the container, and on a Pi put that volume on a USB SSD. A historian writing to an SD card all day is how you buy a lot of SD cards.

What’s the same: every table, every column, every feature. What’s different:

• History is one table with an index, not partitions or a hypertable, and retention deletes old rows instead of dropping months. A site with years of history at one second wants Postgres or TimescaleDB, and moving there is a backup and a restore (BACKUP.md).
• One writer at a time. SQLite takes turns on writes and reads alongside them (WAL), which is plenty for one HotLoop and an --mcp-stdio helper beside it on the same file. It isn’t for two replicas on network storage: SQLite on NFS corrupts, so a SQLite deployment is one pod with a local volume.
• Commits are synchronous=FULL: when the API says it saved, it’s on disk, power cut or not.

HotLoop IoT defaults to 30 days of history (HOTLOOP_HISTORIAN_RETENTION_DAYS still wins when it’s set) and never builds a hypertable: TimescaleDB is part of Edge and Gateway. Set HOTLOOP_HISTORIAN_TIMESCALE=true on IoT and it says so at start and runs on plain partitions. A database moved down from an Edge or a Gateway that already has a hypertable keeps it and keeps working.

Picking the database in the chart

One value, database.type, and each product takes the ones it runs on:

database.type	What gets deployed	IoT	Edge	Gateway
timescaledb	The app and TimescaleDB (a StatefulSet and its volume). History in a hypertable	refused	default	default
postgresql	The app and plain PostgreSQL 16 (a StatefulSet and its volume). History in native partitions	default	yes	yes
sqlite	The app alone, with the database file on a volume of its own	yes	yes	yes

helm install hotloop hotloop/hotloop-iot --namespace hotloop --create-namespace   --set database.type=sqlite

Asking IoT for timescaledb fails the install and says what IoT runs on (database.type is "timescaledb", and HotLoop IoT runs on sqlite, postgresql), and the form in the App Store doesn’t offer it. helm lint does not catch it (helm 3.22 lints it clean), so the render is what refuses. Leaving database.type out is the product’s own default, so an upgrade with --reuse-values from before the key existed keeps the database it had.

sqlite renders:

• HOTLOOP_DB_TYPE=sqlite and HOTLOOP_DB_PATH=/data/db/hotloop.db, and none of the Postgres connection settings. No database pod, no database password in the Secret, no init container waiting for one.
• A PersistentVolumeClaim named <release>-db (database.sqlite.persistence: size, storageClass, accessMode, 10Gi and the cluster default to start), mounted at /data/db. The directory, not the file: SQLite keeps a -wal and a -shm beside the database. Put it on local storage; SQLite on NFS corrupts.
• The Deployment on Recreate, not a rolling update, because the volume is ReadWriteOnce and the file has one writer. replicaCount above 1 fails the install.

The NetworkPolicy is unchanged and on: the app’s own policy still applies, and the database policy is only rendered when there is a database pod to protect.

Changing database.type on a release that already has data does not move the data. The new database starts empty, so moving a plant between databases is a backup and a restore (BACKUP.md), the same as with the variable. In particular a database that was made as a TimescaleDB hypertable refuses to start against plain PostgreSQL, so do not flip an existing Edge or Gateway from timescaledb to postgresql and expect it to come up.

Picking the database in Quadlet

Each product’s directory under deploy/quadlet/ has the PostgreSQL or TimescaleDB files as they were, and a sqlite/ directory beside them: hotloop.container and hotloop-data.volume, the app alone with the file at /data/hotloop.db on the hotloop-data volume. It follows :stable and rolls back a bad update the same way. Install those two files instead of the others (they use the same unit names), and make two secrets rather than three: there is no database password. deploy/quadlet/README.md has the steps.

For plain Postgres partitions on Edge or the Gateway under Quadlet, keep the TimescaleDB unit and add Environment=HOTLOOP_HISTORIAN_TIMESCALE=false to hotloop.container. IoT’s unit already runs plain PostgreSQL.

The device credential key

secretKey:
  enabled: true
  existingSecret: ""     # or a Secret you manage, with the key under secret.key

Device passwords (a UniFi console login, an OPC UA or MQTT password, HTTP API keys) are encrypted in the database with this key, so a dump or a backup holds ciphertext. Out of the box the chart generates one into <release>-secret-key on the first install, keeps it across upgrades, and keeps it on helm uninstall too (helm.sh/resource-policy: keep), because a database kept on its volume is no use for its credentials without it. It’s mounted as a file and passed as HOTLOOP_SECRET_KEY_FILE, never as an environment variable.

Back the Secret up separately from the database backups: a key sitting next to the backups it protects protects nothing. To bring your own, hotloop gen-secret-key prints one. Outside Kubernetes, write it to a file only HotLoop can read and point HOTLOOP_SECRET_KEY_FILE at it. Without a key HotLoop runs, says in its startup log that device passwords are stored as typed, and refuses to save a UniFi console. Credentials already stored are encrypted the first time it starts with a key. See BACKUP.md for what that means for a restore.

Configuration, and what the Settings screen can change

Almost every setting comes from an environment variable that the chart sets from a Helm value. That is the right place for it: configuration deciding whether this process can command machinery belongs in the deployment, reviewed, not in a form on a web page.

The Settings screen shows all of it, every setting, its effective value, whether it came from the database, an environment variable or the built-in default, and the variable that changes it. Most of it is read-only, which is the point: the question it answers is “why is it doing that, and what do I change?”

Four settings are plant identity rather than behavior, and are editable there. They are stored in the database and take effect immediately, because a plant is renamed far more often than it is redeployed:

setting	variable it overrides
Plant name	HOTLOOP_PLANT_NAME
Site ID	HOTLOOP_SITE_ID
Timezone	HOTLOOP_TIMEZONE
Confirm before writing	HOTLOOP_CONFIRM_WRITES

A stored value wins over the environment variable, so the chart’s value becomes the starting point rather than the last word. A timezone the machine cannot resolve is refused rather than accepted and quietly ignored, which would leave every timestamp falling back to UTC with nothing saying why.

The same goes for the deployment’s own value. If HOTLOOP_TIMEZONE (or TZ, when that’s unset) names a zone that won’t load, HotLoop refuses to start and the error names the variable and the value. It used to shrug and run on UTC. Reports, schedules and shift times all run on the plant’s clock, and a guess hours off is worse than a pod that says why it won’t come up.

The site’s location is HOTLOOP_SITE_LATITUDE and HOTLOOP_SITE_LONGITUDE (chart: plant.latitude, plant.longitude), decimal degrees, north and east positive. Sunrise, sunset and the sun’s entities and triggers are worked out from them on the box (SUN.md). Both or neither, and a bad one stops startup with the variable named. The Settings screen shows them read-only: rules fire on a sunrise, so moving it is a deploy.

The safety switches are not editable and will not be. HOTLOOP_ALLOW_WRITES is the master control over whether this process can command equipment; a control that can be switched back on from the same browser session that issues the writes is not a control. HOTLOOP_HISTORIAN_RETENTION_DAYS is likewise deployment-only, the retention policy is applied to the database at startup, so a field on a screen would look like it worked and move no data. The endpoint refuses any key outside the editable set by name, rather than ignoring it.

One limitation, stated because the screen states it too: the MCP server builds its instructions at startup, so it advertises the plant name it booted with until the pod restarts. Everything else picks a rename up on the next request.

Upgrading from 3.x

This is a replacement, not an upgrade. 3.x was a different product. There is no migration path for its data, because it shares no data model with a tag historian.

helm uninstall hotloop -n hotloop
# keep or delete the old PVCs deliberately, they hold the HA recorder database
helm install hotloop ./charts/hotloop -n hotloop

Things that no longer exist and will be rejected if set: sidecarProxy, homeassistant, mqtt/nodered/esphome/zigbee2mqtt add-ons, devices, cameras, customConfig, network.hostNetwork.

Rolling back an update

helm rollback hotloop -n hotloop puts the previous release’s image back. It does not put the database back: a release migrates on start, and that stays done. This is safe because migrations only add (a table, a column with a default, an index, never a drop, a rename or a type change), so the previous release works on the schema the newer one left, and test/prevrelease.sh proves it on the real previous image before each release ships. The rule and the check are in MIGRATIONS.md.

It is safe for one release back. The database records which release applied each migration, and a release that finds migrations from two or more releases newer than itself refuses to start:

refusing to start: this database's schema is more than one release ahead of
HotLoop 4.18.10. ...

That is not a fault to work around. It means the database has moved further than this release was ever run against. Roll forward to a release that knows the schema, which is the newest one, or restore a backup taken by the release you are on (BACKUP.md). Exactly one release back starts, with a warning in the log naming both versions. Take a backup before you upgrade; the rollback can’t undo data the newer release wrote.

Verifying an install

kubectl -n hotloop get pods
kubectl -n hotloop logs deploy/hotloop | head -20

A healthy start says the protocols it speaks, applies the migrations, reports the history mode, and warns about anything worth knowing:

msg=starting version=4.0.0 protocols="[ethernet-ip http modbus mqtt mtconnect opcua s7comm]"
msg="writes are ENABLED: the UI, API, automations and MCP can all command devices"
msg="tag_history promoted to a hypertable" timescale=2.30.1
msg="history schema ready" mode=timescale
msg=listening addr=:8080 mcp=/mcp writes=true

Then check the labels the dashboard needs. This is the check that catches drift, assert the set, not the count, because a count check passes while the set is wrong:

kubectl -n hotloop get svc <release> -o json \
  | jq -r '.metadata.labels | keys[] | select(startswith("embernet.ai/"))'

Exactly four, on the Service and on the pod:

embernet.ai/app-name
embernet.ai/gui-port
embernet.ai/gui-type
embernet.ai/store-app

If the app is running but missing from Running Apps, the Service labels are the first place to look, the dashboard queries Services by embernet.ai/store-app.

A deploy from the App Store adds five more, from the tenantLabels the dashboard passes: embernet.ai/tenant, app-id, source, deployed-by and deployment-id. They go on every resource the chart renders and on the gateway pod. The dashboard reads the tenant off the Service and nowhere else, so a store-deployed Gateway without embernet.ai/tenant on its Service belongs to nobody. A plain helm install renders none of the five.

Probes

• /healthz, liveness. Answers while the process is up.
• /readyz, readiness. Additionally requires the database, so a pod that has lost its historian stops taking traffic rather than serving a UI that cannot answer anything.

The startup probe allows two minutes, because a first install runs migrations and the liveness probe must not kill it mid-migration.

Shutdown

terminationGracePeriodSeconds: 45. On SIGTERM the app stops accepting requests, then drains the historian queue so readings already taken are written rather than thrown away. Cutting the grace period too short loses whatever was in flight.

Troubleshooting

A device sits in connecting. Look at lastError on the device card. Most often a firewall between the pod and the equipment, or the wrong port. Egress is unrestricted by the chart, so a NetworkPolicy blocking it is something else on the cluster.

Tags read bad quality. The device answered but the point did not. For Modbus, check the address base, oneBased if your drawings use 40001 numbering. For OPC UA, check the node id resolved; prefer the nsu= form.

Values look plausible but wrong. On Modbus, that is word order, try CDAB. This is the failure that does not announce itself, because the number is not obviously wrong.

Nothing is being historised. Check the Protocols screen diagnostics. filtered counts readings suppressed by deadband, which is working as intended; dropped counts readings lost because the database could not keep up, which is not.

MCP returns 401. The bearer token. Read it from the release Secret.

Every /api/ request returns 401. Expected once auth.enabled is true and nobody has logged in yet, that is the point of 4.4.0. Read the generated admin password from the release Secret (see Auth) and log in through the UI, or POST /api/auth/login directly. See docs/AUTH.md.

Lost the admin password and there is no other admin account. Generate a new hash and patch the Secret, then restart the pod so it re-reads the environment, EnsureAdminUser will not touch an account that already exists, so this replaces the row directly rather than going through the chart:

hotloop hash-password -password 'a-new-one'
kubectl exec -n hotloop deploy/hotloop -- \
  psql "$DATABASE_URL" -c \
  "UPDATE users SET password_hash = '<the hash>' WHERE username = 'admin'"

An automation is enabled but not running. It failed to compile. The Automations screen shows it as NOT ARMED with the reason, and the system log has the error.