The logbook

One timeline of what happened on the plant, newest first: the pump went on, somebody wrote 180 to the zone 1 setpoint, the high temperature alarm was raised and acknowledged, the cooldown automation ran, a device was reconfigured. For the whole plant, one place on the equipment tree and everything under it, or one entity. It is the Logbook tab, a section of every entity’s drawer, GET /api/logbook (see API.md) and the get_logbook MCP tool (see MCP.md).

The first question after anything goes wrong is “what happened just before?”, and until now the answer was in five places: the write audit, the alarm history, the automation runs, the system log, and nobody’s memory of what the equipment itself did. The logbook is those five in one list.

What is in it

Kind	From	What a line says
state	entity_state_log	an entity’s state changed: changed to on (was off)
write	write_audit	a write to a tag, written, refused or failed, by whom and through which door
alarm	alarm_events	an alarm raised, acknowledged, cleared, shelved, with the note given
automation	automation_runs	a rule ran, how it ended, what started it
event	events	configuration changes, service calls’ summaries and the reasons given for writes, and what a UniFi console reports the moment it happens, filed under the switch or access point it is about (a switch losing contact is a warning)
script	script_runs	a script ran, how it ended, who started it and what called it
recipe	recipe_applications	a recipe was applied, or refused: which version, by whom, how many targets went out and were read back

Only state is new. The others are read where they already are, at query time, and merged. Nothing is written twice, so nothing can disagree with the trail it came from, and each trail keeps its own retention (below).

From the system log, the logbook takes the config category and the write category’s own lines: the summary of a service call, an alarm service call, the reason given for a write. The write gate’s own “wrote …” and “write of … refused” lines are left out, because they say exactly what the write audit already holds and every write would otherwise show twice.

What is deliberately not in it

Numeric readings. A sensor, a number, a loop or a PLC counter changes every poll, and every reading of it is already in the historian. Logging them again would bury every line anybody wants to read under a million a day that nobody does. Trend them instead.

Buttons. A button’s state is the time of its last press, and every press is already a write in the audit. It also resets to unknown at every restart, which would be a line per button per restart saying nothing.

Alarm entities’ states. The alarm’s own history is in the logbook as alarm lines, with more in them (the value, the note, who acknowledged).

Everything else is recorded: binary sensors, switches, text, selects, motors, valves, helpers, and scripts as they arrive.

Helpers are logged by whoever changed them. A helper’s change is written to the logbook in the same transaction as its value, by the process that made the change, with who did it, from where and under which call id, and never rate-limited. The recorder leaves helpers alone: it only runs in the Gateway, so an agent’s change through --mcp-stdio would be missed, and every process holding the helper sees the change, so it would be logged once per process. The schedule is the exception, because nobody sets it and the clock moves it: the recorder logs it like any other entity.

Chatter

A contact bouncing at 10 Hz is one fact (“it chattered, this many times”), not thirty-six thousand lines an hour. Each entity gets ten changes straight through, then one every two seconds. A change past that is not logged, and is counted on the entity’s next line instead:

changed to closed (was open); 14 changes before this not logged, it was changing faster than the logbook records

The count is exact: the lines plus their counts are every change the entity made. When the chatter stops, the entity’s final state is always logged, within two seconds, so the logbook never ends on a state the equipment has left. The number suppressed since start is suppressed in /api/diagnostics under logbook.

last_changed survives a restart

An entity’s state is computed from the live tags, never stored, so before the logbook every restart made every entity look as though it had just changed. The logbook keeps every tracked entity’s newest state and when it last changed (entity_last_state), and the entity engine reads it when it starts. An entity whose state comes back the same keeps its last_changed.

A restart looks like this from inside: a device is not connected yet, so its entities go unavailable, then unknown, then show their reading. Those steps are this process catching up, not the plant changing, so they are waited through. A state that really did change while the Gateway was down gets a fresh last_changed and a line saying what it was before. For the same reason, the logbook holds back a change into unavailable or unknown for the first minute after a start: a device still down after that is logged then, at the time it was first seen down.

Filters

• Where: a node of the equipment tree and everything under it. A place is turned into every name the trails know it by: the entities there (by their own place, else their device’s, the same rule the Entities screen uses), the devices there and every tag on them, the alarms on those tags, the automations with a trigger on any of it, and the node itself for its configuration changes. An automation run is also in a place if it wrote there while it ran, whatever triggered it. A UniFi console’s events are found by the network, not by anything a person placed on the tree: the switch a linked device hangs off, and every switch between it and the gateway, so a switch dropping shows under every line it feeds. A console placed on the tree finds the events about everything it adopted. See UNIFI.md.
• Entity: the same, for one entity: its state changes, writes to it and its tags, its alarms, the rules it triggers or that wrote to it, and config changes to it or its tags. An entity that has since been deleted is still found by the name it had.
• Who: part of the actor, ignoring case: alice, automation:cooldown (the rule’s runs and every write it made), mcp: (everything any agent did). A state change the equipment made on its own has no actor.
• Kinds: any of the five.
• When: from and to.

Paging is by a cursor, the time, kind and id of the last line, so a page is never shifted by lines added since the one before it.

The screen updates live. The live feed looks at the trails once a second while somebody is watching, and sends a logbook event naming the kinds that have something new; the screen then asks for its page again. It watches the tables, not this process, so a write an agent makes through --mcp-stdio, which is another process, shows up all the same.

How long it keeps

entity_state_log is partitioned by month, in both history modes (it is a plain PostgreSQL partitioned table whether or not tag_history is a hypertable), and a month is dropped whole once every line in it is older than HOTLOOP_LOGBOOK_RETENTION_DAYS. Default 90. Dropping a month is instant; deleting its rows one by one is not, which is why it is partitioned. So a line is kept between 90 days and 90 days plus a month. 0 keeps everything, and the Gateway says so at startup, because the table then grows until the volume fills.

The logbook’s own maintenance does this, once at start and daily after. It makes last month’s partition and the next three months’ ahead of time, so a line always has one to land in; a line for a month with no partition lands in the default partition, and the maintenance moves those into their month when it makes it (which is what happens after a restore, see BACKUP.md) or deletes them once they are past the retention.

The other trails keep what they always kept:

Trail	Kept
write_audit	everything, until somebody trims it
alarm_events	everything, until somebody trims it
automation_runs	the newest 500 runs of each automation
events	the newest 100,000 lines

So on a busy plant, the logbook’s writes and alarms can go back further than its state changes, and its automation runs and config changes less far. The logbook shows what the trails have.

One process records

The Gateway records. A second process started with --mcp-stdio (an agent’s helper beside the Gateway) reads the logbook and its baseline, so what it says about last_changed agrees with the Gateway’s, but it writes nothing and maintains nothing: two recorders would log every change twice.

The recorder is a hook on the entity engine that only queues. Once a second, one goroutine turns what is queued into lines, applies the rate limit, and writes them with one COPY and one upsert of the newest states, in one transaction. The queue holds 50,000 changes; past that the oldest are dropped and counted. A database that is down keeps the lines queued, up to the same limit, and they go in when it is back. /api/diagnostics shows it all under logbook:

{"recording":true,"retentionDays":90,"queued":0,"dropped":0,"written":212,"suppressed":36,"writeErrors":0}

dropped above zero means changes are missing from the logbook; queued that keeps growing means the database is not keeping up; lastError says why the last write failed.

Backups

Both tables are in every backup: entity_last_state after the entity tables, entity_state_log just before tag_history. See BACKUP.md.