Metrics

Prometheus at /metrics, per node, with node and type labels on everything per-node. No exporter sidecar, because an edge box does not have room for one and you should not need a second container to find out that your inbox is full.

Metric	Type	What it tells you
hotloop_flow_build_info	gauge	Always 1. The version rides in the label.
hotloop_flow_uptime_seconds	gauge	Seconds since the runtime started.
hotloop_flow_nodes_running	gauge	Node instances currently running.
hotloop_flow_node_messages_received_total	counter	Messages delivered to a node.
hotloop_flow_node_messages_sent_total	counter	Messages a node has emitted.
hotloop_flow_node_errors_total	counter	Errors a node raised.
hotloop_flow_node_messages_dropped_total	counter	Messages discarded because an inbox was full.
hotloop_flow_node_sends_blocked_total	counter	Times a sender waited for space. Sustained back-pressure, which is the real signal that a flow cannot keep up.
hotloop_flow_node_queue_length	gauge	Messages waiting right now.
hotloop_flow_node_queue_capacity	gauge	Where the overflow policy starts applying.
hotloop_flow_node_queue_high_water	gauge	The deepest that inbox has ever been.
hotloop_flow_goroutines	gauge	Roughly one per node plus the I/O each holds.
hotloop_flow_memory_heap_bytes	gauge	Heap currently allocated.
hotloop_flow_memory_sys_bytes	gauge	Bytes taken from the OS.
hotloop_flow_gc_cycles_total	counter	Completed GC cycles.
hotloop_flow_git_mirror_behind_deployments	gauge	Deployments the git mirror’s remote doesn’t have yet. Only with a mirror.
hotloop_flow_git_mirror_push_failures_total	counter	Pushes to the mirror that failed and will be retried.
hotloop_flow_git_mirror_pushes_total	counter	Pushes to the mirror that succeeded.
hotloop_flow_durable_wire_depth	gauge	Messages waiting on a durable wire, on disk. Label wire.
hotloop_flow_durable_wire_bytes	gauge	Quota charged to that wire’s waiting messages.
hotloop_flow_durable_wire_delivered_total	counter	Messages the wire handed over and the target took.
hotloop_flow_durable_wire_retries_total	counter	Times the wire sent a message again because its target failed it. A target that is down climbs this.
hotloop_flow_durable_wire_given_up_total	counter	Messages the wire stopped retrying because the node’s ew_durable_attempts ran out. Each went to the flow as an error with the message attached.
hotloop_flow_durable_wire_dropped_total	counter	Messages the quota policy refused or evicted, and queues purged when a wire stopped being durable.
hotloop_flow_durable_bytes, hotloop_flow_durable_max_bytes	gauge	Quota used by every durable wire together, and the quota.
hotloop_flow_context_flushes_total, hotloop_flow_context_rows_written_total	counter	Transactions that wrote context, and rows they actually changed.
hotloop_flow_context_flush_failures_total	counter	Context flushes that failed. The changes are kept and tried again. Should be zero.
hotloop_flow_audit_write_failures_total	counter	Audit entries that couldn’t be written. Should be zero forever.
hotloop_flow_traces_kept	gauge	Message traces in the buffer right now.
hotloop_flow_trace_bytes	gauge	Roughly what they hold. Bounded by trace.maxBytes.
hotloop_flow_traces_started_total	counter	Message traces started.
hotloop_flow_traces_forgotten_total	counter	Traces pushed out to make room.
hotloop_flow_traces_skipped_total	counter	Messages not traced because trace.perSecond was reached.
hotloop_flow_trace_hops_dropped_total	counter	Hops not written down: their trace was full or already forgotten.
hotloop_flow_otlp_spans_exported_total	counter	Spans the OpenTelemetry collector accepted. Only with an endpoint.
hotloop_flow_otlp_spans_dropped_total	counter	Spans dropped because the export queue was full.
hotloop_flow_otlp_spans_failed_total	counter	Spans the collector didn’t take after a retry. Above zero means it isn’t getting them.

The one to alert on is hotloop_flow_node_queue_high_water against hotloop_flow_node_queue_capacity. High water is the early warning that a flow is approaching its ceiling, which arrives before anything is dropped and long before anyone is awake. That alert is the whole reason I built the bounded inbox. You can’t report how close you are to a limit that doesn’t exist, so the limit had to exist first.