Alarm notifications

Added in 4.10.0. internal/notify is the first increment of Phase 2 of the plan of record, whose own acceptance bar for this phase is literally “a 03:00 alarm pages a phone.” It registers with alarm.Engine’s existing OnTransition hook and, when an alarm is raised, sends a notification through every configured, enabled channel whose priority floor the alarm meets. Off by default (HOTLOOP_NOTIFY_ENABLED): a deployment with nobody to page has nothing to configure this against.

Why only “raised”

alarm.Engine.OnTransition fires on every ISA-18.2 transition, raised, cleared, acked, shelved, unshelved. This package listens to all of them but acts on exactly one. Paging on every transition would mean an operator’s phone buzzing when they acknowledge their own alarm, when it clears on its own, when someone shelves it for a shift, training them to swipe pages away unread the same way a chattering alarm trains them to ignore alarms. Raised is the one event that means “something needs a human who might not already be looking at a screen,” which is the entire reason this package exists.

One case that looks like an unshelve is really a raise. If the condition comes up while an alarm is shelved, nobody has been told, so when the shelve ends (timed out or unshelved by hand) the engine records the unshelve and then a raise, and the raise pages. An alarm that was already in alarm when it was shelved and still is comes back unacknowledged but doesn’t page again: it paged the first time.

The channels

Six, and that’s the list (see “Notifications” in ROADMAP.md for what we’re not doing and why). Every one needs no vendor SDK and no paid account, matching this project’s plan of record: cloud integrations are opt-in outbound connections an operator configures, never something this app requires to run.

• ntfy: a push to a phone, with an Acknowledge button that acks the alarm straight back into HotLoop. Free, and you can run the server on the plant network with no internet at all. See “Setting up ntfy”.

• Gotify: a push to a phone. Free, self-hosted, works with no internet. No buttons and no iPhone app, so it’s a pager, not a two-way. See “Setting up Gotify”.

• Discord webhook: an embed in a Discord channel, colored by priority, with the tag, value, limit, area and line. Paste a URL and it works. Critical alarms can ping a role. No buttons. See “Discord”.

• Discord bot: the same message, with Acknowledge and Shelve buttons that act on the alarm and record who pressed them. The presses come back over a connection HotLoop opens to Discord, so the plant needs no public address. See “Discord”.

• Webhook: a POST of HotLoop’s own JSON body (alarmId, name, priority, priorityLabel, message, deviceId, tagName, raisedAt, event) to a configured URL. It works with anything that takes whatever JSON you send it: your own script behind a small HTTP listener, Node-RED, n8n, or a “catch any webhook” trigger in a workflow tool. (Pointed at an ntfy topic it pages, with the raw JSON as the text; use the ntfy channel above instead.) A summary (see “Rate limits and summaries”) has event summary and an alarms list naming every alarm in it. It does not work with services that want the body in their own shape. Discord answers 400 because there’s no content or embeds. Slack, Teams and Google Chat want their own text field. Pushover wants its token, user and message fields, and PagerDuty’s Events API wants a routing_key, an event_action and a payload. Point the webhook at one of those and every page fails. Discord has channels of its own that build the body Discord expects; use those.

• Email over SMTP: sent through whatever relay an operator already has (a corporate mail server, a transactional-email provider’s SMTP endpoint, a self-hosted relay). No API key, no vendor library: net/smtp and a hand-built plain-text message. Many phones already surface email as a push notification on their own, and many paging tools accept an email-to-SMS bridge address as a “contact method”, so this one channel alone covers a meaningful slice of “pages a phone” with zero paid dependency.

The webhook URL is never printed

For a lot of receivers the URL is the password. Discord puts the token in the path (/api/webhooks/{id}/{token}), ntfy takes it as ?auth=, Google Chat as ?key= and ?token=. Anyone holding the URL can post to your channel. So when a send fails, the error names the scheme and host and nothing else:

webhook request to https://discord.com failed: context deadline exceeded
webhook https://ntfy.example.com returned 401 Unauthorized

That’s the same text in the log, in the audit event and in the POST /api/notify/test response. The host is enough to tell which receiver broke; the path and the query never leave the process. A URL that fails validation at startup isn’t repeated in that error either. If you need to see the URL, look at HOTLOOP_NOTIFY_WEBHOOK_URL where you set it.

Setting up ntfy

ntfy is a publish/subscribe push server. HotLoop publishes each alarm to a topic; the ntfy app on the phone subscribes to it.

1. Run a server, or use ntfy.sh. On a plant network, run your own (binwiederhier/ntfy, one container) and lock it down: set auth-default-access: deny-all, create a user for HotLoop that may write the topic, and one per person that may read it. On ntfy.sh, or any server anyone can reach, the topic name is the password: anybody who guesses it reads your alarms. Make it long and random.
2. Give HotLoop an access token (ntfy token add hotloop) or a username and password for that user.
3. Set HOTLOOP_PUBLIC_URL to the address a phone reaches HotLoop at, for example https://hotloop.plant.example. Without it the notification still arrives, but with no Acknowledge button and no tap-to-open, because the phone would have nowhere to send either.
4. Configure it and restart:

HOTLOOP_NOTIFY_ENABLED=true
HOTLOOP_NOTIFY_NTFY_ENABLED=true
HOTLOOP_NOTIFY_NTFY_URL=https://ntfy.plant.example
HOTLOOP_NOTIFY_NTFY_TOPIC=boiler-house-alarms
HOTLOOP_NOTIFY_NTFY_TOKEN=tk_...
HOTLOOP_PUBLIC_URL=https://hotloop.plant.example

5. POST /api/notify/test sends a test notification. It has no Acknowledge button, because there’s no alarm behind it to acknowledge.

The URL is the server alone. The topic goes in its own setting, never in the URL: that way HotLoop knows which part is the secret and keeps it out of every error, log line and audit event.

What arrives:

Details	Details
Title	The alarm’s name
Body	The alarm’s message, its priority, where it came from (device/tag) and when it was raised
Priority	Critical 5 (max: on Android it can break through do-not-disturb, if you let it), High 4, Medium 3, Low 2
Tags	An emoji by severity and the priority as a word, so the list on the phone can be scanned
Tap	Opens HotLoop’s alarm list, when HOTLOOP_PUBLIC_URL is set
Button	Acknowledge, when HOTLOOP_PUBLIC_URL is set. See “Acknowledge links”

The iPhone caveat

Android works with no internet at all: the ntfy app keeps its own connection to your server and gets every message instantly over the plant Wi-Fi.

iPhones don’t. Apple only lets a push through its own servers (APNs), and only ntfy.sh can talk to APNs. A self-hosted server has to be configured with upstream-base-url: https://ntfy.sh, and then it forwards a poll request (no message content, just “something arrived on this topic”) to ntfy.sh, which wakes the phone, which then fetches the message from your server. So on a plant with no internet, iPhones do not get instant notifications from a self-hosted ntfy. They see them when the app is opened. If your people carry iPhones and the plant is air-gapped, ntfy is not your pager; email through a relay is, or Discord if the plant can reach it. Gotify has no iPhone app at all.

The Acknowledge button is a request from the phone to HOTLOOP_PUBLIC_URL, so the phone has to be able to reach that address: on the plant Wi-Fi, or through whatever VPN you use. It works the same on either platform.

Setting up Gotify

Gotify is a small self-hosted push server with an Android app. No internet needed, no account anywhere.

1. Run gotify/server and log in to its web UI.
2. Under Apps, create an application called HotLoop. Copy its token.
3. Configure it and restart:

HOTLOOP_NOTIFY_ENABLED=true
HOTLOOP_NOTIFY_GOTIFY_ENABLED=true
HOTLOOP_NOTIFY_GOTIFY_URL=https://gotify.plant.example
HOTLOOP_NOTIFY_GOTIFY_TOKEN=A...
HOTLOOP_PUBLIC_URL=https://hotloop.plant.example

The token goes in the X-Gotify-Key header, never as ?token= in the URL, where it would end up in every proxy’s access log.

Priorities map onto what the Android app does with them: Critical 10 and High 8 are high importance (sound and a heads-up pop-up), Medium 5 is a normal notification with sound, Low 2 is silent. The message is Markdown (the alarm’s name bold, the details as a list, the alarm’s own text escaped so a name with an asterisk in it reads as that name), and tapping it opens HotLoop’s alarm list when HOTLOOP_PUBLIC_URL is set.

No acknowledge. Gotify notifications have no action buttons, so there’s no way to put an Acknowledge button on one, and we won’t fake it with a link that acknowledges on a plain GET: link previewers and security scanners open links on their own, and an alarm must never be acknowledged by a robot. Acknowledge in HotLoop. No iPhone app either.

Discord

Two ways in. The webhook is a pager: paste a URL, get alarms in a channel. The bot is a two-way: the same messages with Acknowledge and Shelve buttons, and the audit trail says which person pressed them. Both need the plant to reach Discord over the internet (outbound HTTPS to discord.com, and for the bot a WebSocket to *.discord.gg). Neither needs Discord to reach the plant.

What arrives

Details	Details
Text	[Critical] Boiler Trip, which is what a phone’s push notification shows. On a Critical alarm, a ping of the configured role in front of it
Embed title	The same, linking to HotLoop’s alarm list when HOTLOOP_PUBLIC_URL is set
Color	Critical red, High orange, Medium yellow, Low blue
Description	The alarm’s message
Fields	Tag (device/tag), Value (the reading that raised it), Limit (hi 90, from the definition), Area / Line (from the device), Raised (in each reader’s own time zone)
Footer	`HotLoop
Buttons (bot only)	Acknowledge, Shelve 1h, Open in HotLoop

A flood becomes a summary, the same as every other channel: the first five go out, then one message listing the rest, as many names as fit and a count of the others, so the count is never cut off.

Discord refuses a message that’s too long (2000 characters of text, 256 of title, 4096 of description, 25 fields, 6000 in all), and a refused message is a page that didn’t go out. So HotLoop cuts to fit, on a character boundary, with ... where it cut.

Nobody gets pinged by accident

Every message goes out with allowed_mentions: {"parse": []}. Without it Discord pings whatever the text mentions, so an alarm somebody named “@everyone tank overflow” would wake the whole server. With it, nothing in the text pings anyone. The one exception is the role you configure for Critical (..._CRITICAL_ROLE), which is named explicitly as the only mention allowed, and only on a Critical alarm or a summary with a Critical alarm in it. A test notification never pings. The alarm’s own text is Markdown-escaped too, so a name with * in it reads as that name and a <@&role> in a name is shown, not used.

For the ping to work the role has to be mentionable: in the server’s role settings turn on Allow anyone to @mention this role, or give the bot the Mention @everyone, @here, and All Roles permission.

Discord’s rate limits and bans

Discord answers “too fast” with a 429 and says how long to wait, in the body (retry_after, to the millisecond) and in X-RateLimit-Reset-After. That’s how long HotLoop waits before the retry, inside the usual three attempts and 30 seconds (see “Retries”). When Discord says the budget is spent before it refuses (X-RateLimit-Remaining: 0), the next send waits for the reset instead of drawing a 429.

Discord bans the address of anything that racks up 10,000 refused requests (401, 403, 429) in ten minutes, and the ban takes every Discord integration on that address with it. So when Discord answers 401, 403 or 404 (a wrong or deleted webhook, a bad bot token, a channel the bot can’t see), HotLoop switches that channel off after that one request. It logs it once, every alarm after that is refused without a request (and audited as not sent, so “was anybody told” still has an answer), and it shows in GET /api/diagnostics under notify.channels, with the reason and when:

{"channel": "discord-webhook", "broken": true,
 "reason": "discord webhook https://discord.com returned 404 Not Found: Unknown Webhook (Discord error 10015)",
 "since": "2026-09-27T03:00:01Z"}

Fix the setting and restart HotLoop. A 500 or no answer at all doesn’t switch anything off; those are retried like any other channel’s.

Setting up the webhook

1. In Discord, open the channel’s settings (the gear), Integrations, Webhooks, New Webhook. Name it HotLoop. Copy Webhook URL.
2. Optional: a role to ping on Critical. Turn on Developer Mode (User Settings, Advanced), then right-click the role in Server Settings, Roles, and Copy Role ID. It’s a number, not a name.
3. Configure it and restart:

HOTLOOP_NOTIFY_ENABLED=true
HOTLOOP_NOTIFY_DISCORD_WEBHOOK_ENABLED=true
HOTLOOP_NOTIFY_DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/1234.../AbCd...
HOTLOOP_NOTIFY_DISCORD_WEBHOOK_CRITICAL_ROLE=123456789012345678
HOTLOOP_PUBLIC_URL=https://hotloop.plant.example

4. POST /api/notify/test sends a test message.

The whole URL is the password. Anyone holding it can post to your channel as the webhook. HotLoop never prints it: not in the log, not in an error, not in an audit event, not in an API response, and a URL that fails validation at startup isn’t repeated in that error either. In the Helm chart it goes into a Secret. If it leaks, delete the webhook in Discord and make a new one.

A webhook can’t carry a button that does anything, so there’s no Acknowledge here. The title links to the alarm list in HotLoop. For buttons, use the bot.

Setting up the bot

1. Go to the Discord developer portal, New Application, call it HotLoop. The Application ID on the General Information page is what the invite link needs.
2. Bot tab: Reset Token and copy it. Discord shows it once. That’s HOTLOOP_NOTIFY_DISCORD_BOT_TOKEN. On the same page turn Public Bot off, so nobody but you can add it to a server, and leave all three Privileged Gateway Intents off. HotLoop asks Discord for no events at all (intents 0); button presses arrive regardless.
3. Invite it to your server with the least it needs, View Channels, Send Messages and Embed Links (permissions 19456), and no other scope than bot:

https://discord.com/oauth2/authorize?client_id=<APPLICATION ID>&scope=bot&permissions=19456

Add Mention @everyone, @here, and All Roles (permissions=150528) only if the Critical role isn’t mentionable and you’d rather not make it so. Editing its own message after a press needs no permission; that goes through the press itself. 4. The channel: with Developer Mode on, right-click it, Copy Channel ID. If the channel is private, add the bot’s role to it. 5. The roles that may press the buttons. Make a role (say, “Alarm ack”), give it to the people who may acknowledge from Discord, and copy its id. Shelving takes an alarm off the list for an hour, so it has its own setting; it can be the same role or a narrower one. 6. Configure it and restart:

HOTLOOP_NOTIFY_ENABLED=true
HOTLOOP_NOTIFY_DISCORD_BOT_ENABLED=true
HOTLOOP_NOTIFY_DISCORD_BOT_TOKEN=MTIz...
HOTLOOP_NOTIFY_DISCORD_BOT_CHANNEL=223456789012345678
HOTLOOP_NOTIFY_DISCORD_BOT_ACK_ROLES=323456789012345678
HOTLOOP_NOTIFY_DISCORD_BOT_SHELVE_ROLES=423456789012345678
HOTLOOP_NOTIFY_DISCORD_BOT_CRITICAL_ROLE=323456789012345678
HOTLOOP_PUBLIC_URL=https://hotloop.plant.example

7. The log says connected to the Discord Gateway; Discord buttons work, and GET /api/diagnostics shows "gateway": {"connected": true} under the discord channel. POST /api/notify/test posts a test message (no buttons, there’s no alarm behind it).

Until you set the roles, nobody can press anything. With HOTLOOP_NOTIFY_DISCORD_BOT_ACK_ROLES or _SHELVE_ROLES empty, every press of that button is refused, the person pressing it is told which setting is empty, and HotLoop says so at startup:

WARN NOBODY can acknowledge alarms from Discord, because HOTLOOP_NOTIFY_DISCORD_BOT_ACK_ROLES is empty...

That’s on purpose. A button anyone in the server can press is not a default to ship: servers collect people.

Who pressed it. An acknowledgement from Discord is recorded under discord:<username>#<user id> (the id, because a Discord username can be changed and the id can’t), with the note “acknowledged from Discord by alice (1010…)”. To record it under the HotLoop user instead, map them: HOTLOOP_NOTIFY_DISCORD_BOT_USERS=101010101010101010=alice,202020202020202020=bob. The mapping only names the actor. It doesn’t check the HotLoop user’s permissions: the Discord role is what decides who may press.

Shelve shelves for HOTLOOP_NOTIFY_DISCORD_BOT_SHELVE, one hour by default, at most 24 hours like any shelve. It’s a fixed time, not a menu: at 03:00 one button that does one known thing beats a dropdown. It refuses an alarm that’s back to normal (shelving it would hide whatever trips it next) and one that’s already shelved.

After a press, the message says who did what and when, and both buttons go gray, so the next person to look at the channel sees it’s handled. A refused press is answered only to the person who pressed it, with the reason, and recorded in the system log.

How a press is authorized

The same machinery as ntfy’s Acknowledge link (see “Acknowledge links”), plus Discord’s own word on who pressed:

• The button is signed. Each button carries a token signed with the same secret as the ack links, for that action (acknowledge or shelve, signed in, so one can’t be replayed as the other), on that alarm, raised at that moment. Discord allows a button 100 characters of data and an alarm id can be 64, so this is a compact form of the same claim: 8 bytes of raise time, 4 of expiry and a 96-bit HMAC. Forging one means guessing 96 bits, one Discord press per guess, from inside the server with the right role.
• The person holds a role you named. Discord tells the bot the presser’s roles with the press, over the bot’s own authenticated connection. Anyone who can hand out roles in your server can hand out this one, so treat those roles like the permission they are.
• The occurrence is still current. The engine checks, under its own lock, that the alarm’s current raise is the one on the button. Yesterday’s message can’t acknowledge today’s trip.
• Once. The first action on an occurrence from any notification (ntfy or Discord) is recorded and every later one refused. Two people pressing at once: one wins, the other is told.
• Expiring. After HOTLOOP_NOTIFY_ACK_TTL (12 hours) the buttons stop working; act in HotLoop instead.

The bot token is a credential for the whole bot. It is never logged, never in an error, never in an API response; in the Helm chart it goes in a Secret. So is each press’s own interaction token, which Discord gives the bot for fifteen minutes to answer it. If the bot token leaks, reset it in the developer portal.

The Gateway connection

Presses arrive over the Discord Gateway, a WebSocket HotLoop opens to Discord and keeps open. The client is HotLoop’s own, on gorilla/websocket (v1.5.3, pinned; it was already in this module’s dependency graph, it’s small and it’s the most used WebSocket library in Go). We didn’t take a Discord library such as discordgo: it’s a whole Discord SDK (REST, voice, a state cache) for the one event we need, and this project doesn’t carry vendor SDKs. The client:

• identifies with intents 0, so Discord sends it nothing but presses;
• heartbeats at the interval Discord gives, answers a heartbeat request at once, and treats a heartbeat that isn’t acknowledged as a dead connection;
• resumes the session after a drop, a Reconnect (op 7) or a resumable Invalid Session (op 9), so a press made while it was reconnecting is replayed, not lost; a session that can’t be resumed waits one to five seconds and starts a new one;
• identifies no more than once every five seconds, and not at all when Discord says the day’s session starts are used up, until they reset;
• reconnects with a backoff that doubles from one second to a minute;
• stops for good on a refused token or refused intents, switches the channel off and shows it in /api/diagnostics, rather than reconnecting forever.

Discord gives the bot three seconds to answer a press, so the first thing HotLoop does is answer “got it” (a deferred update), then does the work. That answer is also what keeps two HotLoop replicas from both acting on one press: both receive it, Discord takes the first answer and refuses the second, and the replica that was refused stops. Run as an MCP helper (--mcp-stdio), HotLoop doesn’t open a Gateway connection at all; the real one handles the presses.

What’s verified, and what needs a live run

Discord can’t run in our lab, so the channels are tested against internal/notify/discordtest: a real HTTP server standing in for Discord’s REST API, and a real WebSocket server speaking the Gateway frames the client uses, both in the test process. It enforces Discord’s documented limits and answers the way Discord’s documentation says Discord answers. Tested that way, in CI:

• the exact JSON of a message, byte for byte; allowed_mentions locked to nothing, the role pinged only on Critical and never on a test, a mention in an alarm name escaped;
• every length limit kept, cut on a character boundary, with emoji; a summary of forty long-named alarms keeping its count;
• a 429 retried after the body’s retry_after (not the whole-second Retry-After beside it) and after X-RateLimit-Reset-After; an empty bucket waited out before sending;
• 401, 403 and 404 each switching the channel off after exactly one request, logged once, shown in /api/diagnostics; a 500 and a refused connection not switching it off;
• a flood of twelve through the default budget: five messages, then one summary naming the other seven;
• no webhook URL, token, bot token, interaction token or button token in any error, log line, audit event or API response;
• the Gateway client, as protocol conformance: Identify with intents 0, the heartbeat interval, an immediate heartbeat on request, a clean 1000 close on shutdown; Reconnect resuming with the right session and sequence and getting the missed press replayed, closing with a code that keeps the session; Invalid Session both ways, with the wait and the identify gap; a zombie connection resumed; a dropped one resumed; 4009 starting a new session; 4014 and a refused token stopping for good; the day’s session starts waited out;
• end to end against the real alarm engine on a real Modbus register: the alarm raised, posted with its buttons, Acknowledge pressed as Discord sends it, and the real alarm acknowledged as discord:alice#<id>, the message edited to say so with its buttons off; Shelve pressed by a mapped user and the alarm shelved for the configured time under the HotLoop name; a forged button, a person without the role, a second press and a stale button from an earlier raise each refused, only to the presser, and in the system log; a bot with no roles set refusing everyone and saying so at startup.

What that proves: HotLoop sends what Discord documents and copes with every answer Discord documents. What it can’t: that Discord behaves as documented, how the message looks in the Discord app, whether a push reaches a phone, and the Gateway features this client never asks for (compression, sharding). Those need a live run, and there are two, skipped unless you give them credentials:

# the webhook: sends a Critical, a High and a test message
HOTLOOP_TEST_DISCORD_WEBHOOK=https://discord.com/api/webhooks/...
HOTLOOP_TEST_DISCORD_ROLE=<role id to ping on the Critical one>   # optional
go test ./internal/notify/ -run TestDiscordWebhookLive -v

# the bot: connects to the real Gateway, raises a real alarm, posts it with
# buttons, and with PRESS_WAIT set waits for you to press Acknowledge
HOTLOOP_TEST_DISCORD_BOT_TOKEN=...
HOTLOOP_TEST_DISCORD_CHANNEL=<channel id>
HOTLOOP_TEST_DISCORD_ROLE=<role id you hold>
HOTLOOP_TEST_DISCORD_PRESS_WAIT=5m
go test ./internal/api/ -run TestDiscordBotLive -v

The bot test needs the database and the rest of the integration services like every test in internal/api. Not yet run live: neither test has been run against the real Discord as of this writing. Until it has, the Discord channels are tested, not verified.

Rate limits and summaries

A power dip or a PLC reboot trips forty alarms in a second. One send per alarm gets a Discord webhook’s IP banned, trips ntfy’s per-IP limit, and buries the one alarm that matters under thirty-nine that don’t on the phone that gets them. So every channel has a budget, a token bucket: a burst that may go out at once, then one more every refill.

Channel	Default burst	Then one every
ntfy	10	5 s (under ntfy.sh’s own limit of 60, then one per 5 s, per IP)
Gotify	10	5 s
Webhook	10	5 s
SMTP	5	12 s
Discord webhook	5	2 s (Discord allows a webhook about 5 requests every 2 s, and a channel 30 messages a minute)
Discord bot	5	2 s

Change them per channel with HOTLOOP_NOTIFY_<CHANNEL>_BURST and HOTLOOP_NOTIFY_<CHANNEL>_REFILL (WEBHOOK, SMTP, NTFY, GOTIFY, DISCORD_WEBHOOK, DISCORD_BOT).

What the budget allows goes out on its own, straight away. What it doesn’t is held, and when the next send is due everything held goes out as one summary:

7 more alarms raised in the last 12 s:
- [High] Line 3 Stopped
- [Critical] Boiler Trip
...

No alarm is dropped: the summary names every one, up to 25, and counts the rest (“…and 13 more”). It’s as severe as the worst alarm in it, and the audit event for it lists every alarm id it stood for. It has no Acknowledge button, because there’s no single alarm for it to acknowledge. If only one alarm was held, it goes out as itself, button and all. Once alarms are being held, new ones join the batch even if the budget has come back, so a summary never arrives after an alarm raised later than everything in it. On shutdown, whatever is still held goes out at once, over budget: the choice then is between one send too many and alarms nobody is ever told about.

The budget is per channel, so a flood that has the SMTP relay queueing doesn’t hold up ntfy, and each channel’s floor still applies: a Low alarm never ends up in the summary of a channel set to High and above.

Retries

A send that gets a 429, a 5xx, or no answer at all (refused, reset, timed out) is tried again, up to three attempts in all, all inside 30 seconds of the first. When the receiver says how long to wait (Retry-After, in seconds or as a date), that’s how long we wait; otherwise one second, then two. A receiver that asks for longer than the 30 seconds has left is not waited on: the failure is reported now, saying so. After 30 seconds a page that hasn’t gone out is better reported as failed, in the log and the audit trail, than retried quietly for minutes while the alarm sits there.

Any other 4xx (a wrong token, a topic the user can’t write, a malformed request) is never retried. It’ll be the same answer every time until somebody fixes the configuration, and Discord bans an IP that keeps sending requests it has already refused.

POST /api/notify/test makes exactly one attempt per channel. A test is asking what the channel does right now, and a retry would make a flaky receiver look fine.

Acknowledge links

The Acknowledge button on an ntfy notification is a request to POST /api/notify/ack/{token} on HOTLOOP_PUBLIC_URL. The person pressing it at 03:00 isn’t logged in to anything, so the token is the whole authorization. That only works if the token is unforgeable, narrow, single use and short-lived, and each of those is enforced:

• Unforgeable. The token is a claim (which alarm, when it was raised, which channel, when the link expires) plus an HMAC-SHA256 of it with a server secret. Change one byte, or sign with any other key, and it’s refused. The secret is HOTLOOP_NOTIFY_ACK_SECRET if you set it (32 characters at least; openssl rand -hex 32). If you don’t, HotLoop generates 32 random bytes the first time it needs them and keeps them in its database (the settings table, key notify.ackSecret), so links keep working across a restart and every replica agrees. It’s never shown on the settings screen or in any API response. It is in database backups, like everything else in the database; if that’s a problem for you, set the variable from a Kubernetes Secret instead (notify.ackSecret.existingSecret in the chart), and it never touches the database. Changing the secret invalidates every link already sent, which is also how you revoke them all at once.
• Narrow. The claim names one alarm and the moment it was raised. A button from yesterday’s trip of the same alarm can’t acknowledge today’s, which nobody has seen yet: the engine checks, under the same lock as the acknowledgement, that the alarm’s current raise is the one in the claim (alarm.Engine.AckOccurrence). The link can acknowledge and nothing else. It can’t shelve, clear, write a tag or read anything, so the worst a leaked link can do is exactly what pressing the button would have done.
• Single use. The first use of a link for an occurrence is recorded in the database (notify_ack_links), and every later use is refused, even when the alarm needs acknowledging again without a new raise (shelved while still in alarm and unshelved, it comes back unacknowledged). The alarm’s own state isn’t enough for this; the record is.
• Short-lived. It expires, 12 hours after it’s sent by default (HOTLOOP_NOTIFY_ACK_TTL, at most 168h). Long enough for “I’ll look when I get in”, short enough that a screenshot of a notification is not a standing credential.

A successful press is an ordinary ISA-18.2 acknowledgement through the alarm engine, audited exactly like one from the alarm screen: an alarm event and a system-log entry with the actor notify:ntfy and the note “acknowledged from the ntfy notification”. ntfy’s HTTP actions don’t say which phone pressed the button, so there’s no person’s name to add. When several people share a topic, the audit trail says the phone did it, not who. The Discord bot knows the user and puts it there (see “Discord”).

Every refusal is answered with what went wrong and recorded in the system log under notify:<channel> (or notify:ack-link for a token too broken to say which channel sent it), because a button that silently did nothing is indistinguishable from one nobody pressed:

Answer	Why
403	Forged, altered, or signed with a different secret
410	Genuine but expired
409	Already used, or the alarm was raised again since, or there’s nothing left to acknowledge
404	The alarm no longer exists, or acknowledge links are off

The token itself is never logged, never put in an audit event and never returned by any API: for as long as it’s valid it’s a credential. It does sit in the ntfy message, necessarily, so anyone who can read your ntfy topic can press the button. Lock the topic down.

The endpoint answers with Access-Control-Allow-Origin: * so the button also works from the ntfy web app, which presses it from its own origin. That gives nothing away: nothing on this route reads a cookie or a session.

TLS on the SMTP channel

The channel upgrades to TLS via STARTTLS automatically whenever the relay advertises the extension, regardless of any setting below. This is not optional behavior to opt into: Go’s own net/smtp.PlainAuth refuses to transmit a username and password over a connection that is not already TLS (or to a server literally named localhost), and essentially every real provider that requires authentication, Gmail, SendGrid, Mailgun, an ISP’s relay, requires STARTTLS on port 587 or implicit TLS on 465 (see below) to get there. A relay with no authentication configured, reachable on an isolated network, may not offer STARTTLS at all, and this channel does not require it in that case.

HOTLOOP_NOTIFY_SMTP_TLS does not turn the upgrade on, it turns the absence of it into a hard failure: if set and the relay does not advertise STARTTLS, Send refuses before ever attempting AUTH, rather than quietly authenticating in plaintext against a relay that merely happens to be reachable as localhost. HOTLOOP_NOTIFY_SMTP_TLS_SKIP_VERIFY skips certificate verification on the upgrade, the same escape hatch HOTLOOP_EDGE_PUBLISH_TLS_SKIP_VERIFY and HOTLOOP_FLEET_TLS_SKIP_VERIFY already provide for their own MQTT connections, for a relay presenting a self-signed certificate.

Implicit TLS, port 465

A relay on port 465 expects the whole connection to be TLS from the first byte (RFC 8314), not a plaintext greeting and an upgrade. HotLoop used to only do the upgrade, so on 465 the first thing it said was plaintext to a relay waiting for a TLS handshake and the send died. Now port 465 is implicit TLS and every other port is STARTTLS as before, with no setting to touch:

HOTLOOP_NOTIFY_SMTP_PORT=465

If a relay does implicit TLS on a port of its own, or speaks STARTTLS on 465, HOTLOOP_NOTIFY_SMTP_IMPLICIT_TLS says which, true or false, and wins over the port. In the Helm chart set it under extraEnv. Everything else is the same on both: the username and password go over the encrypted connection, HOTLOOP_NOTIFY_SMTP_TLS is met by it (there is nothing to require), and HOTLOOP_NOTIFY_SMTP_TLS_SKIP_VERIFY skips the certificate check on its handshake. The relay’s certificate is checked unless you turn that off. Scheduled reports use the same relay settings and so the same rule.

If the handshake fails, the error says so and names HOTLOOP_NOTIFY_SMTP_IMPLICIT_TLS=false, which is what a relay that starts in plaintext needs.

Proved against a real SMTP server library with a TLS-only listener, and against Mailpit run with --smtp-require-tls, the pinned lab version v1.31.2. Not verified against a commercial provider’s 465 endpoint.

One SMTP code path, shared with scheduled reports

Since 4.14.0 the alarm channel sends through notify.SendMessage, a general function that sends one message, with optional attachments, to a list of recipients. Scheduled reports (see REPORTS.md) send their CSV and HTML files through the same function and the same relay settings (HOTLOOP_NOTIFY_SMTP_HOST, _PORT, _USERNAME, _PASSWORD, _FROM, _TLS, _TLS_SKIP_VERIFY), so there is one place that dials, upgrades with STARTTLS, authenticates, and decides what a header may contain. Only the relay is shared: HOTLOOP_NOTIFY_ENABLED, HOTLOOP_NOTIFY_SMTP_ENABLED, and HOTLOOP_NOTIFY_SMTP_TO are about alarms, and a relay configured for reports does not turn the alarm channel on.

Two behaviors of the alarm email changed, both in the direction of safety. The sender and every recipient are now validated as plain addresses before anything is dialed, where before a malformed one failed at the relay. And a subject with non-ASCII characters is sent as an RFC 2047 encoded word instead of raw UTF-8. For an ordinary ASCII alarm the message is the same bytes as before apart from the Date header, which TestAlarmMessageBytesAreUnchanged pins. An alarm name that contains a control character is flattened to spaces rather than refused, because a page has to go out whatever an alarm is called.

Priority floors

Each channel has its own MinPriority, the least severe alarm priority it still fires for. model.AlarmPriority runs 1 (Critical, most severe) to 4 (Low), so “High and above” is MinPriority = 2: an alarm meets a channel’s floor when its own priority number is less than or equal to the floor. A webhook configured for Critical-only and an SMTP relay configured for Medium-and-above can watch the same alarm engine and fire on entirely different sets of alarms, nothing requires them to agree.

What it is not

Not a live-reload of who gets paged. Channels are read once, from configuration, at process start, the same as EdgePublishConfig and FleetConfig. Changing HOTLOOP_NOTIFY_* needs a restart to take effect, there is no API to add or remove a channel at runtime in this MVP.

Not a table of its own. Every notification attempt, success or failure, is written through runtime.Manager.Event, the same generic audit-event mechanism every alarm and fleet action already logs through. There was no need invented for a dedicated notifications table for a feature this size; the system log already answers “was a notification sent, and did it work.”

Not a durable queue. Retries are bounded (three attempts, 30 seconds, see “Retries”) and held alarms live in memory until their summary goes out. A page that still fails is logged and audited and the dispatcher moves on. An alarm that stays raised because nobody acknowledged it is still visible on the alarms screen and in alarm_events regardless of whether a page ever arrived: the notification is a convenience on top of the alarm system, not a replacement for it.

Configuration

Variable	Default	Details
HOTLOOP_NOTIFY_ENABLED	false	The feature is off until this is set
HOTLOOP_NOTIFY_WEBHOOK_ENABLED	false
HOTLOOP_NOTIFY_WEBHOOK_URL	n/a	Required when the webhook channel is enabled. Must be http:// or https://
HOTLOOP_NOTIFY_WEBHOOK_MIN_PRIORITY	high	critical, high, medium, low, or 1-4
HOTLOOP_NOTIFY_SMTP_ENABLED	false
HOTLOOP_NOTIFY_SMTP_HOST	n/a	Required when the SMTP channel is enabled
HOTLOOP_NOTIFY_SMTP_PORT	25
HOTLOOP_NOTIFY_SMTP_USERNAME / _PASSWORD	n/a	Optional. Omit both for an anonymous relay
HOTLOOP_NOTIFY_SMTP_FROM	n/a	Required when the SMTP channel is enabled
HOTLOOP_NOTIFY_SMTP_TO	n/a	Required when the SMTP channel is enabled. Comma-separated addresses
HOTLOOP_NOTIFY_SMTP_TLS	false	Require the relay to offer STARTTLS; see “TLS on the SMTP channel” above, STARTTLS is used automatically whenever offered regardless of this setting
HOTLOOP_NOTIFY_SMTP_TLS_SKIP_VERIFY	false	Skip certificate verification on the STARTTLS upgrade, or on the handshake of implicit TLS. Per-deployment opt-in; never set this by default
HOTLOOP_NOTIFY_SMTP_IMPLICIT_TLS	true on port 465, otherwise false	TLS from the first byte instead of STARTTLS; see “Implicit TLS, port 465”. Set it to override the port
HOTLOOP_NOTIFY_SMTP_MIN_PRIORITY	high	Same values as the webhook’s
HOTLOOP_NOTIFY_NTFY_ENABLED	false
HOTLOOP_NOTIFY_NTFY_URL	none	Required. The server alone, https://ntfy.example.com, no path
HOTLOOP_NOTIFY_NTFY_TOPIC	none	Required. 1 to 64 letters, digits, - and _. On a public server it’s the password
HOTLOOP_NOTIFY_NTFY_TOKEN	none	An ntfy access token (tk_...), sent as a bearer token
HOTLOOP_NOTIFY_NTFY_USERNAME / _PASSWORD	none	Basic auth instead of a token. Not both
HOTLOOP_NOTIFY_NTFY_ACK	true	Put an Acknowledge button on every alarm (needs HOTLOOP_PUBLIC_URL)
HOTLOOP_NOTIFY_NTFY_MIN_PRIORITY	high	Same values as the webhook’s
HOTLOOP_NOTIFY_GOTIFY_ENABLED	false
HOTLOOP_NOTIFY_GOTIFY_URL	none	Required. https://gotify.example.com
HOTLOOP_NOTIFY_GOTIFY_TOKEN	none	Required. The application token Gotify gave you for HotLoop
HOTLOOP_NOTIFY_GOTIFY_MIN_PRIORITY	high	Same values as the webhook’s
HOTLOOP_NOTIFY_DISCORD_WEBHOOK_ENABLED	false
HOTLOOP_NOTIFY_DISCORD_WEBHOOK_URL	none	Required. The URL Discord’s Copy Webhook URL gives. The whole of it is the password
HOTLOOP_NOTIFY_DISCORD_WEBHOOK_CRITICAL_ROLE	none	A role id to ping on Critical alarms
HOTLOOP_NOTIFY_DISCORD_WEBHOOK_MIN_PRIORITY	high	Same values as the webhook’s
HOTLOOP_NOTIFY_DISCORD_BOT_ENABLED	false
HOTLOOP_NOTIFY_DISCORD_BOT_TOKEN	none	Required. The bot’s token, alone, without Bot in front
HOTLOOP_NOTIFY_DISCORD_BOT_CHANNEL	none	Required. The channel id to post in
HOTLOOP_NOTIFY_DISCORD_BOT_ACK_ROLES	none	Comma-separated role ids whose members may press Acknowledge. Empty is nobody
HOTLOOP_NOTIFY_DISCORD_BOT_SHELVE_ROLES	none	The same for Shelve. Empty is nobody
HOTLOOP_NOTIFY_DISCORD_BOT_USERS	none	discordUserId=hotloopUser, comma-separated: whose actions are recorded under their HotLoop name
HOTLOOP_NOTIFY_DISCORD_BOT_SHELVE	1h	How long Shelve shelves for. At most 24h
HOTLOOP_NOTIFY_DISCORD_BOT_CRITICAL_ROLE	none	A role id to ping on Critical alarms
HOTLOOP_NOTIFY_DISCORD_BOT_MIN_PRIORITY	high	Same values as the webhook’s
HOTLOOP_NOTIFY_DISCORD_API_URL	https://discord.com/api/v10	Only to send the bot’s requests through a proxy
HOTLOOP_NOTIFY_<CHANNEL>_BURST / _REFILL	per channel	The rate limit; see “Rate limits and summaries”. <CHANNEL> is WEBHOOK, SMTP, NTFY, GOTIFY, DISCORD_WEBHOOK or DISCORD_BOT
HOTLOOP_PUBLIC_URL	none	The address a phone reaches HotLoop at, base path included. Needed for tap-to-open and the Acknowledge button
HOTLOOP_NOTIFY_ACK_SECRET	generated	Signs acknowledge links. At least 32 characters. Unset: generated once and kept in the database
HOTLOOP_NOTIFY_ACK_TTL	12h	How long an acknowledge link works. At most 168h

In the Helm chart these are under notify.ntfy, notify.gotify, notify.discordWebhook, notify.discordBot, notify.ackSecret, notify.ackTtl and the top-level publicUrl. Tokens, passwords and the Discord webhook URL go into a Secret the chart manages, or name your own with existingSecret.

API

Route	Permission	Details
POST /api/notify/test	notify.test	Sends a synthetic test notification through every configured channel right now
POST /api/notify/ack/{token}	none; the token is the authorization	Acknowledges the alarm occurrence the token was issued for. See “Acknowledge links”
GET /api/diagnostics	status.read	Under notify.channels, each channel’s standing: whether Discord has refused it (and why, and since when), and the Discord bot’s Gateway connection

notify.test is admin-only, granted by the wildcard the admin role already carries and deliberately not added to the operator role’s permission list, sending a real external message and spending whatever quota a paging provider or SMTP relay allows is closer in kind to fleet.restart and fleet.config (also admin-only, also “do something to the outside world right now”) than to a read, the same narrowing this app already applies to those two.

The response reports each channel’s own outcome rather than a single pass/fail for the request:

{
  "results": [
    {"channel": "webhook", "success": true},
    {"channel": "smtp", "success": false, "error": "dial smtp server mail.example.com:587: i/o timeout"}
  ]
}

What has and has not been verified

Tested, not verified, see this repo’s own distinction between the two. The webhook channel is tested against a real net/http/httptest server: the exact JSON body, an unreachable URL failing within its bounded context rather than hanging, and a non-2xx response surfacing as an error rather than a false success. The SMTP channel is tested against two real SMTP servers, axllent/mailpit, run via Podman, never Docker, with the message’s actual arrival, subject, body and recipients confirmed afterward through Mailpit’s own HTTP API, not just that Send returned nil:

• A bare instance with no authentication, proving the ordinary path.
• A second instance configured to require STARTTLS and a real login (Mailpit’s --smtp-require-starttls and --smtp-auth-file), proving the regression this feature actually shipped with once: net/smtp.PlainAuth silently refuses to send credentials over a connection that is not already TLS, so a channel configured with a username and password only ever worked against an unauthenticated local relay until the STARTTLS upgrade was added, the bare instance’s tests could not have caught this, since PlainAuth’s refusal only triggers when a username is actually configured against a non-TLS, non-localhost connection. A third test proves HOTLOOP_NOTIFY_SMTP_TLS is a real refusal, pointed at a relay that offers no STARTTLS at all, Send fails before ever attempting AUTH.

The dispatcher’s priority-floor filtering and its raised-only firing are both covered directly, with a fake in-package channel standing in for “some channel,” the same way internal/fleet’s tests use a real edgepublish.Publisher to stand in for “some edge relay” rather than a hand-rolled mock.

The webhook’s failures are also tested for the URL: a Discord-shaped URL with a token in the path and in the query, against a real server that refuses the connection, one that answers 500 and one that never answers, through POST /api/notify/test, and neither token turns up in the response, the log or the audit event.

ntfy and Gotify are tested against the real servers, binwiederhier/ntfy and gotify/server, pinned in test/services/up.sh and run in CI, with what arrived read back through each server’s own API, the way a phone gets it. The ntfy server requires a login for everything, so bearer and basic auth are both exercised for real. Covered: the title, priority, tags, tap URL and Acknowledge button of a raised alarm; a flood of ten alarms through a budget of three producing exactly three sends and one summary naming the other seven; held alarms going out on shutdown; a 429 with Retry-After waited out and retried (ntfy itself never sends Retry-After, so a small proxy in front of the real server answers the first request with one); a 5xx retried on the backoff; a Retry-After past the budget not waited on; a wrong token sent once, never retried; and no token, password, topic or acknowledge link in any error, log line or result. Gotify: every priority’s mapping, the Markdown and the tap URL, a summary, and a wrong token reported with Gotify’s own reason and never printed.

The Acknowledge button is tested end to end in internal/api: a real alarm on a real Modbus register raises, the notification lands on the real ntfy, and the button’s own request, read back from ntfy, is fired at HotLoop with no session. The alarm is acknowledged in the real engine, audited as notify:ntfy. A forged link, a used one (also after a shelve and unshelve), one from an earlier occurrence of the same alarm and an expired one are each refused and each leave the alarm unacknowledged. Nothing in the log, the audit trail or the API carries the ntfy token, the topic, the signing key or any link.

Not verified: an iPhone receiving a push from a self-hosted ntfy through ntfy.sh’s relay, and the ntfy and Gotify phone apps themselves. The tests prove what the servers hold and what the button sends; what a given phone does with it is the app’s business.

The Discord channels have their own account of this; see “Discord”, “What’s verified, and what needs a live run”.

Not verified against any specific commercial paging provider’s webhook format (PagerDuty, Opsgenie, ntfy.sh, Pushover, Slack, Discord; see the Webhook section above for which of those can’t work with it at all) or against a phone carrier’s email-to-SMS gateway, those are real-world integrations this app cannot certify from the inside, only document, the same honesty docs/FLEET.md applies to NCMD authorization and a certified Sparkplug host. Also not verified: implicit TLS (a relay expecting TLS from the first byte, typically port 465), only STARTTLS’s in-band upgrade of a plaintext connection is supported, which covers the common case (port 587 or 25) but not every relay an operator might point this at.