Alarms

The Alarms page is GEM's centralized alarm console: every alarm — attribute thresholds, device/zone connectivity, connection flapping — lands in one place with severity, an acknowledge workflow, shelving, and a full event history.

Open Alarms

Alarms vs. Triggers

Attribute Triggers are stateless: they fire an action every time a matching attribute write meets the condition. Alarms are stateful — a condition crossing into truth opens one alarm event, and crossing back out closes it. An alarm event records when it activated, who acknowledged it, when it cleared, and every notification it sent. Use a trigger to automate a reaction; use an alarm when a human needs to see, acknowledge, and track a problem.

Overview

The page is split into four tabs:

1. Active — open alarms with acknowledge and shelve controls
2. History — past alarm events with filters
3. Definitions — create and manage alarm definitions
4. Shelved — suppressed alarm sources, including sources whose latest occurrence has cleared

Active Tab

A strip of tiles summarizes open alarms: Open, Unacknowledged, Shelved, and per-severity counts (Critical / High / Medium / Low). The table below lists open alarm events under Open Alarms — severity, alarm name, message, source entity, activation time, running duration, and state chips (active/cleared, acknowledged-by, shelved). The list updates live as alarms activate, clear, and get acknowledged, and re-reads itself after a reconnect (a GEM update or a network blip) so nothing that happened while the socket was down is missed. If the list itself fails to load, the page says so with a Retry button rather than showing an empty "No open alarms".

Ordering

Open alarms are ordered by unshelved first, then unacknowledged, severity, and oldest activation. The table loads 100 at a time with Previous/Next controls. Tile counts include every open event, regardless of page size. Click a tile to filter, or narrow by state, source, definition, and search. The Performance report link opens alarm-performance reporting.

• Ack — marks the event acknowledged with your username and timestamp. Acknowledging stops repeat notifications and any pending escalation steps.
• Shelve… — suppresses the event's notifications for 1 hour, 8 hours, 24 hours, 7 days, 14 days, or 30 days without closing it. Shelved rows dim; re-occurrences of the same alarm source inherit the shelve window, and shelve windows survive a server restart — a 30-day shelve does not silently re-arm because the controller rebooted. Unshelve from the same menu. Unshelve lifts the shelf from the alarm + source pair as a whole — every occurrence raised while it was shelved is released too, so a restart cannot bring the shelf back from one of them.
• Log (clock icon) — expands the event's notification delivery log: every send with its phase (activation, escalation step, repeat, cleared), channel, and per-recipient outcome — including recipients blocked by their notification profile schedule, so a suppressed delivery is visible instead of silently lost. The log keeps the 50 most recent entries per event; on a long-running alarm with a short repeat interval, the oldest sends roll off.
• Explain with AI (AI icon) — opens the AI assistant with a causal brief for the event. The assistant gathers everything around the activation itself — the rule that fired and the value that tripped it, the source entity's live state, the attribute history leading up to activation, whether the same alarm has recurred over the last 30 days, and what else happened in the same window (other alarms activating together, automation runs, user actions) — and narrates what happened, why, and what to do about it. Ten devices alarming in the same minute reads as a switch failure, not ten device failures; ask follow-ups to dig further. Available on both the open list and the History tab.

A latched alarm stays listed after its condition clears, until someone acknowledges it — use latching for events that must not be missed (a leak that dried before anyone looked).

Click an alarm name to inspect its captured condition, trigger and current values, delays, clear reason, acknowledgement, next escalation, and delivery jobs with attempts and retry times. Shelving accepts an optional reason and records who changed the shelf. The Shelved tab lets an operator release the entire definition/source pair even after the alarm clears; its source column cross-links the same way the open and history lists do — a device or zone opens in a reference modal over the page, other sources open the admin page that owns them. The last 100 shelving actions are retained on the pair.

Scoped to one source

Another page can open the console narrowed to a single source — the alarm chip on a Monitoring card does this. The URL carries system_target and target_id (both are required; either alone is ignored), plus an optional target_name for display, e.g. /admin/alarms?tab=1&system_target=monitor&target_id=12&target_name=Front%20Gate. Both the Active and History lists then show only that source, and a banner above each names it (Showing alarms for monitor › Front Gate only). The scope is kept when you change the other filters; Show all alarms / Show all events clears it from both tabs. Without these parameters nothing is narrowed.

Live updates received during a snapshot load are merged into that snapshot. Reconnect triggers a fresh read; the console also refreshes every 30 seconds and explicitly labels disconnected or failed reads rather than claiming all-clear. The end-user feed widget does not poll — it relies on the live feed and the reconnect resync, so a wall panel never flashes a loading state on a timer.

History Tab

Every alarm event, newest first, filterable by date range (default: last 30 days), severity, and definition. Shows activation/clear times, total duration, and who acknowledged. Events load 200 at a time — Load 200 More appends the next page and the count beside it tells you how many are on screen. Export CSV downloads the full filtered range regardless of what's loaded on screen. Closed events are pruned by database maintenance after the retention period (alarm_event_retention_days system attribute, default 90 days, configurable on the Data Retention page); open and latched-unacknowledged events are never pruned.

Definitions Tab

An alarm definition describes what to watch and how to escalate.

The Alarm Definitions grid lists every definition with its ID, Name, Label, Severity, Source, Category, Schedule, Latched, and Enabled state. Add opens an empty editor, Edit opens the selected one, Delete removes it. Name, Label, and Enabled can be edited straight in the grid — handy for muting one alarm during commissioning without opening the editor; the rest are edited in the form.

A definition an installed application package created carries a package icon at the end of its row. The tooltip names the package and version; click it to open that package on the Applications page. Deleting a package-installed definition changes what an uninstall removes, and its alarm history keeps naming a definition that no longer exists.

Every editor starts with the same header fields:

• Name — the internal identifier, lower-case letters, numbers and underscores only (freezer_temp_high). Rejected otherwise. Locked on system definitions.
• Label — the human-readable name (Freezer Temp High). This is what appears on the alarm list, in emails, and on end-user panels, so fill it in — without one, the raw Name is shown.
• Description — free text for the next integrator.
• Category — a free-form grouping tag (hvac, security, network). Optional.
• Enabled — off leaves the definition on file without arming it. Disabling a definition also closes any of its still-open events on the next sweep.

Save Alarm Definition commits; the engine re-arms immediately, with no restart needed.

Suggested alarms (registry templates)

The attribute registry ships curated alarm templates for common attributes — low battery, water leak, smoke, CO, gas, panic. When this site actually reports one of those attributes, a suggestion card appears at the top of the tab: "Water Leak — 14 zones report leak. Suggested: leak = true, latched."

• Enable… opens the definition editor pre-filled with the template (wildcard [name:*] target, operator, threshold, severity, latching, re-notify cooldown, message) — nothing goes live until you review the threshold and save. The saved definition is an ordinary one, editable like any other.
• Dismiss hides a suggestion you don't want; restore it by removing its key from the dismissed_alarm_templates system attribute.
• A suggestion disappears permanently once a definition has been created from it (even if that definition is later disabled). Templates are never auto-activated.

Life-safety templates (smoke, CO, gas, leak, panic) ship latched — the event stays open until acknowledged even if the sensor clears itself. Event-style templates — the camera detections, where a person walking past activates the alarm once per event — ship with a re-notify cooldown instead, so a busy entrance is one notification per window rather than one per passer-by. Driver authors can ship their own templates by adding alarm_template to a registry seed row, on the target the driver actually writes: a template on a device row for an attribute the driver writes on zones is a suggestion no site can ever see.

Condition definitions

Attached to an attribute using the same condition builder as triggers (target type, entity, attribute, operator, value — including subsystem-wide targets, [name:*] patterns, [driver:…] scopes with an optional single-device limit, and expression values). "Any lock from the Seam integration with battery below 20%" is: target type zone, by driver seam_api, attribute battery, less than 20. Alarm-specific behavior:

• Deadband — hysteresis for numeric thresholds. temperature > 80 with deadband 2 activates above 80 and clears only at or below 78, so a value hovering at the threshold produces one alarm, not a storm. It applies only to the four numeric threshold operators (greater than, greater than or equal, less than, less than or equal) and only when the comparison value is a plain number — on an equals/not-equals condition, or one comparing against a token or expression, the deadband is ignored and the alarm clears as soon as the condition goes false. Reach for Off Delay instead in those cases.
• On Delay — the condition must hold true this many seconds before the alarm activates (rides through momentary spikes).
• Off Delay — the condition must hold false this many seconds before the alarm clears (rides through momentary recoveries).
• Message — template with [token] substitution: [value], [previous_value], [attribute_name], [target_name], [severity], plus the entity's own fields ([name], [label], …). Secure attributes are not substituted — a token naming one ([password], [api_key], anything flagged secure) is left in the message literally, unresolved. The resolved message is stored on the alarm event, pushed to every subscribed client and sent out by email and SMS, so a credential must never be able to ride along in it. The same rule applies to every notify step's message.

System definitions

Seeded automatically and raised by the platform rather than by a condition:

Definition	Seeded severity	Raised when	Cleared when
Device Offline	High	a device fails its connection checks past the configured threshold	stable recovery is confirmed
Zone Offline	Medium	a zone reports disconnected past the threshold while its device is online, or a LoRaWAN sensor goes quiet past its offline window	the zone reconnects (or the sensor uplinks again)
Device Connection Flapping	Medium	a device disconnects repeatedly inside the flap-detection window	a full window passes quietly
Zone Connection Flapping	Low	a zone disconnects repeatedly inside the flap-detection window	a full window passes quietly
Device Refusing Commands	High	a device stays connected but answers every command with an error — wedge_threshold failures in a row spanning at least wedge_min_span (today: Global Caché IR sends, where a wedged blaster keeps answering its ping)	the first command the device accepts
Network Monitor Down	High	a network monitor confirms a host down	the monitor returns to up (also cleared when the monitor is disabled or deleted)
Monitor Tag Down	High	a monitor tag rollup goes fully down	the tag recovers to up or degraded (partial outages stay visible through the per-monitor alarms)
Remote Offline	Low	a remote control loses its connection (after the debounce window)	it reconnects
AV Zone Power Divergence	Medium	a driver-reported power_state shows an AV zone physically on while GEM shows it off and no source could be resolved to auto-resync (a successful auto-resync is logged, not alarmed)	the states converge
Camera Stream Wedged	Medium	a camera stream process survives a force-kill (stuck in uninterruptible I/O — dead camera or severe network stall); the stream stays down and its restart is held until the process exits	the stuck process finally exits
Access PIN Scanning	High	several different credentials are refused at one access point inside the window — the signature of someone trying combinations rather than a user mistyping one code	the window passes with no further denials
Access Denied Outside Hours	Medium	a credential is refused at an access point during its configured off-hours window (opt-in, per site or per door)	the window passes with no further denials
Attribute History Flooding	Medium	attribute series that no deadband is filtering are together writing more than attribute_history_flood_alarm_rate rows/hour (default 5000), measured over a closed hour	the first hourly evaluation that falls back under the threshold
Disk Space Low	High	the filesystem GEM runs from passes disk_alarm_percent (default 90%)	usage falls five points back under
System Memory Low	High	available system memory falls past memory_alarm_percent (default 90% used)	usage falls five points back under
CPU Saturated	Medium	CPU load holds at or above cpu_alarm_percent (default 95%) for cpu_alarm_minutes (default 15)	load falls five points back under
Network Route Ambiguous	Low	two or more default routes share the lowest metric, so which interface carries outbound traffic is decided by DHCP lease order rather than by configuration	one route wins outright
Water Alert	Critical	a water monitor reports a standing critical alert — leak, unusual activity, or an automatic shutoff. Warning-tier alerts publish as attributes only, so a valve self-test does not page anyone	the alert is resolved at the device or in the vendor app (see Moen Flo)
Unknown Device Schedule	Medium	a device runs an on-board schedule GEM was not told about — a WattBox schedule created in OvrC, which the unit's own log reveals the first time it fires	every schedule the unit has run is listed in the device's known_schedules
Wireless Mic Battery Low	Medium	a wireless microphone transmitter on a Shure SLX-D receiver reports its battery at or below the low threshold, in bars or runtime minutes. Raised on the channel's zone	the battery reads above the threshold again, or the transmitter is switched off
UPS On Battery	High	a UPS-backed power device (a WattBox with a battery module) reports input power lost, so the rack is running down its battery	mains power returns
Macro Expectation Failed	Medium	a macro finished but the expectations declared on it still do not hold when they are read — the scene ran and the system is not in the state it was supposed to reach	a later run of the same macro finds every expectation satisfied (also cleared when the macro is disabled or deleted)
Application Error	Low	a browser page or the server throws an uncaught error that has not been seen before, or one acknowledged on an earlier GEM version comes back. Repeats of a known error never raise it	never: it is latched, so it stays open until acknowledged here or on the Diagnostics page (acknowledging either one acknowledges both)

The seeded severity is a starting point, not a verdict — it is the one field integrators most often retune per site (a flapping gate at a commercial entry may deserve High). Seeding happens once per definition: an existing row is never overwritten on upgrade, so your edits stick.

New definitions are seeded at boot, so the ones added by an upgrade appear in this list the first time the controller starts on the new build — enabled, with no recipients. Attribute History Flooding, Disk Space Low, System Memory Low, CPU Saturated, Network Route Ambiguous, Water Alert, Unknown Device Schedule, UPS On Battery and Application Error arrive this way on existing sites; until someone gives them notify channels they are dashboard-only.

Monitor dependency suppression carries over: a monitor skipped because its upstream is down neither raises nor clears — only confirmed down/up transitions move the alarm.

Attribute History Flooding

The one system alarm whose source is not an entity. A device that starts writing an attribute far faster than anyone intended fills attribute_history — the largest table in the system — with rows nobody will ever read, and nothing else on the controller is watching. Left alone it consumes the disk, and a controller with no disk stops controlling the building.

Its source cross-link opens Data Retention, which both names the responsible attributes and carries the fix.

Why the threshold is so much higher than the panel's badge. The flooding badge on that page marks any series writing 60+ rows/hour that the deadband is not filtering. That is the right bar for something you look at, and on a site with two or three chatty attributes it is permanently true — which would make an alarm on it permanently active, and a permanently active alarm gets disabled. The alarm asks the second question instead: is the combined unfiltered write rate on a trajectory that fills the disk. 5000 rows/hour is roughly 120,000 rows a day; the residence that prompted this feature ran about 37,000 rows/hour of unfiltered metering and reached 2.7 million rows and 1.66 GB.

So: the badge means you have something worth tuning, the alarm means the database is in trouble. Both come from the same counters.

Tune the threshold with the attribute_history_flood_alarm_rate system attribute. Setting it to 0 disables the alarm while leaving the badge, the diagnostics check, and the hourly log line exactly as they are.

Two properties worth knowing:

• It evaluates hourly, when a measured window closes — a rate extrapolated from a partial window would flap. Detection latency is therefore up to an hour, which is immaterial at a threshold measured in thousands of rows per hour.
• The window closes on a write. A controller that goes completely silent stops evaluating and leaves an active alarm standing, which is the honest reading: nothing has disproved it.

Host resources

Three alarms watch the machine GEM runs on, driven by one sampler that reads disk, memory and CPU every five minutes.

Alarm	Why it matters
Disk Space Low	A controller with no disk stops controlling. Postgres refuses writes, logging stops, and lighting, HVAC, AV and security go with them. The message names the database size and its largest table, because history is nearly always the answer — its source link opens Data Retention.
System Memory Low	Sustained pressure ends with the kernel's out-of-memory killer terminating the largest process, which is GEM. Measured against available memory, not free — on Linux, free excludes page cache the kernel would hand back on demand, and reads as an emergency on a perfectly healthy machine. Its source link opens Diagnostics, which reports all three together.
CPU Saturated	Degradation rather than failure: device polling, macros and the UI all run late. Requires the load to be sustained across the window, because backups, theme builds and camera restarts legitimately peg the CPU for a minute. Its source link opens Diagnostics as well.

Every threshold is a system attribute — disk_alarm_percent, memory_alarm_percent, cpu_alarm_percent, cpu_alarm_minutes — and setting any of them to 0 disables that alarm while leaving the measurement in place, so the numbers still reach Diagnostics and the disk_used_percent / memory_used_percent / cpu_load_percent / database_size_mb attributes.

Each clears five points back under its threshold, not at the threshold itself. A disk sitting exactly on 90% would otherwise raise and clear on alternate samples all night.

note

Three alarms, not one "system health" alarm. An alarm event is keyed by definition and target, so a single merged definition could only ever represent one condition at a time — a full disk at 03:00 would mask a pegged CPU at 09:00 and still describe itself as a disk problem. They also deserve different severities, which is the field integrators retune most often: a full disk is fatal, a busy CPU is not.

And no separate database-size alarm. A database size is not a fault on its own — 40 GB on a 2 TB disk is fine and 40 GB on a 64 GB appliance is an emergency, so the only meaningful reading is against the headroom, which is what Disk Space Low measures. How fast it is growing is Attribute History Flooding's job. Between them the two ends are covered: growing too fast, and grown too big.

warning

The disk alarm measures the filesystem this controller runs from. A site whose Postgres lives on a separate host is not covered — that machine's disk is invisible from here, and needs watching wherever the rest of that host is watched.

Default route ambiguity

A controller usually has a second NIC on a deliberately isolated segment — Dante, cameras, BACnet, KNX. Under a plain dhcp4: true netplan every link takes route-metric 100, so when both come up the kernel holds two default routes at the same cost and the winner is whichever DHCP lease landed first. That is a coin flip, and it is re-tossed on every reboot and carrier event.

Lose the toss and outbound traffic leaves over a segment with no path to the internet. The cloud tunnel, update checks and remote support all stop — while local control keeps working perfectly and hides it. DNS usually breaks too, because both links advertise themselves to systemd-resolved as default-route resolvers and the stub returns SERVFAIL even though each upstream answers correctly on its own.

This alarm is predictive, and that is the point. It fires on the structural fact — two routes tied at the lowest metric — and never on whether the internet is currently reachable. Two reasons:

• Alarming on reachability would make an ordinary ISP outage raise it, and an alarm that cries wolf during every blip is one people stop reading.
• By the time the wrong link actually wins, no notification can get out anyway. Email and push both need the egress that just broke.

So it fires while the site is still healthy and the fix is unhurried, telling you that you are one power blip away from going dark rather than reporting that you already have. It is seeded Low for the same reason: nothing is broken at the moment it raises.

The fix is to give the intended link a lower route metric in netplan, and to keep the isolated one out of DNS:

network:
  version: 2
  ethernets:
    enp4s0:                      # the link with internet
      dhcp4: true
      dhcp4-overrides:
        route-metric: 100
    enp3s0:                      # the isolated segment
      dhcp4: true
      dhcp4-overrides:
        route-metric: 2000
        use-dns: false

Apply it with sudo netplan apply. The alarm clears on the next five-minute sample. The installer offers to write exactly this as /etc/netplan/99-gem-route-priority.yaml when it detects the tie during preflight — deleting that file undoes it.

note

A controller that is genuinely offline cannot tell anyone. This alarm covers the window before that happens; once the tunnel is actually down, the cloud side's tunnel-down notice is what reaches you.

Devices that ride on a parent controller (the Controller field on the device) are suppressed the same way: their connection state mirrors the parent's, so when a controller drops, it raises the one meaningful Device Offline alarm for the shared link and its child devices neither raise duplicate offline alarms nor trip flap detection — a controller with many slaves produces one alarm, not a storm.

The alarm center is the only notification sender, and notifications are explicit. An alarm with no configured recipients anywhere is dashboard-only — nothing is ever mailed to the system default address just because a monitor went down. When a system definition has no notify steps configured, the engine sends a built-in notification — batched over a short window so an outage storm becomes one digest email instead of many — but only to recipients someone actually configured:

• Monitors — the monitor's Email/SMS recipients and profile-override setting (from the Monitoring editor). Both lists empty = dashboard-only.
• Monitor tags — the tag's rollup recipients, same rule. Tag down/up macros run independently of notifications.
• Devices / zones / remotes — classic device_alerts semantics, for the connection family only (Device Offline, Zone Offline, the two Flapping definitions, Device Refusing Commands, Remote Offline) plus Water Alert: a per-device address wins, explicit false suppresses, otherwise cfg.device_alerts (where boolean true means the system default address — a deliberate site-level opt-in — and nothing configured means no email). That address was set up for outages, so the other built-ins — host resources, history flooding, ambiguous routes, access scans, a wedged camera relay, AV power divergence — never ride it. They are dashboard-only until their definition carries notify steps of its own.

The built-in notification announces recovery only where someone was waiting on an outage to end — the offline definitions, Network Monitor Down, Monitor Tag Down and Water Alert. A cleared CPU spike or full disk is a second message with nothing left to do, so it is not sent. Notify steps you configure on a definition, including their own cleared list, are unaffected.

Configuring notify steps on a definition replaces the device_alerts fallbacks, not per-source subscriptions. Recipients named on a monitor or tag itself keep receiving their alerts alongside the steps — a facilities engineer subscribed to one BACnet monitor still hears about it after site-wide routing is configured. Every built-in send is stamped onto each event's delivery log, so the Log view always answers "who was told," and a skipped dashboard-only alarm is noted in the server log.

Alarm source cells cross-link back to the row that raised the alarm. A device or zone source opens its record in a reference modal over this page, so the severity and state filters, the history window, the expanded notification log and any AI brief on screen all survive the look-up; closing the modal puts you back where you were. When the record is gone, or the modal stack is already full, it falls back to the entity's admin page filtered to that row. Every other source — monitors and tags to Monitoring, a macro to Macros, the flooding-attribute alarm to Data Retention, the host-resource alarms to Diagnostics — names a page rather than a row, and changes the page as it always has. The Monitoring page shows an alarm chip on any monitor or tag with an open event.

The thresholds are the existing device attributes (connection_alert_threshold, flap_threshold, flap_window, flap_alert_cooldown). Severity, latching, notifications, repeat, and re-notify cooldown settings on system definitions are editable; the condition itself is not — opening one shows a note in place of the condition builder, and the Name field is locked so the definition stays findable by the platform.

A 4-hour re-notify cooldown is seeded on the definitions that are prone to cycling — Device Connection Flapping, Zone Connection Flapping, Zone Offline (a LoRaWAN sensor at the edge of range goes stale and back all day), Remote Offline, AV Zone Power Divergence and Camera Stream Wedged (a dead camera re-wedges its relay on every restart) — so a source that cycles all day produces one notification per window instead of one per cycle. The seed fills in on an existing site too, but only where the definition's cooldown is still blank; a tuned value is never touched, and 0 is an explicit "no cooldown" that stays. To turn pacing off, set 0 rather than clearing the field.

The two flapping definitions are paced a second time at the source: the driver holds back repeat flap alarms for the same device or zone for a week by default, so the alarm itself is raised once per week rather than once per storm. See flap_alert_cooldown.

Behavior

• Severity — critical, high, medium, or low. It drives triage ordering, tile counts, the Alarm Feed widget's minimum-severity filter and worst-severity summary, and whether a web push is sent time-sensitive (critical only). History remains newest first.
• Latched — event stays open until acknowledged, even after the condition clears.
• Repeat — while an alarm is active and unacknowledged, re-send its notifications at this interval. Acknowledging or shelving stops repeats.
• Re-notify Cooldown — an anti-spam throttle for sources that cycle in and out of alarm (a flapping device, a sensor oscillating around a threshold without a deadband). After an activation notification goes out for a given alarm + source, further re-activations of that same pair stay silent until the cooldown elapses — at most one notification per source per window (240 = once per 4 hours, 1440 = once per day). Suppressed cycles are fully silent (the paired "cleared" notice is skipped too), but the events are still recorded on the dashboard and each suppression is stamped into the delivery log, so the Log view shows why nothing was sent. The cooldown clock survives a server restart, only advances on a real send, and never affects other sources of the same alarm. Repeat and escalation for a still-open unacknowledged event are not gated — the cooldown paces re-activations, not the nag chain for an alarm nobody has dealt with.

Active Schedule

By default a definition is armed around the clock. The collapsible Active Schedule section restricts it to a day/hour window — the classic case is a camera person-detection alarm that should only exist overnight (10 PM–8 AM) while the property is supposed to be empty. Pick the active days, then paint the active hours on the ribbon (the Overnight preset is one click), or switch to Time Range mode for a start/end pair — including sunset/sunrise date variables with minute offsets, which track the seasons where a fixed hour mask can't.

The schedule gates the alarm's existence, not just its notifications:

• Outside the window, condition crossings are ignored entirely — no event is recorded, nothing counts on the tiles, no macro runs. A person walking past the camera at noon leaves no trace.
• An alarm still active when the window closes clears automatically within a minute (schedule inactive in History). Latched events keep their acknowledge requirement — a 3 AM person alarm stays listed at 9 AM until someone acknowledges it.
• An on-delay that straddles the closing edge does not activate; escalation and repeats stop with the event.
• System definitions accept a schedule too — the platform re-raises on its own cadence, so a device still offline when the window opens alarms within about a minute.

Evaluation mode distinguishes persistent state from events. Auto treats numeric threshold comparisons as persistent state; State treats equality and every other condition that way too; Event evaluates incoming reports only and never creates an occurrence from an old reading. Use Event for motion, person detections and button presses, and State for persistent boolean faults. A persistent-state definition is re-checked against every current target of its attribute at three moments: when the controller boots, when the definition is saved or enabled, and when its schedule window opens — so a condition that is already true at any of those moments raises then rather than on the next attribute write. It is deliberately not re-checked every minute: the minute sweep reconciles alarms that are already open, and re-evaluating every target of every numeric definition sixty times an hour would have written a log line for each.

System sources (device connectivity, monitors, hosts) persist their latest raise or clear intent with each call, against the definition even while it is disabled — so a device that recovers while its Device Offline definition is switched off does not come back as a phantom alarm when the definition is re-enabled. The sweep replays a persisted intent only where it disagrees with what the engine holds in memory: an event write that failed, or a schedule window that has just opened.

A schedule that blocks every day or every hour is flagged in the editor ("never runs") — it would silence the definition completely. A broken time range (unresolvable variable) fails active, never silent: alarming at the wrong hours is recoverable, an alarm that never fires is not.

To keep the alarm always-on but quiet people's phones at certain hours, use notification profiles or the per-step Override schedules flag instead — those gate delivery, this gates the alarm itself.

Notifications

Tick any combination of the four Channels — Email, SMS, Web Push, Call. Ticking a channel reveals its recipient field. Every delivery is logged onto the event, per recipient.

• Email / SMS — enter recipients, or leave the field empty to use the system default addresses. Each recipient's notification profile is checked individually; blocked recipients are recorded in the delivery log by address rather than silently dropped.
• Web Push / Call — pick recipients under Push / Call Users; each chosen user is notified on their own devices. Critical-severity pushes are sent time-sensitive. These two channels require at least one user — saving with neither set is rejected, because an empty target would broadcast to every connected panel.
• Override Schedules — bypass notification profiles for must-deliver alarms. This toggle applies to the immediate notification; each escalation step carries its own Override schedules checkbox.
• Notify On Clear — send a "back to normal" notice to the same recipients as the immediate notification.

Two things that quietly do nothing

Leaving all four channels unticked is valid and means "dashboard-only" — the alarm still opens, counts on the tiles, appears in history, and runs its macro, but nobody is notified. It is the right setting for an alarm you want visible but not noisy.

Because Notify On Clear mirrors the immediate notification, it has nothing to copy when no channel is ticked. Turning it on without ticking a channel sends no clear notice. Tick the channel you want first, then Notify On Clear.

A definition whose clear steps were authored outside this form — the bulk_create_alarms assistant skill can give the clear notice different recipients or channels from the activation — keeps them exactly as they are: the editor says so under the switch, saves them untouched, and only turning Notify On Clear off removes them. The same holds for anything else a step carries that the form does not show (a per-step message, UI-targeted push): every save overlays the form's fields on the step as it was loaded rather than rebuilding it.

Escalation steps

Beyond the immediate notification, a definition can carry escalation steps — each with its own delay, channels, recipients, and override flag. Add Escalation Step appends one (defaulting to after 15 min unacknowledged); a step fires once the alarm has been active and unacknowledged for its delay. The shape to aim for: notify the on-duty person immediately, escalate to SMS after 15 minutes, call the manager after 60.

Each step needs at least one channel and a non-negative delay; zero means an additional immediate notification. You do not have to add them in order — steps are sorted by delay when you save. Rules:

• Steps fire once each, earliest delay first, at most one step per minute per event — a backlog after a shelve window expires drains gradually instead of bursting.
• Acknowledging or shelving the alarm stops the chain; if it is still unacknowledged when a shelve expires, pending steps resume.
• Each step has a stable ID and produces persistent per-channel delivery jobs. Failed channels retry without re-sending to recipients whose acceptance was already recorded. A controller crash during sending releases the job after a five-minute lease. A rare crash between provider acceptance and saving that result can duplicate a notice: delivery is at least once, not exactly once.
• Escalations also count toward the repeat interval, so a repeat won't stack on top of an escalation in the same minute.

Delivery reliability

Activation and clear transitions store their notification jobs in the same transaction as the occurrence change. The worker is nudged the moment a transition queues a job, so an immediate notification leaves within milliseconds as it always did, and it also polls every five seconds to pick up retries and anything a crash left behind. Built-in system notices (a device or monitor with no notify steps on its definition) keep their digest: every notice raised inside the same 30-second window goes out as one email with the classic GEM Alarm: … subject, Location header, and per-event source, time and action buttons. Jobs show pending, sending, accepted, suppressed, failed, or cancelled status in alarm details. Provider acceptance is not a guarantee that a person read a message.

Failures retry with backoff (5, 10, 20, 40, then 60 minutes) up to eight attempts — roughly five hours — after which the job is marked failed, the outcome is stamped on the delivery log with a gave_up marker and a server-log warning, and the occurrence is released for retention. No recipients or notification-profile suppression is terminal for that job, not a transport failure. Shelving pauses queued escalation retries; acknowledgement cancels escalation and repeat jobs. Clear cancels undelivered activation jobs, and a recovery notice is skipped if none of that occurrence's activation jobs reached a recipient. An already in-flight provider call cannot be recalled.

Queueing a notice does not advance cooldown: only recorded acceptance does. Outstanding jobs survive restart and prevent retention from deleting their occurrence. Notification routing and the rule description are captured for each new occurrence, so editing or reordering a definition's steps cannot change an existing event's notification ladder. Definition disable/removal and current schedules still gate delivery.

Automation

Run Macro On Activate and Run Macro On Clear optionally fire a macro on each transition — turn on a fault light, close a valve, log to an external system. The macro runs whether or not anyone was notified, so a dashboard-only or cooldown-silenced alarm still drives its automation.

Housekeeping you will see in History

Once a minute the engine reconciles open alarms against reality, which closes events nobody explicitly cleared. Two show up in History with a telling clear value:

• definition removed — the definition was deleted or disabled while it had open events. They close rather than linger unresolvable.
• entity removed — the zone, device or subsystem the alarm watched no longer exists. Only those three can be "removed"; an alarm on a variable, site, site space, monitor or system attribute is re-read from the attribute itself.
• schedule inactive — the definition's active schedule window closed while the event was still open.

The same pass re-checks each condition alarm against its target's current value, so an alarm whose attribute quietly recovered (no change event to react to) still clears within a minute. The re-check is fed through the same state machine as a live change: it compares numbers as numbers and booleans as booleans, honors the deadband, and starts the off delay rather than clearing on the spot — so a definition with a 5-minute off delay still waits 5 minutes after a restart.

Email action buttons

Alarm emails carry action buttons — Acknowledge, Shelve 1h / 24h / 7d, and Open Alarms — so a recipient can deal with an alarm straight from their inbox. The links deep-link into this page (/admin/alarms?alarm_action=…); the recipient logs in if needed (the link survives the login round-trip), confirms the action in a prompt, and the acknowledge/shelve executes through the normal authenticated API with full attribution and delivery-log stamping. The links carry no credentials — forwarding the email grants nothing, and recipients without an admin-capable GEM login cannot act on them. Cleared latched events still offer Acknowledge; cleared non-latched events get no buttons.

The button URLs need a base address recipients can actually reach. GEM auto-detects the best URL admins use to browse to it, captured only from authenticated browser sessions so unauthenticated requests can't poison the emailed links (preferring https and hostnames over bare IPs; stored in the detected_admin_origin system attribute) — set the alarm_email_base_url system attribute to override it, e.g. https://customer.mygem.us. If neither yields a usable URL, emails are sent without buttons and the server log notes it; logging into the admin from a browser seeds the detection.

End-user alarm feed widget

The Alarm Feed widget (category Security) puts the alarm center on end-user panels: a summary tile with open-alarm count and worst severity, expanding to a live list with per-alarm acknowledge and, when allowed, shelve. Add it to any page from the UI Pages editor. Settings:

• Minimum Severity — Low (all), Medium, High, or Critical only. Default Low.
• Only Unacknowledged — hide alarms someone has already picked up. Default off.
• Allow Acknowledge — default on; turn it off for a read-only display panel.
• Allow Shelve — default off. Puts a Shelve button on each open alarm that unfolds into a choice of eighteen durations, 5 minutes to 30 days. A shelved alarm stops notifying and leaves the open list until the time is up; who shelved it, for how long, is recorded on the pair.
• Show Shelved — default off. Lists the alarms currently shelved under the open ones, with the time remaining and an Unshelve button when shelving is allowed, and a SHELVED count in the badge.
• Definition Filter / Source Filter — comma-separated names with * and ? wildcards, matched against the definition name and the source name (Beam*, door_forced; Tower*). Empty shows everything. A feed per building wing, or a feed of only the perimeter beams, is two widgets with two filters.
• Max Rows — default 10.

Shelved alarms are hidden from the open list — shelving is the "stop bothering people" control — and shown separately only when Show Shelved is on.

Set Minimum Severity, not just Max Rows

The feed prioritizes unacknowledged alarms, then severity and age. Minimum Severity still determines which alarms belong on a panel; Max Rows limits how many of those are displayed. Server-side user scope is applied before the result limit, so unrelated alarms cannot crowd out a user's own sources.

Listing, watching, acknowledging and shelving alarms are available to regular (non-admin) users so the feed works from any panel; acknowledgement and shelving both record who. What a panel user sees is scoped and redacted: on a site whose UIs carry site spaces or zone lists, the feed lists only zone and device alarms on UIs that user can reach (the same scope that gates their commands — see Roles), never operator sources such as monitors or the controller's own disk; the acknowledge, shelve and unshelve calls are held to the same scope, and the shelved list a panel receives is filtered to it and stripped of the operator audit trail. Delivery details — who was emailed or texted — never leave the admin: panel payloads carry the event without its notification log. The "recent history" list the tile shows when nothing is open comes through a panel-safe read capped at 25 rows; the operator History tab, detail view and statistics remain admin-only.

Per-zone alarm state, arming and out of service

The alarm center maintains one derived word per zone and per device, the attribute alarm_state, so a map pin, a tile or a keypad light can read a single value instead of joining events, shelves and arming itself:

Value	Meaning
normal	Nothing open.
alarmed	An active alarm nobody has acknowledged.
alarmed_acked	Active, acknowledged.
unacked	Nothing active, but a latched alarm still awaits acknowledgement.
shelved	A pair on this target is shelved.
disarmed	The target's armed attribute is false.
out_of_service	The target's out_of_service attribute is true.

Precedence runs from the bottom of the table up: out of service beats shelved beats disarmed beats the event states. Every activation, clear, acknowledgement, shelve, unshelve and arming change recomputes it; a boot pass reconciles rows left stale by events that closed while the controller was down. Never write it. The Zone Map widget's Alarm state palette is its first reader.

Two more attributes drive it and gate the engine:

• armed (zone or device, boolean; unset means armed). false takes the target out: nothing raises on it — condition alarms and system alarms alike — and anything active clears with the reason disarmed. Setting it true again re-checks conditions that are already true, so a beam still broken raises at once rather than on its next change. A macro that writes armed on every perimeter zone is the "Arm All" preset.
• out_of_service (zone or device, boolean; unset means in service). The same gate with the reason out of service, for a sensor being repaired or awaiting a battery.

Both are ordinary attributes: set them from the attribute editor, a macro's Set Attribute step, a trigger, a script or an installed package. The sweep applies the gate too, so an arming change that happened while the engine was not listening is caught within the minute. map_label (zone, string) is a companion for maps: the short text a pin prints — a beam number such as 13 or 8A — when the Zone Map's Pin Label is set to label.

Arming and shelving as commands

Attribute writes are an operator's action, so a panel cannot set armed directly. Instead six server verbs exist on every zone and device, dispatched through the same scoped, elevation-gated command path a light switch uses and never reaching a driver:

Command	Effect
arm / disarm	armed true / false
out_of_service / in_service	out_of_service true / false
shelve	every alarm on the target shelved for args.minutes (default 60), with an optional args.reason — alarms already open are shelved individually so their escalations and repeats stop, and the target's shelved_until attribute suppresses later activations until the time is up
unshelve	the shelve lifted, on the target and on every alarm it covered

A Button widget, a Two Button zone control (control_left_action: arm, control_right_action: disarm), a macro step, the AI assistant and a packaged application all send them the way they send on; zone_id may be a list, which is what an Arm All preset is — one command over every perimeter zone. Who shelved, for how long and why is recorded on each alarm pair, and a shelve from a panel is held to the panel's scope.

AI assistant

• Voice — "any alerts?" (the alert_summary skill) now reports the alarm center's open events for the asker's accessible zones alongside the raw sensor sweep.
• Admin chat — the get_alarms skill reads open events and history with severity, acknowledge state, and the notification delivery log: "what alarms are active?", "who was notified about the freezer alarm?".
• Admin chat (explain) — the alarm_context skill deep-dives one alarm event for a causal brief: the rule that fired and the value that tripped it, the source entity's live state, the attribute timeline around activation, 30-day recurrence for the same alarm + source, and correlated alarms / automation runs / user actions in the same window — "explain alarm event #123", "why does the freezer alarm keep firing at night?". The Explain with AI button on each alarm row asks this for you.
• Admin chat (commissioning) — the bulk_create_alarms skill creates many definitions in one atomic, confirmed call: "set up freezer temperature alarms — critical over 32°F on freezer_1, freezer_2, freezer_3". Every entry is validated before any write, a shared defaults block covers common severity/notify settings, and the engine reloads once at the end so new alarms are armed immediately.
• Admin chat (cloning) — the clone_alarm skill duplicates a tuned definition — conditions, notify escalation ladder, actions, deadband/delays, active schedule, repeat, and cooldown — to one or many new names, or "stamps" it across many similar targets with the condition's target rewritten per clone: "clone the freezer_high_temp alarm for freezer_2 through freezer_5". Atomic with rollback, capped at 25 clones per call.

For developers

Drivers and server code can raise alarms programmatically:

await GemServer.getInstance().raiseAlarm({
    system_key: 'device_offline',        // or definition_id / definition_name
    system_target: 'device', target_id: dev.id,
    message: 'device offline: pool controller',
    value: 'disconnected'
});
await GemServer.getInstance().clearAlarm({
    system_key: 'device_offline',
    system_target: 'device', target_id: dev.id,
    value: 'connected'
});

Both calls are idempotent (re-raising an active alarm is a no-op), never throw, and are safe to call before the engine has initialized (unresolved definitions are ignored). Transitions are serialized per definition/source pair, preserving raise–clear–raise order even during a slow insert. The latest source intent is persisted before applying the event change — against the definition even while it is disabled — and the sweep replays it only where it disagrees with the engine's memory, which is how a failed event write is retried. If intent persistence itself fails, retries survive only in memory until the database becomes writable.

Definition rows are validated when a write changes a validated field (severity, source, evaluation mode, condition, delays, deadband, schedule masks, notify steps). A definition written before those rules can still be disabled, relabelled or given a new message without being held to them.

Schema migration 013_alarm_delivery_state backfills pair shelving, accepted-delivery timestamps, and legacy rule snapshots without replaying historical notifications. New occurrences capture the exact definition; legacy snapshots necessarily use the definition available at migration time. The additive schema/data are retained on rollback. Automation macros still execute directly after a successful transition; the durable delivery guarantee applies to notifications, not macro side effects.

Live update subscribers (subscribe_alarm_updates) receive {type, event} with type active, cleared, acknowledged, shelved, unshelved, or notified — the last is sent after a delivery-log entry is stamped onto an event, so an expanded log refreshes without a reload. The client app hands every consumer a {type: 'resync'} after a socket reconnect; consumers re-read their snapshot on it. get_recent_alarms ({limit}, capped at 25) is the panel-safe recent-history read the feed widget uses.