Alarms

The Alarms page is GEM's centralized alarm console: every alarm — attribute thresholds, device/zone connectivity, connection flapping — lands in one place with severity, an acknowledge workflow, shelving, and a full event history.

Open Alarms

Alarms vs. Triggers

Attribute Triggers are stateless: they fire an action every time a matching attribute write meets the condition. Alarms are stateful — a condition crossing into truth opens one alarm event, and crossing back out closes it. An alarm event records when it activated, who acknowledged it, when it cleared, and every notification it sent. Use a trigger to automate a reaction; use an alarm when a human needs to see, acknowledge, and track a problem.

Overview

The page is split into three tabs:

1. Active — open alarms with acknowledge and shelve controls
2. History — past alarm events with filters
3. Definitions — create and manage alarm definitions

Active Tab

A strip of tiles summarizes open alarms: Open, Unacknowledged, Shelved, and per-severity counts (Critical / High / Medium / Low). The table below lists open alarm events under Open Alarms — severity, alarm name, message, source entity, activation time, running duration, and state chips (active/cleared, acknowledged-by, shelved). The list updates live as alarms activate, clear, and get acknowledged.

Ordering

Open alarms are listed newest first, capped at the 500 most recent — the list is never re-sorted by severity, so a critical alarm raised this morning sits below a low one raised since. Read the severity picture off the tiles, not the top of the table.

• Ack — marks the event acknowledged with your username and timestamp. Acknowledging stops repeat notifications and any pending escalation steps.
• Shelve… — suppresses the event's notifications for 1 hour, 8 hours, 24 hours, 7 days, 14 days, or 30 days without closing it. Shelved rows dim; re-occurrences of the same alarm source inherit the shelve window, and shelve windows survive a server restart — a 30-day shelve does not silently re-arm because the controller rebooted. Unshelve from the same menu.
• Log (clock icon) — expands the event's notification delivery log: every send with its phase (activation, escalation step, repeat, cleared), channel, and per-recipient outcome — including recipients blocked by their notification profile schedule, so a suppressed delivery is visible instead of silently lost. The log keeps the 50 most recent entries per event; on a long-running alarm with a short repeat interval, the oldest sends roll off.
• Explain with AI (AI icon) — opens the AI assistant with a causal brief for the event. The assistant gathers everything around the activation itself — the rule that fired and the value that tripped it, the source entity's live state, the attribute history leading up to activation, whether the same alarm has recurred over the last 30 days, and what else happened in the same window (other alarms activating together, automation runs, user actions) — and narrates what happened, why, and what to do about it. Ten devices alarming in the same minute reads as a switch failure, not ten device failures; ask follow-ups to dig further. Available on both the open list and the History tab.

A latched alarm stays listed after its condition clears, until someone acknowledges it — use latching for events that must not be missed (a leak that dried before anyone looked).

History Tab

Every alarm event, newest first, filterable by date range (default: last 30 days), severity, and definition. Shows activation/clear times, total duration, and who acknowledged. Events load 200 at a time — Load 200 More appends the next page and the count beside it tells you how many are on screen. Export CSV downloads the full filtered range regardless of what's loaded on screen. Closed events are pruned by database maintenance after the retention period (alarm_event_retention_days system attribute, default 90 days, configurable on the Data Retention page); open and latched-unacknowledged events are never pruned.

Definitions Tab

An alarm definition describes what to watch and how to escalate.

The Alarm Definitions grid lists every definition with its ID, Name, Label, Severity, Source, Category, Latched, and Enabled state. Add opens an empty editor, Edit opens the selected one, Delete removes it. Name, Label, and Enabled can be edited straight in the grid — handy for muting one alarm during commissioning without opening the editor; the rest are edited in the form.

Every editor starts with the same header fields:

• Name — the internal identifier, lower-case letters, numbers and underscores only (freezer_temp_high). Rejected otherwise. Locked on system definitions.
• Label — the human-readable name (Freezer Temp High). This is what appears on the alarm list, in emails, and on end-user panels, so fill it in — without one, the raw Name is shown.
• Description — free text for the next integrator.
• Category — a free-form grouping tag (hvac, security, network). Optional.
• Enabled — off leaves the definition on file without arming it. Disabling a definition also closes any of its still-open events on the next sweep.

Save Alarm Definition commits; the engine re-arms immediately, with no restart needed.

Suggested alarms (registry templates)

The attribute registry ships curated alarm templates for common attributes — low battery, water leak, smoke, CO, gas, panic. When this site actually reports one of those attributes, a suggestion card appears at the top of the tab: "Water Leak — 14 zones report leak. Suggested: leak = true, latched."

• Enable… opens the definition editor pre-filled with the template (wildcard [name:*] target, operator, threshold, severity, latching, message) — nothing goes live until you review the threshold and save. The saved definition is an ordinary one, editable like any other.
• Dismiss hides a suggestion you don't want; restore it by removing its key from the dismissed_alarm_templates system attribute.
• A suggestion disappears permanently once a definition has been created from it (even if that definition is later disabled). Templates are never auto-activated.

Life-safety templates (smoke, CO, gas, leak, panic) ship latched — the event stays open until acknowledged even if the sensor clears itself. Driver authors can ship their own templates by adding alarm_template to a registry seed row.

Condition definitions

Attached to an attribute using the same condition builder as triggers (target type, entity, attribute, operator, value — including subsystem-wide targets, [name:*] patterns, [driver:…] scopes with an optional single-device limit, and expression values). "Any lock from the Seam integration with battery below 20%" is: target type zone, by driver seam_api, attribute battery, less than 20. Alarm-specific behavior:

• Deadband — hysteresis for numeric thresholds. temperature > 80 with deadband 2 activates above 80 and clears only at or below 78, so a value hovering at the threshold produces one alarm, not a storm. It applies only to the four numeric threshold operators (greater than, greater than or equal, less than, less than or equal) and only when the comparison value is a plain number — on an equals/not-equals condition, or one comparing against a token or expression, the deadband is ignored and the alarm clears as soon as the condition goes false. Reach for Off Delay instead in those cases.
• On Delay — the condition must hold true this many seconds before the alarm activates (rides through momentary spikes).
• Off Delay — the condition must hold false this many seconds before the alarm clears (rides through momentary recoveries).
• Message — template with [token] substitution: [value], [previous_value], [attribute_name], [target_name], [severity], plus the entity's own fields ([name], [label], …).

System definitions

Seeded automatically and raised by the platform rather than by a condition:

Definition	Seeded severity	Raised when	Cleared when
Device Offline	High	a device fails its connection checks past the configured threshold	stable recovery is confirmed
Zone Offline	Medium	a zone reports disconnected past the threshold while its device is online	the zone reconnects
Device Connection Flapping	Medium	a device disconnects repeatedly inside the flap-detection window	a full window passes quietly
Zone Connection Flapping	Low	a zone disconnects repeatedly inside the flap-detection window	a full window passes quietly
Network Monitor Down	High	a network monitor confirms a host down	the monitor returns to up (also cleared when the monitor is disabled or deleted)
Monitor Tag Down	High	a monitor tag rollup goes fully down	the tag recovers to up or degraded (partial outages stay visible through the per-monitor alarms)
Remote Offline	Low	a remote control loses its connection (after the debounce window)	it reconnects
AV Zone Power Divergence	Medium	a driver-reported power_state shows an AV zone physically on while GEM shows it off and no source could be resolved to auto-resync (a successful auto-resync is logged, not alarmed)	the states converge
Camera Stream Wedged	Medium	a camera stream process survives a force-kill (stuck in uninterruptible I/O — dead camera or severe network stall); the stream stays down and its restart is held until the process exits	the stuck process finally exits

The seeded severity is a starting point, not a verdict — it is the one field integrators most often retune per site (a flapping gate at a commercial entry may deserve High). Seeding happens once per definition: an existing row is never overwritten on upgrade, so your edits stick.

Monitor dependency suppression carries over: a monitor skipped because its upstream is down neither raises nor clears — only confirmed down/up transitions move the alarm.

Devices that ride on a parent controller (the Controller field on the device) are suppressed the same way: their connection state mirrors the parent's, so when a controller drops, it raises the one meaningful Device Offline alarm for the shared link and its child devices neither raise duplicate offline alarms nor trip flap detection — a controller with many slaves produces one alarm, not a storm.

The alarm center is the only notification sender. When a system definition has no notify steps configured, the engine sends a built-in notification — batched over a short window so an outage storm becomes one digest email instead of many — honoring each source's own recipient configuration:

• Monitors — the monitor's Email/SMS recipients and profile-override setting (from the Monitoring editor); system default address when none are configured.
• Monitor tags — the tag's rollup recipients, same fallback. Tag down/up macros run independently of notifications.
• Devices / zones / remotes — classic device_alerts semantics: a per-device address wins, explicit false suppresses, otherwise cfg.device_alerts (where boolean true means the system default address, and nothing configured means no email).

Configuring notify steps on a definition replaces the built-in notification entirely — recipients, channels, escalation, and repeats then come from the definition. Every built-in send is stamped onto each event's delivery log, so the Log view always answers "who was told."

Alarm source cells cross-link back to the owning page — monitors and tags to Monitoring, devices and zones to their admin pages — landing pre-filtered to the specific row that raised the alarm rather than the full list. The Monitoring page shows an alarm chip on any monitor or tag with an open event.

The thresholds are the existing device attributes (connection_alert_threshold, flap_threshold, flap_window). Severity, latching, notifications, repeat, and re-notify cooldown settings on system definitions are editable; the condition itself is not — opening one shows a note in place of the condition builder, and the Name field is locked so the definition stays findable by the platform.

Fresh installs seed a 4-hour re-notify cooldown on the four definitions that are prone to cycling — Device Connection Flapping, Zone Connection Flapping, Remote Offline, and AV Zone Power Divergence — so a device that flaps all day produces one notification per window instead of one per storm. Existing sites keep their current behavior until the definition is edited.

Behavior

• Severity — critical, high, medium, or low. It drives the tile counts, the Alarm Feed widget's minimum-severity filter and worst-severity summary, and whether a web push is sent time-sensitive (critical only). It does not re-order any list — alarms are listed newest first everywhere.
• Latched — event stays open until acknowledged, even after the condition clears.
• Repeat — while an alarm is active and unacknowledged, re-send its notifications at this interval. Acknowledging or shelving stops repeats.
• Re-notify Cooldown — an anti-spam throttle for sources that cycle in and out of alarm (a flapping device, a sensor oscillating around a threshold without a deadband). After an activation notification goes out for a given alarm + source, further re-activations of that same pair stay silent until the cooldown elapses — at most one notification per source per window (240 = once per 4 hours, 1440 = once per day). Suppressed cycles are fully silent (the paired "cleared" notice is skipped too), but the events are still recorded on the dashboard and each suppression is stamped into the delivery log, so the Log view shows why nothing was sent. The cooldown clock survives a server restart, only advances on a real send, and never affects other sources of the same alarm. Repeat and escalation for a still-open unacknowledged event are not gated — the cooldown paces re-activations, not the nag chain for an alarm nobody has dealt with.

Notifications

Tick any combination of the four Channels — Email, SMS, Web Push, Call. Ticking a channel reveals its recipient field. Every delivery is logged onto the event, per recipient.

• Email / SMS — enter recipients, or leave the field empty to use the system default addresses. Each recipient's notification profile is checked individually; blocked recipients are recorded in the delivery log by address rather than silently dropped.
• Web Push / Call — pick recipients under Push / Call Users; each chosen user is notified on their own devices. Critical-severity pushes are sent time-sensitive. These two channels require at least one user — saving with neither set is rejected, because an empty target would broadcast to every connected panel.
• Override Schedules — bypass notification profiles for must-deliver alarms. This toggle applies to the immediate notification; each escalation step carries its own Override schedules checkbox.
• Notify On Clear — send a "back to normal" notice to the same recipients as the immediate notification.

Two things that quietly do nothing

Leaving all four channels unticked is valid and means "dashboard-only" — the alarm still opens, counts on the tiles, appears in history, and runs its macro, but nobody is notified. It is the right setting for an alarm you want visible but not noisy.

Because Notify On Clear mirrors the immediate notification, it has nothing to copy when no channel is ticked. Turning it on without ticking a channel sends no clear notice. Tick the channel you want first, then Notify On Clear.

Escalation steps

Beyond the immediate notification, a definition can carry escalation steps — each with its own delay, channels, recipients, and override flag. Add Escalation Step appends one (defaulting to after 15 min unacknowledged); a step fires once the alarm has been active and unacknowledged for its delay. The shape to aim for: notify the on-duty person immediately, escalate to SMS after 15 minutes, call the manager after 60.

Each step needs at least one channel and a delay greater than zero, or the save is rejected naming the offending step. You do not have to add them in order — steps are sorted by delay when you save. Rules:

• Steps fire once each, earliest delay first, at most one step per minute per event — a backlog after a shelve window expires drains gradually instead of bursting.
• Acknowledging or shelving the alarm stops the chain; if it is still unacknowledged when a shelve expires, pending steps resume.
• Fired steps are persisted on the event, so a server restart never re-sends an escalation.
• Escalations also count toward the repeat interval, so a repeat won't stack on top of an escalation in the same minute.

Automation

Run Macro On Activate and Run Macro On Clear optionally fire a macro on each transition — turn on a fault light, close a valve, log to an external system. The macro runs whether or not anyone was notified, so a dashboard-only or cooldown-silenced alarm still drives its automation.

Housekeeping you will see in History

Once a minute the engine reconciles open alarms against reality, which closes events nobody explicitly cleared. Two show up in History with a telling clear value:

• definition removed — the definition was deleted or disabled while it had open events. They close rather than linger unresolvable.
• entity removed — the zone or device the alarm watched no longer exists.

The same pass re-checks each condition alarm against the entity's current value, so an alarm whose attribute quietly recovered (no change event to react to) still clears within a minute.

Email action buttons

Alarm emails carry action buttons — Acknowledge, Shelve 1h / 24h / 7d, and Open Alarms — so a recipient can deal with an alarm straight from their inbox. The links deep-link into this page (/admin/alarms?alarm_action=…); the recipient logs in if needed (the link survives the login round-trip), confirms the action in a prompt, and the acknowledge/shelve executes through the normal authenticated API with full attribution and delivery-log stamping. The links carry no credentials — forwarding the email grants nothing, and recipients without an admin-capable GEM login cannot act on them. Cleared latched events still offer Acknowledge; cleared non-latched events get no buttons.

The button URLs need a base address recipients can actually reach. GEM auto-detects the best URL admins use to browse to it, captured only from authenticated browser sessions so unauthenticated requests can't poison the emailed links (preferring https and hostnames over bare IPs; stored in the detected_admin_origin system attribute) — set the alarm_email_base_url system attribute to override it, e.g. https://customer.mygem.us. If neither yields a usable URL, emails are sent without buttons and the server log notes it; logging into the admin from a browser seeds the detection.

End-user alarm feed widget

The Alarm Feed widget (category Security) puts the alarm center on end-user panels: a summary tile with open-alarm count and worst severity, expanding to a live list with per-alarm acknowledge. Add it to any page from the UI Pages editor. Settings:

• Minimum Severity — Low (all), Medium, High, or Critical only. Default Low.
• Only Unacknowledged — hide alarms someone has already picked up. Default off.
• Allow Acknowledge — default on; turn it off for a read-only display panel.
• Max Rows — default 10.

Shelved alarms are hidden from the feed — shelving is the operator's "stop bothering people" control.

Set Minimum Severity, not just Max Rows

The feed shows the newest matching alarms, not the worst. Max Rows trims from the bottom of a newest-first list, so on a busy site a critical alarm can fall off a 10-row feed under newer low-severity ones. If a panel exists to surface the serious stuff, raise Minimum Severity rather than relying on row count.

Listing, watching, and acknowledging alarms are available to regular (non-admin) users so the feed works from any panel; acknowledgement records who acknowledged. Shelving, history, and statistics remain admin-only.

AI assistant

• Voice — "any alerts?" (the alert_summary skill) now reports the alarm center's open events for the asker's accessible zones alongside the raw sensor sweep.
• Admin chat — the get_alarms skill reads open events and history with severity, acknowledge state, and the notification delivery log: "what alarms are active?", "who was notified about the freezer alarm?".
• Admin chat (explain) — the alarm_context skill deep-dives one alarm event for a causal brief: the rule that fired and the value that tripped it, the source entity's live state, the attribute timeline around activation, 30-day recurrence for the same alarm + source, and correlated alarms / automation runs / user actions in the same window — "explain alarm event #123", "why does the freezer alarm keep firing at night?". The Explain with AI button on each alarm row asks this for you.
• Admin chat (commissioning) — the bulk_create_alarms skill creates many definitions in one atomic, confirmed call: "set up freezer temperature alarms — critical over 32°F on freezer_1, freezer_2, freezer_3". Every entry is validated before any write, a shared defaults block covers common severity/notify settings, and the engine reloads once at the end so new alarms are armed immediately.
• Admin chat (cloning) — the clone_alarm skill duplicates a tuned definition — conditions, notify escalation ladder, actions, deadband/delays, repeat, and cooldown — to one or many new names, or "stamps" it across many similar targets with the condition's target rewritten per clone: "clone the freezer_high_temp alarm for freezer_2 through freezer_5". Atomic with rollback, capped at 25 clones per call.

For developers

Drivers and server code can raise alarms programmatically:

await GemServer.getInstance().raiseAlarm({
    system_key: 'device_offline',        // or definition_id / definition_name
    system_target: 'device', target_id: dev.id,
    message: 'device offline: pool controller',
    value: 'disconnected'
});
await GemServer.getInstance().clearAlarm({
    system_key: 'device_offline',
    system_target: 'device', target_id: dev.id,
    value: 'connected'
});

Both calls are idempotent (re-raising an active alarm is a no-op), never throw, and are safe to call before the engine has initialized.