Alarms and notifications
The Alarms page shows what is wrong in the cluster right now and who hears about it.
Current problems
The list shows each problem with its type and how long it has lasted. Click a problem to go to the page where you fix it. All clear means every node is online, every heartbeat path answers, and every copy and backup is current.
Right after staxmaxd starts, the list shows Starting and fills in within a minute, once every node has been heard.
A problem is only sent once it has lasted a little while, so a brief blip sends nothing: about 15 seconds for a node offline or cut off, two minutes for the rest. A problem counts as cleared after it has been gone for a minute. Until a problem is sent, it shows not sent yet.
Alarm types
| Alarm | When | Sent by default |
|---|---|---|
| A node is offline | Its heartbeat stopped on every path. | Email and SMS |
| A node is cut off | It lost the other nodes (and the router), stopped its VMs to prevent split-brain, or was removed from the cluster. | Email and SMS |
| Failover | VMs were started on another node, or a takeover was held back. | Email and SMS |
| A VM is in an error state | ||
| Replication is failing or behind | A protected VM's copy is not current. | |
| A disk is filling up | A node's storage pool is over 80% full, or so full that copies to it are paused. | |
| A backup failed | A VM's last backup failed, or a node could not mount the backup storage. | |
| An offsite backup failed | Copying a VM to offsite storage did not work. | |
| A heartbeat path is down | A path between nodes does not answer, or (two nodes) a node cannot reach the router. | |
| Maintenance could not move VMs | ||
| An update failed | Or the nodes run different versions. | |
| A watchdog is missing | A node set to use a hardware watchdog cannot open it; it fences by rebooting instead. | |
| Preserved copies after a failover | Copies to keep or delete (see Virtual machines). | Not sent |
Set up email and text messages
Under Settings → Notifications:
Email (SMTP)
| Field | Notes |
|---|---|
| Mail server | Like smtp.example.com. Leave empty to send no email. |
| Port | 587 by default. Changing Security switches to its usual port unless you typed another. |
| Security | STARTTLS (usually port 587), TLS (usually port 465), or None (only on a trusted network) |
| User, Password | Leave empty if the server needs none. |
| Send from | The sender address, like staxmax@example.com |
Text messages (Telnyx)
| Field | Notes |
|---|---|
| API key | From the Telnyx portal, under API Keys |
| Send from | A number on your Telnyx account with a messaging profile, in international format, like +15551234567, or an alphanumeric sender ID of up to 11 letters and digits |
| Messaging profile ID | Optional. Give either Send from or this. |
Click Save notifications. Passwords and keys are copied to every node, so any of them can send, and are never shown again. To replace one, type the new value. Remove Telnyx key turns text messages off.
Once saved, Send test email and Send test message send a test to the address or number you type. Unsaved changes are saved first.
Choose who gets which alarms
Back on the Alarms page:
- Under Who gets which alarms, tick Email and SMS for each alarm type. all and none set a whole column.
- Tick Also send a message when a problem clears to be told when it is fixed.
- Under Recipients, add Email addresses and Phone numbers (international format, with the country code). Everyone listed gets every alarm routed to their channel.
- Click Save alarms.
The page warns if email or text messages are not set up yet.
Notifications are sent by one node, the first node in the cluster that is online (named on the page), so each goes out once. Problems only one node knows about (it is cut off, it held back a takeover, or an update it ran failed) are sent by that node itself.
Problems that appear together go out in one message. Each email's subject starts with [staxmax], and the email also lists the problems still open. A text message is cut at 400 characters.
Sent from this node
The Sent from this node list shows the last 50 notifications this node sent, tests included, with the channel, subject, recipients, and the error for any that failed. It starts empty when staxmaxd restarts.