Failover
When a node stops answering, staxmax has to answer two questions safely: is that node really down? and if I am the one cut off, should I stop my VMs? Getting either wrong could leave the same VM running on two nodes at once (split-brain), which corrupts its data. This page explains how each node decides.
Heartbeats
Every node sends a heartbeat to every other node every second, over every path it has: the LAN address, any extra addresses you added under Nodes → Settings → Heartbeat paths, and any direct cable it found. A node counts as reachable while any one path answers. With no heartbeat on any path for 5 seconds, it counts as lost.
Heartbeats also carry the shared VM table, the member list and the cluster settings, so every node stays in step.
A single path is a single point of failure: one unplugged cable can look like a dead node. A second path — a cable straight between spare ports, or a second network — is the cheapest way to avoid false failovers.
Who keeps running
Two nodes: the router is the tiebreaker
With two nodes, a node that loses its peer cannot tell whether the peer died or whether it is the one cut off. So both nodes also ping a tiebreaker every 2 seconds — by default your default gateway (the router). Under Settings → Tiebreaker addresses you can list other addresses instead; any one answering counts.
- Hears the peer → normal operation.
- Lost the peer, still reaches the tiebreaker → the peer is the one in trouble. Start the takeover countdown.
- Lost both the peer and the tiebreaker (for 6 seconds) → this node is isolated and must stop its VMs.
If both nodes still reach the router but not each other (for example a firewall blocks staxmax's port), both would try to take over. The probes described below normally stop both, since each machine still answers on the network. On top of that, the secondary node waits 10 seconds longer than the primary, so if one of them does go ahead it is the primary.
Three or more nodes: a majority
With three or more nodes the router plays no part. A node may keep running VMs only while it is in contact with more than half of the nodes (itself included), or exactly half if that half includes the top-ranked node.
A lost node is only taken over once no node that we still hear can hear it either. If another node still reaches it, only the link between two nodes is broken, and the Overview says so instead of starting a takeover.
Fencing: the cut-off node stops its VMs
A node that is isolated (two nodes: no peer and no tiebreaker; more: no majority) waits 10 seconds in case the network returns, then self-fences: it stops every VM it runs. In total that is about 16 seconds after it lost contact (about 15 with three or more nodes, where there is no tiebreaker to wait for).
If its VMs are not confirmed stopped within 5 seconds, it escalates:
- With a hardware watchdog configured, it stops feeding the watchdog, which resets the machine within 10 seconds.
- Without one, staxmax reboots the machine itself.
A fenced node stays fenced — VMs stopped, nothing started — until it hears its peer again (or, with 3+ nodes, regains a majority). This holds even if the router comes back first, and even across a restart of staxmaxd or a reboot, because the other node has most likely taken over its VMs by then.
If staxmaxd itself restarts while the node is cut off and VMs from before the restart are still running, it fences 10 seconds later, exactly as if it had not restarted.
If the machine has a hardware watchdog (/dev/watchdog), add --watchdog /dev/watchdog to STAXMAXD_OPTS in /etc/staxmax/staxmaxd.env and restart staxmaxd. The installer tells you when a watchdog is present. It also reboots the node if staxmaxd itself hangs. If the device is missing or its timeout cannot be set to 10 seconds or less, staxmaxd runs without it, fences by rebooting instead, and raises the alarm A watchdog is missing.
Takeover: the surviving node starts the VMs
Once a node is lost, each surviving replica node counts down the takeover delay (Settings → Failover → Take over after (seconds), 35 seconds by default). The Overview shows Takeover in Ns while it counts. If the node comes back before the countdown ends, nothing happens.
The delay can never be set below 31 seconds: the worst case for the cut-off node to notice, wait, try to stop its VMs and be reset by its watchdog. Any lower value is raised to 31 when saved, and the most it accepts is 3600 (an hour). The countdown starts only after the 5-second heartbeat timeout, which adds a margin on top.
When the countdown ends, the node takes over only the protected VMs whose replica it holds. Before starting any of them it looks for evidence that the old node is still running:
- Smart plug. If the lost node has a smart plug configured (under Nodes → Settings → Smart plug, Shelly only), the survivor switches it off first. That is proof enough, and the probes below are skipped. If the plug cannot be switched off, it falls back to the probes. The plug is left off: switch it back on yourself once you have looked at the node.
- The node's own addresses. If the lost node's machine still answers ping or ARP on any of its addresses (a direct cable included), its staxmaxd is hung but the machine is running. No VM is taken over; the Overview says so, and the check is repeated every 10 seconds.
- Each VM's IP addresses. If a VM still answers ping or ARP on any of its known addresses, it is still running somewhere and is not started. ARP matters for guests such as Windows, which drop ping but always answer ARP. The check is retried every 10 seconds.
A VM is not taken over automatically when:
- staxmax knows no IP address for it (no guest agent and no traffic seen on the LAN), or
- it is on a VLAN the replica node has not reached it on in the last 10 minutes (see Networking).
In both cases silence proves nothing, so staxmax waits for you. Once you are sure the old node is off, use Force start on <node> from the VM's menu on the Virtual machines page — or add a smart plug, which removes this limit.
Just before switching each VM over, the node re-checks that the lost node is still silent and that it still has the right to take over (tiebreaker or majority). Once a VM is taken over, a new replica node is chosen for it right away if one is available. With three or more nodes, VMs that kept running elsewhere but had their replica on the lost node also get a new replica node once its takeover delay has run out.
Typical timeline (two nodes, defaults)
| Time | Cut-off node | Surviving node |
|---|---|---|
| 0 s | Loses the network | — |
| ~5 s | — | Peer lost; tiebreaker still answers; countdown starts |
| ~6 s | Peer and tiebreaker lost: isolated | — |
| ~16 s | Self-fences: stops its VMs | — |
| ~21 s | If VMs did not stop: stops feeding the watchdog (reset by ~31 s), or reboots itself | — |
| ~40 s | — | Countdown ends (50 s on the secondary); probes, then starts the VMs |
Tested on hardware: a node cut off from the network fenced at +17 s, the other node took over at +47 s and the VM was back at +63 s with the same IP address.
When the failed node returns
- If it returns before the countdown ends, nothing moves; it keeps (or, if it had already fenced, restarts) its own VMs.
- If it returns after a takeover, the shared VM table says the VMs now belong to another node. Any copy it still has that was live is stopped and renamed
<name>-orphan-<time>, shown as Preserved copy, in case it held newer data. Delete it once you no longer need it.
After a restart, a node waits until it can hear its peer (or a majority) before starting VMs. A lone node of a two-node cluster whose peer never answers starts its own VMs once the takeover delay plus 5 seconds has passed, as long as it reaches the tiebreaker, and then takes over the other node's protected VMs as usual. With three or more nodes there is no such shortcut: it waits for a majority.
Protection levels
The banner at the top of the Overview sums up the cluster:
| Level | Headline | When |
|---|---|---|
protected | Your VMs are protected | Every node online, every protected VM copied and current, every heartbeat path up. |
degraded | Protection is degraded | Starting up and waiting for the other nodes; or running, but something reduces protection: a VM not protected, not yet copied, failing or out of date; a VM with no known IP and no smart plug; a heartbeat path down; a node's tiebreaker unreachable (two nodes); a node offline (3+ nodes); a node into its disk reserve. |
unprotected | Your VMs are not protected | A single node; the other node offline (two nodes); or this node cut off or removed from the cluster. |
The banner lists the reasons under the headline.
HA states
Next to the banner, the heartbeat state of the node you are viewing:
| State | Shown as | Meaning |
|---|---|---|
single-node | Single node | No other node; VMs run without failover. |
starting | Starting up | Waiting to hear the other node (or a majority) before starting VMs. |
healthy | Heartbeat healthy | In contact with every node (with 3+ nodes, a node only other nodes still reach counts as in contact). |
peer-lost | Other node unreachable / A node is unreachable | A node is lost; the takeover countdown or takeover is shown. |
isolated | This node is isolated | This node is cut off; it stops its VMs in a few seconds unless the network returns. |
fenced | VMs stopped to prevent split-brain | This node stopped (or is stopping) its VMs and waits to hear the cluster again. |
removed | Removed from the cluster | This node was removed and runs no VMs. |