Troubleshooting
Start with the Overview. The banner says whether your VMs are protected and why not, Needs attention lists open problems, and Recent activity shows what staxmaxd did and why. On a node's shell, journalctl -u staxmaxd -f shows the same and more.
Signing in
The browser warns about the certificate. Each node uses a self-signed certificate. Accept it once per node, or reach the UI through your own reverse proxy.
I was signed out. Sign-in sessions are kept in memory, so restarting staxmaxd (including during an update) signs everyone out of that node. Resetting the password signs out every browser.
Nobody knows the admin password. On any node run sudo staxmax reset-password; see Command line.
Reaching VMs
I can't reach a VM from the node it runs on. That is a limitation of macvlan, the way VMs attach to your LAN: a host cannot talk to its own VMs over that network. Other machines on the LAN, including the other nodes, can. Use the console in the web UI, or connect from another machine.
A VM has no IP address shown. The address comes from the guest agent, or, for guests without it (Windows), from the traffic the VM sends on the LAN. Until one of them reports an address, the VM shows none — and a protected VM with no known IP is not taken over automatically (see below).
Failover
The Overview says "Other node unreachable" (with three or more nodes, "A node is unreachable") and counts down "Takeover in …s". A node has stopped answering heartbeats on every path. When the countdown ends, this node starts the protected VMs it holds replicas of — unless one of the checks below holds it back. The delay is Take over after (seconds) under Settings → Failover.
The countdown ended but a VM was not started. Before starting a VM it doesn't own, a node needs evidence the other copy is really stopped. Recent activity says which check held it back:
| Message starts with | Meaning |
|---|---|
| Not taking over from NODE: the machine still answers at … | The failed node's host still answers ping or ARP: the machine is running, only staxmaxd is not responding. Fix it, or power it off. |
| Not taking over VM: it still answers at … | The VM itself still answers on its last IP, so it may be running on a node that is cut off from you. Retried every 10 seconds. |
| Not taking over VM: it has no known IP address | There is no way to check it stopped. |
| Not taking over VM: it is on VLAN … | This node has not reached the VM on its VLAN lately, so silence proves nothing. |
Once you have checked the other node is powered off or unplugged from the network, open the VM's menu on the VMs page and choose Force start on NODE. It starts from the last synced copy. A smart plug per node lets staxmax switch the failed node off itself and avoids most of these holds; see Nodes and Failover.
Never force-start a VM while the other copy might still be running. Two copies drift apart and the changes on one of them are lost.
A node shows "This node is isolated". It can reach neither the other node(s) nor (in a two-node cluster) the router. After about 16 seconds it stops its own VMs so the rest of the cluster can safely take them over. If it cannot confirm they stopped within a few seconds, it resets itself (through the hardware watchdog if configured, otherwise a forced reboot).
A node shows "Starting up" and its VMs don't start. After staxmaxd starts, a node waits to hear the other nodes before it starts its VMs, so it never runs a VM another node took over meanwhile. In a two-node cluster it gives up waiting after the takeover delay plus 5 seconds if the router answers, and then starts them alone. With three or more nodes it waits until it reaches a majority. Starting a VM there meanwhile is refused with this node is still starting up. Check the node's network and that enough of the other nodes are running.
A node shows "VMs stopped to prevent split-brain" even though its network is back. That is deliberate. A fenced node keeps its VMs stopped until it hears from the other nodes again (or, with three or more, regains a majority), because they have probably taken its VMs over. Restarting staxmaxd or rebooting doesn't clear it. Fix the link between the nodes; the node then rejoins and its old copies are kept as orphans (<name>-orphan-<time>) rather than deleted.
Two nodes keep losing each other though the LAN looks fine. Add a direct cable between spare ports of the nodes as a second heartbeat path — see Networking. If the cable is never found, check that a host firewall lets 7443/udp in on that port; see Ports and files.
Storage
"…'s disk is N% full". Replication to a node pauses once its pool has less than 10% free, and its own VMs may soon be unable to write. Free space (delete old snapshots, orphans or VMs) or add storage. See Replication.
"… does not exist on NODE. Mount the backup share there." Backups go to the same Folder path on every node, and that node has nothing there. Mount the share at that path on every node, or let staxmax mount it (choose NFS share or SMB share under Backup storage). A folder on the node's own disk is accepted with a warning, but a backup there is lost with the node. See Backups.
"The … share is not mounted on NODE: …" staxmax could not mount the backup storage on that node; the reason follows, as the mount reported it. access denied by server (NFS): the export does not allow that node's address, or not as root. Permission denied (SMB): wrong user or password. could not connect / No route to host: the server is off or unreachable from that node. staxmax tries again every minute; systemctl status mnt-staxmax-backup.mount on the node shows the details.
"…, which has the backup drive, is offline." The backups go to a USB drive on another node, and that node is down. The backups are retried an hour later; or move the drive to a node that is up and choose it again.
"the server's host key changed" (offsite, SFTP) The SFTP server shows a different key from the one trusted at the first connection. If you replaced or reinstalled the server, click Forget next to its fingerprint in Settings → Offsite backups and save; otherwise find out why before trusting it.
Joining and removing nodes
See Adding nodes for the messages a join can give.
A node says Removed from the cluster
The other nodes removed it (Nodes → Retire), so it keeps its VMs stopped, and Start and Force start are refused there. To use the machine again, reset it and add it back with a fresh join command:
sudo systemctl stop staxmaxd && sudo rm -rf /var/lib/staxmax && sudo systemctl start staxmaxd
Then, on a node in the cluster, open Nodes → Add a node, choose Show the join command, and use the token under Install it yourself instead: open the reset machine's web page, choose Join an existing cluster and paste it. (The one-line join command is for machines without staxmax installed; on this machine it stops with staxmax is already installed here.)
Updates
A node whose new release fails its health check goes back to its previous release on its own, and the rollout stops there: nodes already updated keep the new release, the rest are not touched. Settings → Updates shows what each node runs. On the node, sudo staxmax-update status shows the last run and sudo staxmax-update rollback goes back to the previous release. See Updates.
An update is only installed if its signature verifies against the node's own trusted keys (/etc/staxmax/keyring.json), so an uploaded .run signed by another key is refused.
"… was not installed from a staxmax .run" (the node shows Not updatable). That node was set up without the installer, so it has no staxmax-update. Run the .run on it once with sudo sh ./staxmax-<version>-linux-x64.run; it keeps the node's setup, VMs and settings.
The log of each install, update and rollback on a node is in /var/log/staxmax/.
staxmaxd does not start
systemctl status staxmaxd shows it restarting, and the journal says why. incus socket not found; is incus installed and running? means Incus is not running: sudo systemctl start incus. If you changed STAXMAXD_OPTS in /etc/staxmax/staxmaxd.env, check it: an unknown or misspelled option stops staxmaxd. See Command line.
Collecting information
systemctl status staxmaxd
journalctl -u staxmaxd --since "1 hour ago"
incus list
staxmax version