Skip to main content

How staxmax works

staxmax turns two or more ordinary Linux machines into a small, self-healing virtualization cluster. It deliberately avoids the heavy parts of a classic cluster: there is no shared storage, no Ceph, and no separate quorum device.

The building blocks​

  • Standalone nodes. Every node runs its own Incus with a local ZFS pool (named stax by default). Nodes are not joined into an Incus cluster; each one keeps working on its own if the others disappear.
  • The staxmax service (staxmaxd) runs on every node. It serves the web UI and API on port 7443, talks to the other nodes, and decides which VMs run where. You can open the UI on any node — they all show the same cluster.
  • A shared VM table. Every node holds a copy of the list of VMs and, for each one, which node owns it (runs it) and which node holds its replica. The nodes exchange this table in their heartbeats every second, so every node knows the whole picture.

Owner and replica​

Each VM has exactly one owner, the node where it runs. A protected VM also has exactly one replica node: the node that keeps a recent copy of its disk and is the only node allowed to start it if the owner fails.

  • In a two-node cluster the replica node is simply the other node.
  • With three or more nodes, the owner picks the node with the most memory left over after the VMs it would already have to take over (and that has the disk room for the copy). See Replication.

VMs are never started by Incus on its own at boot — staxmax decides what starts, so a node that comes back after a failure does not start a second copy of a VM that has already moved.

Replication every minute​

Once a minute (by default), the owner takes a ZFS snapshot of each protected VM and sends only the blocks that changed since the last copy to the replica node. The copy on the replica node stays stopped until it is needed.

Failover​

Nodes send each other a heartbeat every second over every path they have. A node that has not answered on any path for 5 seconds counts as lost; if it is still silent once the takeover delay (35 seconds by default) has also run out, the replica node of each of its protected VMs starts its copy. Before it does, the cut-off node has already stopped its own VMs, so the same VM never runs twice. How a node decides it is the one cut off is covered in Failover.

Two nodes or more​

Two nodes is the default and the simplest setup: the router (your default gateway) breaks ties when the two nodes lose sight of each other. With three or more nodes, a majority of the nodes decides instead and the router plays no part. Adding a node is a normal action on the Nodes page — see Adding nodes.

What staxmax never does​

  • It never deletes your data on its own. The only things it removes automatically are replicas. A live copy of a VM that lost ownership (for example on a node that comes back after a failover) is stopped and kept as <name>-orphan-<time>, shown as a Preserved copy on the Virtual machines page, in case it holds newer data.
  • Replication never overwrites a VM on the target node that is not a replica.
note

Replication protects against a node failing, not against mistakes inside a VM. A deleted file or ransomware reaches the replica within a minute. Use Backups as well.