Replication

A protected VM is one staxmax copies to a second node so it can be started there if its own node fails. Protection is on by default in the New VM dialog (Protect this VM) and can be switched at any time from the VM's menu on the Virtual machines page (Turn on protection / Turn off protection).

An unprotected VM runs normally, but if its node fails it stays down until that node comes back.

How a copy is made

On every replication run (every 60 seconds by default), for each protected VM the owning node:

1. Takes a ZFS snapshot of the VM's disk.
2. Copies any ISO images attached to the VM to the replica node if they are missing there, and makes sure the VM's VLAN networks exist there.
3. Sends only what changed since the previous snapshot to the replica node.
4. Brings the replica's settings in line with the VM: CPU, memory, disk size, speed limits and priority, network cards, TPM, Secure Boot and legacy BIOS, and snapshot schedule. A VM you resized fails over with its new size, and a removed network card does not come back after a failover.
5. Keeps the three most recent replication snapshots and removes older ones. Your own snapshots travel with the copy, so they are there after a failover; one you delete is removed from the replica too.

Replication works while the VM is running. The copy holds what the guest had written to disk at the moment of the snapshot — the same as if the power had been cut at that moment. Data still in the guest's memory is not copied.

Each node copies up to three of its VMs at once, so a short interval stays achievable with many VMs. Copies that only send changes go first.

A reboot started from inside a VM while a copy of it is running cannot complete until the copy ends, so the VM is left off; staxmax then starts it again.

info

The replication interval is the most recent data you can lose if a node fails. Set it under Settings → Replication → Copy every (seconds). The cluster accepts 15 seconds to 24 hours; 60 seconds is a good default for mini PCs.

The first copy

The first copy of a VM sends its whole disk, which can take a while for a large VM. Until it finishes, the VM shows First copy in progress and cannot fail over yet; the Virtual machines page shows how much has been sent, the speed and about how long is left. A first copy (and any later full copy, such as after restoring a snapshot) is written under a temporary name and only takes the VM's name once it is complete, so a copy that is cut short is never mistaken for a usable replica. A node runs only one full copy at a time; the other VMs' regular copies carry on beside it, so a large first copy does not hold them up.

Replication status

The Protection column on the Virtual machines page shows each VM's state:

Status	Meaning
Synced … ago	The last copy succeeded recently.
First copy in progress	Protected, but no copy has completed yet. It cannot fail over.
Last synced … ago	No successful copy for more than three intervals plus 30 seconds.
Copy failing	The last attempt failed; the error is shown beside the badge. It goes to Recent activity on the Overview only if the next attempt fails too.
Not protected	Protection is off.
Preserved copy	An orphaned copy kept after a failover (see How staxmax works).

Any VM that is not Synced makes the cluster's protection degraded on the Overview page.

Choosing the replica node

In a two-node cluster the replica node is always the other node.

With three or more nodes, the owner picks one replica node per VM:

• It only considers nodes that are online and not in maintenance.
• It skips any node that already holds a live or preserved copy under the VM's name.
• The node must have disk room for the copy (see below).
• Of those, it picks the node with the most free memory after subtracting the memory of the VMs it would already have to start in a failover.

A VM keeps its replica node until that node is declared down (lost for longer than the takeover delay) or retired. The owner then picks a new one straight away and makes a fresh copy there; the old replica is deleted only once the new node holds a good copy.

Disk room and the 10% reserve

Disks are thin, so a copy takes about what the VM's disk actually uses on its owner, not its full size. A node only becomes a replica node — and only receives a first copy — if, after the copy, at least 10% of its ZFS pool stays free. ZFS slows down as a pool fills, and a full pool stops the node's own VMs from writing.

• When a node's pool is 80% used or more, the Overview warns that it is filling up.
• When a node drops below its 10% reserve, later copies to it pause: the affected VMs show Copy failing ("… is nearly out of disk space, so copies to it are paused.") and the Overview shows protection as degraded until space is freed.
• If no node has room for a first copy, the VM shows Copy failing on the Virtual machines page with the reason (for example "No other node has room for a copy (about 40 GiB) while keeping 10% of its disk free.") and cannot fail over until space is freed.

What "protected" requires in practice

For a protected VM to fail over automatically, all of these must hold:

• Its replica is current (status Synced).
• staxmax knows at least one IP address for it, or the owning node has a smart plug configured. Without either, a failover waits for you to confirm the old node is off. See Failover.
• If it is on a VLAN, its replica node has reached it on that VLAN within the last 10 minutes. See Networking.