Skip to main content

Backups

Replication is not a backup: a deleted file or ransomware reaches the copy on the other node within a minute. Backups are full exports of a VM, snapshots included, written each night to backup storage you choose (a USB drive, a NAS share, or a folder) and, if you set it up, copied offsite to S3-compatible storage such as Backblaze B2 or Wasabi, or to an SFTP server.

You can use the backup storage, offsite storage, or both.

Choose the backup storage​

In Settings → Backups, under Backup storage, choose one of:

  • USB drive: a drive plugged into one node.
  • NFS share or SMB share: a share on a NAS or another computer, mounted on every node.
  • Folder: a path you mount yourself on every node.
  • None: no backups to a drive or folder (offsite backups still run if they are set up).

Then click Save backup storage. For a USB drive or a share, staxmax mounts it on the nodes that need it, at /mnt/staxmax/backup, and mounts it again after a reboot or if it drops. You don't edit /etc/fstab. Below the choice, each node shows Mounted (with the free space), Mounting, or Not mounted with the reason, such as a wrong password or an export the server refuses. A node that cannot mount the storage raises the A backup failed alarm, and its backups fail until it can; they are never written to the node's own disk instead.

Choosing a USB drive or a share needs every node on the same staxmax version; finish an update first.

Then set the schedule under Schedule: Back up at (an hour, in each node's local time; 02:00 by default) and Keep per VM (1 to 365; 7 by default), and click Save settings.

USB drive​

The list shows the USB drives plugged into every node, with the filesystems on them. Choose the one to use. The drive stays on the node it is plugged into: the other nodes send their backups to that node over the network, and list and restore them through it. Plug the drive into the node with the most room to spare.

Drives formatted as exFAT, NTFS, ext4, XFS or Btrfs work as they are. FAT32 cannot hold files over 4 GB, so it cannot be chosen. Erase… wipes a drive and gives it one ext4 filesystem for backups; type erase to confirm. It cannot be undone, and is not offered for a drive that is in use. The backup drive cannot be passed through to a VM.

warning

While the node with the drive is offline, the other nodes' backups fail (and are retried an hour later). Unplugging the drive while a backup runs ruins that backup. A USB drive next to the nodes does not protect against a fire or a theft: add offsite backups for that.

NFS or SMB share​

Give the Server (a name or address) and the Export (NFS, the full path on the server, such as /volume1/backup) or Share (SMB, such as backup; it can name a folder in the share too, such as backup/staxmax). For SMB, give a User and Password, or leave the user empty to sign in as a guest. The password is copied to every node and never shown again; leave it empty to keep the saved one.

The share must let every node write as root. For NFS, allow every node's address on the export with root access (no_root_squash; on a Synology, Squash: Map root to admin or No mapping). staxmax installs the tools it needs to mount a share (nfs-common or cifs-utils) if a node lacks them.

Folder​

Mount a NAS share (or another drive) yourself at the same path on every node, for example /mnt/backup, in /etc/fstab, and enter that path as Folder.

warning

If the folder is on the node's own disk, the backup list warns that a backup there is lost with the node. Mount a NAS share or another drive there.

If the folder does not exist on a node (the share is not mounted), the list says so, and backups on that node fail until it is there.

Turn backups on for a VM​

Backups are per VM. On Virtual machines, choose ⋯ → Edit, tick Back up every night under More, and click Save.

Each night at the chosen hour, the node running the VM exports it to the backup storage, one VM at a time. The VM keeps running. A node that was off at that hour backs up its VMs when it is back, the same day. A failed backup is retried about an hour later, and raises the A backup failed alarm (see Alarms).

Under the VM's name, the Virtual machines page shows Backed up … ago, Backing up now, Last backup failed (hover for the reason), or Backup scheduled tonight. With offsite storage set up, it also shows Offsite … ago, Copying offsite now, or Last offsite copy failed.

For each VM, staxmax keeps the newest backups up to Keep per VM and deletes older ones. It only touches files it wrote.

Back up now​

Choose ⋯ → Back up now to export a VM right away, in the background. It goes to the same backup storage and counts toward the same retention. Without backup storage, it goes offsite instead.

With offsite storage set up, ⋯ → Back up offsite now sends a copy offsite right away. It counts toward the offsite retention.

⋯ → Download backup is separate: it builds an export and downloads it to your browser, without using the backup storage.

A backup restores into staxmax or Incus only. To open a VM in VMware or VirtualBox, use ⋯ → Download as OVA instead (see Taking a VM to VMware or VirtualBox).

Restore a backup​

The Backups in … list on the Backups tab of the Settings page shows the backups in the backup storage, grouped by VM, newest first. Click Show all N to see more than the newest three, and Refresh to read it again.

  1. Click Restore next to a backup.
  2. Give the Name of the new VM (it defaults to <vm>-restored).
  3. Tick Protect it (keep a copy on another node) if you want it protected.
  4. Click Restore.

The backup becomes a new VM on the node whose page you are on, stopped, with its snapshots. It gets new MAC addresses on every network card and its own identity: on first boot it gets a new machine ID and hostname and reboots once, and a fixed address the original had is dropped in favour of DHCP. So it doesn't clash with the original if that still runs. The original is left alone. Start it from Virtual machines.

Any node can restore any VM's backup: the storage is the same on every node, or, for a USB drive, read through the node it is plugged into.

Offsite backups​

Offsite backups protect against losing the whole site: a fire, a flood or a theft that takes every node and the NAS with them. The VMs that have Back up every night on are also exported to a bucket on S3-compatible storage, or to an SFTP server. Any provider that speaks the S3 API works; Backblaze B2 and Wasabi are the usual choices. For SFTP, any SSH server works: a Hetzner Storage Box, rsync.net, a NAS or a Linux machine at another site.

Each copy is encrypted on the node before it leaves it, with a passphrase you choose. The storage provider never sees the contents.

warning

Write the passphrase down and keep it somewhere other than the nodes. Without it, nobody can read the offsite backups, including you after losing the cluster.

Set up offsite storage on S3​

  1. Create a bucket at your provider, and an access key for it. A key limited to that bucket is best. With Backblaze B2, create an Application Key with Read and Write access to the bucket; its keyID and applicationKey are the access key ID and secret key.
  2. In Settings → Offsite backups, enter:
    • Endpoint: the bucket's S3 endpoint. For Backblaze B2, the Endpoint shown on the Buckets page, such as s3.us-west-004.backblazeb2.com. For Wasabi, s3.<region>.wasabisys.com.
    • Bucket, and Folder in the bucket (staxmax/ by default).
    • Access key ID and Secret key.
    • Region: leave it empty for Backblaze B2, Wasabi, AWS and Cloudflare R2, where it comes from the endpoint. Other providers may need it (us-east-1 is used otherwise).
    • Passphrase, twice: at least 12 characters.
    • Copy every (days) (7 by default) and Keep per VM (4 by default).
  3. Click Save offsite settings.
  4. Click Test connection and speed.

The secret key and the passphrase are copied to every node, so any node can send and restore, and are never shown again. Leave a field empty to keep the saved value.

Set up offsite storage on SFTP​

  1. In Settings → Offsite backups, choose SFTP server. staxmax makes an SSH key for the cluster and shows its Public key.
  2. On the server, add the public key to ~/.ssh/authorized_keys of the user the backups go to (Copy copies it). On a Hetzner Storage Box, use port 23. Instead of the key, you can give a Password; the key is tried first.
  3. Enter the Server, Port (22 by default), User, Folder on the server (staxmax/ by default; without a leading / it is in the user's home folder), the Passphrase twice, Copy every (days) and Keep per VM.
  4. Click Save offsite settings, then Test connection and speed.

The first connection trusts the server's host key and shows its fingerprint; from then on, a server showing a different key is refused, so nobody can pose as it. If you replace or reinstall the server, click Forget next to the fingerprint and save. Changing the server or port forgets it too. New key… replaces the cluster's key; backups stop until the new public key is on the server.

Turn off stops offsite backups and forgets the settings, passphrase included (the SFTP key is kept, since the server still trusts it). What is already stored offsite stays there.

How long a copy takes​

Every offsite copy is a full export, not just the changes: a VM with 30 GB of data sends about 30 GB each time. Home internet connections upload much more slowly than they download.

Test connection and speed writes, lists, reads and deletes a 16 MB test file in the offsite storage from the node whose page you are on. It shows the upload and download speed, and a table of every VM: its disk size, how much data a copy sends, and roughly how long sending it takes. VMs with backups off are listed too, marked not backed up, so you can see what turning them on would cost; the total at the bottom counts only the VMs backed up now. The data size comes from the VM's last offsite copy, else its last backup in the backup storage, else its disk use (an export is usually smaller). If the copies take many hours, copy less often, or turn backups off for VMs that don't need them.

When copies are made​

At the backup hour, the node running each VM sends a copy if none was made in the last Copy every (days) days. With backup storage too, it sends that night's backup from there; without it, it exports the VM straight to the offsite storage, so offsite works without a NAS. VMs are copied one at a time, and the VM keeps running. A failed copy is retried about an hour later, and raises the An offsite backup failed alarm (see Alarms).

For each VM, staxmax keeps the newest copies up to Keep per VM and deletes older ones. It only deletes files it wrote, and only this cluster's.

tip

Check your provider's retention rules. Backblaze B2 keeps every version of a file by default, so deleted copies still take space: set the bucket's lifecycle to Keep only the last version. Wasabi charges each file for at least 90 days, even when it is deleted sooner.

Restore an offsite backup​

With offsite storage set up, the Offsite backups list on the Backups tab of the Settings page shows the copies stored offsite, grouped by VM, newest first. Restore works as for the backup storage (see Restore a backup), except that it runs in the background: the node downloads and decrypts the copy first, so a large VM can take a while. The new VM appears on Virtual machines, stopped, when it is done.

Copies made with an earlier passphrase still restore: staxmax remembers the passphrases you replaced.

After losing the cluster​

The files sit in the bucket (or on the SFTP server) under <folder>/<cluster ID>/<vm>/<vm>_<time>.tar.gz.enc. To get the VMs back on new nodes, set up the new cluster, enter the same bucket or server, folder and passphrase in Settings → Offsite backups (for SFTP, put the new cluster's public key on the server), and restore from the list. The old cluster's copies are listed as another cluster; the new cluster never deletes them.

Without a cluster at all, download a file from the bucket or server and decrypt it on any machine with staxmax installed, then import it into Incus (see Command line).