Backup & Restore

The Backup & Restore page is where you protect a commissioned system: it creates encrypted local backups, schedules them, replicates them offsite (SFTP, S3-compatible, SMB), pushes a nightly disaster-recovery copy to Harness, and restores any of them back onto the controller.

Open Backup & Restore

Every action on this page requires an admin login with elevated access. Non-elevated users are blocked from listing, creating, downloading, restoring, and deleting backups. Elevation has to come from a password login — a session signed in by PIN is refused here even when the user's role is elevated (see Roles → Elevated).

What a backup contains

A backup is a single encrypted archive holding:

• A full PostgreSQL dump of the GEM database — devices, zones, macros, triggers, UIs, users, roles, attributes, history
• Everything in the GEM install directory: the server, widgets, themes, user drivers, gem.json, and the key files that identify the installation

Excluded from the archive: the backups and updates folders (so a restore never eats your other backups), node_modules, and build/cache scratch folders. Backup archives themselves (gem_backup_*.tar.gz / .tar.gz.enc) are excluded by filename wherever they sit, so a stray archive left in the source directory is never packed inside a new one. Dependencies are reinstalled during a restore rather than carried in the archive.

warning

The archive is encrypted with this installation's backup master key. Download your encryption keys before you need them — see below.

Encryption Keys

What the keys file is

GEM keeps four keys in one file on the controller:

• an attribute key that encrypts secure values inside the database (passwords, API keys, private keys, certificates)
• a backup master key, which encrypts every backup archive
• a recovery code, a printable second way into the same archives (see below)
• a backup key, which is this installation's installation ID

All four live in the keys file you download from this page. Without the backup master key, a downloaded archive is unreadable unless you have the recovery code. Without the attribute key, a restored database loads but every secure value in it stays undecryptable — device passwords, API keys and certificates come back as unusable ciphertext.

The installation ID is no longer a password

Archives written by earlier versions were encrypted directly from the installation ID. That ID is an identifier, not a secret — it appears in update requests, support bundles and the update service's records — so current archives use a separate backup master key that is generated on the controller and never transmitted anywhere.

Archives made before this change still open with the installation ID, and still need it. Nothing you already hold stops working; keep old keys files for old archives.

Downloading the keys

The top of the page shows an encryption-key banner:

• Before download (red): "Encryption Keys Required!" with a Download Keys button
• After download (green): "Encryption Keys Backed Up", the date of the last download, a Download Again button, and a collapse (×) button that shrinks the banner to a compact pill. Click the expand icon on the pill to bring the full banner back.

To download:

1. Click Download Keys (or Download Again).
2. A file named gem_encryption_keys_YYYY-MM-DD.json downloads through the browser.
3. Store it somewhere safe and separate from the backups themselves — password manager, encrypted cloud storage, or a physical safe.

The green banner is per-browser

Whether the keys have been downloaded is remembered by the browser you downloaded them in, not by the controller. On a different browser, a different tech's laptop, or after clearing site data, the red "Encryption Keys Required!" banner comes back even though the keys were already saved. Treat the banner as a reminder, not as proof of what has been archived.

Importing keys from a previous installation

Do this on replacement hardware before restoring an archive from the old system. A fresh install generates its own keys, so it cannot decrypt the old archive until you hand it the old keys.

1. Click Need to restore a backup from a previous installation? in the banner to expand the import panel.
2. Click the file field and pick the keys JSON file from the old installation.
3. Click Import Keys.
4. Confirm the browser prompt warning that this overwrites the current keys.

The file must contain both required keys in the format GEM wrote them (a 64-character hex attribute key and a 32-character hex backup key) or the import is rejected with an error naming the offending field. The backup master key and recovery code are carried across when the file has them; keys files exported before those existed still import cleanly.

On success, this controller adopts the old installation's keys, its recovery code, and its installation ID, and can then decrypt archives from that system.

Recovery Code

Alongside the keys file, each controller issues a recovery code — a short printable string like EXD2H-P5CFT-9V7ZX-PZH8A-K1BMT-G. It appears on the Backup page under Recovery Code.

The code opens any backup this controller has made, including the cloud copy, without the keys file and without contacting Harness. It exists for the case where the hardware and the keys file are both gone and you would otherwise be waiting on a support call.

Because it is short and unambiguous, it is meant to be written down. The alphabet omits letters that are easy to confuse when read aloud, and case, spaces and hyphens are all ignored when you enter it — so a code copied by hand still works.

• Reveal shows it; Copy and Save as Text get it off the screen.
• Issue New Code replaces it. Backups already written keep the old code, so keep the old one until every archive made under it is gone. Rotate if the code may have been seen by someone who should not have it, not as routine housekeeping.

To use it, see Opening an archive by hand below.

Anyone holding it can read your backups

The code is a master key in printable form. Store it the way you would store the keys file — somewhere secure, and somewhere that is not the building this controller is in. It is also included in the keys file download, so the two do not need separate hiding places.

Opening an archive by hand

Restoring through this page is the normal route, and it needs none of this. But an archive is also readable on any machine with Node.js installed — a recovery workstation, a laptop, a VM — using bin/gem-recover.js from a GEM install directory. There is no database, no running GEM service, and no installation ID involved.

Look at an archive without opening it. This needs no key at all and is the right first step when you are not sure what you are holding:

node bin/gem-recover.js --inspect gem_backup_2026-07-27.tar.gz.enc

It prints when the archive was made, which installation it came from, and — most usefully — which keys will open it.

Decrypt it, with whichever key you have:

Option	What it takes
--code <code>	Your recovery code, as written down
--key-file <path>	A downloaded gem_encryption_keys_*.json file
--master-key <hex>	The backupMasterKey value out of that file

node bin/gem-recover.js -i gem_backup_2026-07-27.tar.gz.enc -o backup.tar.gz \
    --code EXD2H-P5CFT-9V7ZX-PZH8A-K1BMT-G

tar -xzf backup.tar.gz

The result is the same .tar.gz the controller made: the database dump and the install directory. To put it back on a controller, copy the still-encrypted .tar.gz.enc into that controller's backups folder and restore it from this page — decrypting by hand is for reading an archive, not for restoring one.

Archives from older versions

An archive written before the format change reports itself as a legacy GEM_ENC_V1 container and gem-recover will say so rather than guess. Those open with bin/decrypt_backup.py and the installation ID they were made under.

Available Backups

The Available Backups panel lists the archives in the controller's backups folder, newest first. Each row shows the filename, its creation date, its size, and a badge for how the archive is sealed. Files are named gem_backup_<timestamp>.tar.gz.enc.

• Verified — an authenticated (v2) container. Its contents are checked before a restore writes anything.
• Unverified — a legacy archive, or a file GEM did not write. It restores only after you explicitly accept that its contents cannot be checked. See Only authenticated archives restore.

The badge is read from the file itself rather than its name, because both formats are named .tar.gz.enc and nothing else on the row distinguishes them. Archives written before the container format changed all show as Unverified — that is expected, not a sign anything is wrong with them.

At the bottom of the panel:

• Refresh — re-reads the folder
• Create Backup — makes a new backup now

Creating a backup

Click Create Backup. A progress overlay titled "Creating Backup" appears and the new file lands in the list when done. Typical run time is seconds to a couple of minutes depending on database size and how much sits in the install directory.

If the selected offsite provider is enabled with Auto-upload on, the new archive is uploaded to it as part of the same run, then the offsite copies are trimmed to Max Backups. An upload failure is logged but does not fail the backup — the local archive is still created.

Local retention is trimmed to ten before each new backup

Every backup run first deletes all but the ten newest local archives, then writes the new one. So the backups folder settles at ten or eleven files no matter how high Max Backups is set. Values below ten are honored exactly, and offsite copies honor the full Max Backups value. If you need deeper local history, download the archives you want to keep or replicate them offsite.

Downloading a backup

Click Download on any row. The file streams to the browser through a progress overlay titled "Downloading Backup". Keep a copy off the controller — a backup that only exists on the machine it protects is not a backup.

Deleting a backup

Click the red delete button on the row and confirm the browser prompt. The file is removed from the controller immediately; offsite copies are untouched.

Restoring

Each row offers two restore modes:

• Restore — full restore. Replaces the installed files and the database, reinstalls dependencies, and restarts the service.
• Restore DB — database only. Loads the archive's database dump back into PostgreSQL and leaves every file on disk alone (server, widgets, on-disk settings all stay as they are). Use this to roll back configuration and data without touching the installed application.

To restore:

1. Click Restore or Restore DB.
2. The confirmation dialog reflects the mode — "Confirm Restore" or "Confirm Database Restore" — and states that the action cannot be undone.
3. Click Yes, Restore (or Yes, Restore Database).
4. A progress overlay titled "Restoring Backup" reports each step.
5. The service restarts and the browser reloads on its own once the database has been loaded.

Restore only runs on a production install

The controller refuses to restore unless the GEM service was started in production mode. On a development or source-run instance the request returns "restore only runs in production" and nothing is touched.

Only authenticated archives restore

A restore extracts an archive over the install directory and then reinstalls dependencies, so whatever is inside it ends up running as the service account. GEM therefore checks that an archive is an authenticated (v2) container — decided by the file's own contents, not its name — before it will restore it. Anything else is refused with a message saying its contents cannot be verified.

That covers the archives you did not personally place on the controller. The offsite restore paths download whatever the remote store currently holds into the local backups folder and then restore it, so a NAS share or bucket anyone can write to would otherwise be a way to hand the controller a tampered archive. The legacy .enc format has no integrity check at all — it can be modified without detection — and a plain .tar.gz has none either.

An older archive you know to be good can still be restored. How you are asked depends on where it lives:

• Local archives carry a Verified or Unverified badge in the list, so you can see which of your archives this applies to long before you need one. Restoring an Unverified row shows the usual confirmation with a warning block explaining what cannot be checked, and the confirm button reads Accept the Risk and Restore.
• Offsite archives (SFTP, S3, SMB) cannot be judged until they have been downloaded, so the restore starts normally and stops when the controller sees what arrived. A second dialog, Restore an Unverified Archive?, names the file and asks you to accept it; the restore then continues with the copy already downloaded.

Both paths record the decision in the system log along with the address of the operator who made it. Use it only for an archive whose whole history you can account for — and if that archive is the one you are relying on, make a fresh backup as soon as the site is healthy so your most recent recovery point is an authenticated one.

Dependencies are also reinstalled with package lifecycle scripts disabled, since the package.json being installed against came out of the archive.

What a full restore does

Three things in that sequence matter on a job site:

• The archive is checked before anything is deleted. A truncated, half-downloaded, or database-only archive fails the check and the running installation is left completely intact, with an error telling you to use a database-only restore instead.
• This controller's database connection survives the restore. The archive carries the source host's gem.json, so its database credentials would otherwise be written over this box's. GEM captures the current connection settings first and writes them back over the restored file — everything else in gem.json comes from the backup. That is what makes restoring onto replacement hardware work.
• GEM refuses to restart into an install that cannot boot. Dependencies are reinstalled after extraction (this is the slow step — several minutes, and it needs internet access to fetch packages). If that install fails, or if the extracted tree is missing anything required to start, the restore stops and does not restart the service. The already-running instance keeps serving from the code it loaded at startup, so the building stays live while you sort it out from a shell.

What a database-only restore does

1. Only the database dump is pulled out of the archive, into a temporary folder. No file on disk is cleared or overwritten.
2. A safety-net dump of the current database is written to a private temporary directory, and its path logged.
3. The GEM schema is dropped and rebuilt from the archive's dump.
4. The load is verified to have produced tables.
5. The service restarts so it reloads from the restored database, and the browser reloads.

note

A database-only restore needs the archive to actually contain a database dump; if it doesn't, the restore aborts with "backup contains no database dump". For a full restore, an archive with no dump still restores the bundled files, leaves the database as it was, and does not restart the service.

Secure values follow the attribute key, not the backup

A database-only restore does not replace the keys file on disk. If you load a database from another installation onto a controller with different keys, every encrypted value in it (device passwords, API keys, certificates) will fail to decrypt. Import the source installation's keys first.

Recovering from a failed restore

Before either restore mode replaces the database, GEM dumps the current one to a private temporary directory and logs the exact path. If the restore aborts — for example with "restore produced no tables" — that plain SQL dump is still there and can be loaded back with psql. Read the path out of the system log rather than guessing at it: the directory is created per run with a random name and is readable only by the service account, because the dump is a complete copy of the site's configuration.

Getting an archive onto a replacement controller

There is no upload control for backup archives on this page — the file picker in the banner is for the keys file only. To restore an archive onto a fresh install, either:

• copy the .tar.gz.enc file into the new controller's backups folder (SCP, SFTP, USB), then refresh the page and restore it from the list; or
• configure the offsite provider that holds it, click List Remote Backups, and restore straight from there.

Auto-Backup

The Auto-Backup panel schedules unattended backups.

Control	What it does
Enable Auto-Backup	Turns scheduled local backups on. Schedule and Max Backups only appear while this is on.
Schedule	Hourly, Daily (2 AM), Weekly (Sunday 2 AM), or Monthly (1st at 2 AM), in the controller's local time. Hourly runs on the hour.
Max Backups	How many copies to keep, 1–100. Applied to the offsite copies after each upload, and to the local folder subject to the ten-file trim described above.
Cloud Disaster Recovery	The nightly push to Harness described below. On by default.

Click Save All Settings at the bottom of the page to apply anything you changed on this page — auto-backup, offsite, and database-maintenance settings are all saved by that one button.

Cloud Disaster Recovery

Separate from the offsite providers you control, GEM pushes one encrypted copy of your backup to updates.mygem.us every night at 3:00 AM. It is on by default and does not require auto-backup to be enabled.

Toggle Cloud Disaster Recovery in the Auto-Backup panel to opt in or out.

How it works

• Reuses a recent local backup when there is one — if an encrypted backup was created in the last 24 hours (for example by the 2 AM auto-backup), that file is what gets pushed.
• Otherwise creates one for the push — with auto-backup off, the job makes a backup, pushes it, and deletes the local copy afterward, so installs that never opted into local retention don't accumulate files.
• Two generations per installation — the latest push and the one it displaced. A backup that turns out to be bad does not take the last good one with it.
• Given its recovery path on the way out — the Harness recovery wrap is added to the copy being uploaded, not to the archive on disk. The local file stays openable by your keys and your recovery code only.
• Signed — the push is authenticated with the installation's own signing identity, the same scheme used for license checks.
• Skipped in development mode.

Pushing on demand

With Cloud DR on, Push To Cloud Now runs the nightly job immediately instead of waiting for 3 AM. Use it to confirm the offsite path actually works on a new install, and to close the gap on a site that has missed its window.

It runs the same job the schedule does, by design — a check that took its own route could pass while the scheduled push stayed broken. That means it follows the same rules: an encrypted backup from the last 24 hours is reused if one exists, and otherwise a fresh backup is created for the push, which can take several minutes on a large site. Progress appears under the button, and the result says how many megabytes went up, whether an existing backup was reused, and whether the recovery wrap was applied.

If Cloud DR is switched off the push is refused rather than performed — otherwise it would silently recreate a copy you chose to delete.

Checking that the push is working

Each attempt records its outcome as read-only system attributes, so a site that has quietly stopped pushing can be spotted before it matters. Find them on the Attributes page under System Target: system:

Attribute	Meaning
dr_backup_last_attempt	When the job last ran, successful or not
dr_backup_last_success	When a copy last landed at updates.mygem.us
dr_backup_last_error	Why the last attempt failed. Blank means no current problem
dr_backup_escrowed	Whether the last successful push carried the Harness recovery wrap

A single failed night is not worth chasing — the job runs again the next night and usually fixes itself. A last_success that is days behind last_attempt is worth chasing, because it means there is no current offsite copy.

These only describe pushes that were attempted

A controller that is powered off, or whose scheduler has stopped, records nothing at all — so an old last_attempt is as much of a signal as an error. The absence of pushes is noticed at the update service, which is the only side in a position to see it.

Turning it off deletes the cloud copy

Saving with Cloud DR off after it had been on asks the update service to permanently delete every cloud copy, both generations. The page shows an inline warning before you save so this doesn't happen by accident. If you might ever want that copy, leave the toggle on.

Restoring from the cloud copy

With Cloud DR on, the panel offers Restore From Cloud with two buttons:

• Latest — downloads the most recent cloud copy.
• Previous — downloads the copy the last push displaced. Use this if the newest backup turns out to be bad.

The download lands in this controller's backup list as gem_backup_cloud_dr.tar.gz.enc (or ..._previous.tar.gz.enc) and restores exactly like any other archive — nothing is overwritten by the download itself, and the restore still goes through the usual confirmation. Its checksum is verified against what the update service recorded at push time before it is accepted, so a truncated transfer fails here rather than part-way through a restore.

Re-downloading replaces the previous download rather than piling up copies.

This needs the controller's keys

The request is signed with this installation's own signing identity, so it works while the controller still has its keys — a rebuilt box that imported them counts. Knowing the installation ID is not sufficient. If the hardware is gone entirely, recovery goes through Harness support instead; see below.

How the cloud copy can be recovered

The cloud copy is encrypted before it leaves the controller, and it is the only copy that carries a recovery path beyond your own keys. Three things can open it:

• your keys file — the normal route, if you still have it
• your recovery code — shown on this page, opens the archive with nobody else involved
• Harness's offline recovery key — held offline, never stored on the update service

That last one is what stops a lost server from becoming a lost site. It is a support ceremony rather than a button: identity is confirmed through account ownership, not by knowing an installation ID, and every use is recorded. Reach for it only when the controller is gone and you have neither your keys file nor your recovery code — the first two routes need no support call, and Restore From Cloud above covers the case where the controller is still running.

Your local archives are not recoverable this way

Only the cloud copy carries the offline recovery wrap. Archives sitting on the controller or on your own offsite storage open with your keys file and nothing else — which is the point: a stolen local archive is not openable by anyone but you. Keep your keys file.

tip

Cloud DR is a last-resort safety net, not a substitute for offsite backups you control. For routine offsite redundancy use SFTP, S3, or SMB below.

Offsite Backup

Pick a destination in Offsite Provider:

• None (Local Only)
• SFTP — any SSH server (NAS, VPS, dedicated box)
• S3-Compatible — AWS S3, Backblaze B2, Wasabi, MinIO, and anything else speaking the S3 API
• SMB/CIFS — a Windows or Samba share

Only the selected provider's settings are shown, and its fields only appear once its Enable … Backup toggle is on.

Offsite credentials are encrypted at rest

The whole backup-settings blob — SFTP password and key passphrase, S3 secret access key, SMB password — is stored as a secure attribute, encrypted with the installation's attribute key, and is not readable by a non-elevated session. Earlier versions stored it in the clear. Existing settings are re-saved encrypted the first time this page is saved after the update; nothing needs to be re-entered.

Test, List and Sync use what is on screen

The three buttons under each provider act on the values currently typed into the form, so you can verify credentials before committing them. Auto-upload and the nightly jobs use the saved settings — click Save All Settings once the test passes.

The three buttons

• Test Connection — connects and lists the destination folder. A green "✅ Connection successful!" or a red error appears above the buttons for a few seconds.
• List Remote Backups — opens a modal ("Remote SFTP Backups", "Remote S3 Backups", or "Remote SMB Backups") listing what is on the destination, with Download, Restore, Restore DB, and delete on each row. Restoring from here downloads the archive into the local backups folder first, then runs exactly the same restore described above.
• Sync Now — uploads every local archive the destination doesn't already have, then trims the destination to Max Backups. It is upload-only: it does not pull remote archives down. To bring one back, use Download in the remote list.

SFTP

Field	Notes
Host	Hostname or IP of the SSH server
Port	SSH port, default 22
Username	SSH user
Remote Path	Destination directory, e.g. /backups/gem/
Auto-upload	Upload each new backup as it is created

Authentication has two tabs, Password and SSH Key:

• Password — one password field.
• SSH Key — paste an OpenSSH private key (-----BEGIN OPENSSH PRIVATE KEY----- … ), plus a Passphrase if the key has one. A filesystem path to a key on the controller also works.

A private key left in the field wins over the password

The connection uses the private key whenever that field is non-empty, whatever tab is selected, and only falls back to the password when it is empty. If you are switching from key to password auth, clear the Private Key box, or authentication will keep failing against a key the server doesn't accept.

S3-Compatible

Choosing a Provider pre-fills the endpoint and adjusts the on-screen tip:

Provider	Endpoint	Notes
Amazon S3	Field hidden — the AWS default is used	Use an IAM user's access key, not root credentials
Backblaze B2	Pre-filled with https://s3.us-west-004.backblazeb2.com	Edit it to your bucket's region; use the Application Key ID and Key from Backblaze → App Keys
Wasabi	Pre-filled with https://s3.wasabisys.com	Adjust if your bucket is in another region
MinIO	Blank — enter your MinIO server URL	Force Path Style is switched on for you
Other S3-Compatible	Blank — enter the provider's S3 API URL	Enable Force Path Style if the provider needs path-style addressing

Remaining fields: Access Key ID, Secret Access Key, Bucket, Region (e.g. us-east-1), Remote Path Prefix (optional, e.g. backups/), and Auto-upload. Force Path Style is only shown for MinIO and Other.

Migrating from the old Backblaze integration

A configuration saved under the earlier B2-specific integration is converted to the S3-compatible form automatically on startup, with auto-upload left off. Set Endpoint and Region from your Backblaze bucket settings and turn auto-upload back on.

SMB/CIFS

Requirement

smbclient must be installed on the GEM server: sudo apt install smbclient

Fields: Host (192.168.1.100 or nas.local), Share Name, Username, Password, Domain (optional workgroup/domain), Remote Path within the share (default /gem/), and Auto-upload.

Offsite practice

• 3-2-1: three copies, two kinds of media, one offsite. Local on the controller, SFTP or SMB on site storage, S3 in the cloud.
• Test restores. An untested backup is a hope, not a plan. Restore to a spare box or a VM at least once per site.
• Keep keys separate from archives. An archive plus its keys in the same folder is one theft away from being plaintext.
• Mind the first upload. The initial archive is the big one; nightly deltas are not deltas — each push is a whole archive.

Database Maintenance

The Database Maintenance panel on this page is the on/off switch and manual trigger for the nightly cleanup of historical data. It runs daily at 3:00 AM.

• Enable Auto-Maintenance — gates the cleanup entirely. This also gates the manual button: with the toggle off, Run Maintenance Now reports a failure and deletes nothing. It is the same switch the Data Retention page shows, so turning it off here stops that page's scheduled cleanup too.
• Retention Period — a legacy field. It is saved, but the cleanup no longer reads it: each history store now has its own retention period. Set them on the Data Retention page, which is the page that actually controls how long data is kept.
• Run Maintenance Now — runs the whole cleanup immediately.

What actually gets cleaned

Each store is pruned against its own retention setting from the Data Retention page:

Store	Default retention
Attribute History	7 days
Request History	7 days
Access Activity	30 days
Automation History	14 days for successful runs, 180 days for failures
Alarm History	90 days, closed events only — open and unacknowledged latched events are never pruned
Monitor History	30 days
Site Mode History	180 days

The run also sweeps attributes left behind by deleted devices, zones, and other removed objects.

Not touched by maintenance: current configuration (devices, zones, macros, UIs), user accounts, and current attribute values. Only history is pruned.

Result readout

After a manual run, a summary appears next to the button and clears itself after ten seconds:

✅ Maintenance complete! Deleted 15,234 old records:
• 10,456 attribute history
• 3,891 request history
• 887 access log

That readout counts only those three stores. Automation, alarm, monitor, and site-mode history are cleaned in the same run but are not itemized here — check the Data Retention page for each store's live record count and oldest record.

Scenarios

Before an upgrade

1. Create Backup.
2. Download it to a machine that is not the controller.
3. Confirm your keys file is archived somewhere safe.
4. Run the update.
5. If it goes badly, restore the archive.

Migration to new hardware

1. On the old controller: Create Backup, Download it, and Download Keys.
2. Install GEM on the new hardware and configure its database connection so the new box comes up on its own database.
3. On the new controller, expand Need to restore a backup from a previous installation?, import the keys file, and confirm. Without this step the archive cannot be decrypted.
4. Put the archive in the new controller's backups folder (or configure the offsite provider holding it and use List Remote Backups).
5. Click Restore and let the full restore run — it reinstalls dependencies, so give it several minutes and internet access.
6. After the service restarts, verify devices reconnect, and update anything tied to the old machine's addressing.

Disaster recovery

If you still have the keys file:

1. Install GEM on replacement hardware.
2. Import the encryption keys from the lost installation.
3. Restore from whichever copy you still have: a downloaded archive placed in the backups folder, or an offsite copy via List Remote Backups.
4. Verify operation, then correct anything that was specific to the old box.

If the controller is still running and you simply want its cloud copy back, use Restore From Cloud on the Backup page — no support call needed.

If the keys file is gone too, the Harness cloud slot is the way back, and you do not need to know anything about the old installation to use it:

1. Contact Harness support. You will be identified through account ownership — not by producing an installation ID.
2. Support retrieves your cloud copy and, under a recorded ceremony, unwraps it with the offline recovery key. If you kept your recovery code, you can open the archive yourself instead and skip this step.
3. Install GEM on replacement hardware and restore the recovered archive.

The archive contains the old installation's keys, so secure values come back readable once it is restored.

The cloud slot holds two copies, not a history

Each push keeps the latest archive and the one it displaced — enough that a single bad backup doesn't take the last good one with it, and nothing older than that. It is a last resort, not a substitute for keeping your own copies.

Troubleshooting

Backup creation fails

• Check free disk space on the controller (df -h) — the archive is written to /tmp first, then encrypted into the backups folder, so you briefly need room for both.
• Confirm PostgreSQL is running (systemctl status postgresql); the database dump is the first step and fails loudly if it isn't.
• Check the backup folder is writable by the account running GEM.
• Review Logging for the failing step.

"backup does not look like a full GEM install"

The pre-flight check found the archive is missing files a full install must have. The current installation was left untouched. Usually the archive is truncated or was only partially transferred — download or copy it again and compare sizes. If the archive genuinely holds only a database dump, use Restore DB.

Restore extracted files but the dependency install failed

The service was deliberately not restarted; the running instance is still serving the building. From a shell in the install directory, fix connectivity to the package registry, run npm install, then restart the service. Don't restart before dependencies are in place.

"restore produced no tables"

The database dump loaded without creating anything, so the restore aborted. The pre-restore dump is kept in /tmp (gem_db_pre_restore_<timestamp>.sql) and its path is in the log — load it back with psql to return to the previous state. Common causes are a corrupted archive, a database user without rights to recreate the schema, or a PostgreSQL major-version mismatch between the source and this box.

Restored, but device passwords and API keys don't work

The database came from another installation and this controller's attribute key is different, so the secure values cannot be decrypted. Import the source installation's keys file, then restore again.

SFTP connection fails

1. Credentials — re-check username and password.
2. Key format — the key must be OpenSSH format, not PuTTY .ppk.
3. Stale key in the field — clear Private Key if you intend to authenticate with a password.
4. Firewall — port 22 (or your custom port) must be reachable from the controller.
5. Path — the remote path must exist and be writable.

Test from a shell on the controller with sftp username@host to separate a GEM problem from a network or account problem.

S3 connection fails

1. Credentials — verify access key ID and secret.
2. Bucket — confirm the name and that it exists.
3. Endpoint / region mismatch — for non-AWS providers both must match the bucket. Backblaze endpoints carry the region (e.g. us-west-004); the pre-filled value is a starting point, not necessarily yours.
4. Permissions — the credentials need read/write. On AWS that means s3:GetObject, s3:PutObject, s3:DeleteObject, and s3:ListBucket.
5. Path style — enable Force Path Style for MinIO and self-hosted gateways.

SMB connection fails

1. smbclient missing — install it on the controller.
2. Login failed — try including the domain or workgroup.
3. Share not found — the share name is case-sensitive and must match exactly.
4. Unreachable — confirm the host and that SMB (port 445) is allowed from the controller.
5. Remote path — the directory inside the share may not exist yet; GEM attempts to create it.

Test with smbclient '//host/share' -U 'username' from the controller.

Encryption key problems

• Lost keys — your recovery code opens the same archives, so check for that before assuming the worst. If the code is gone too, the cloud copy can still be recovered through Harness support; local and offsite archives cannot be, by design. Prevention is still the strategy — keep the keys file and the code in two different places.
• Wrong keys imported — the archive will fail to decrypt, or the database restores with unreadable secure values. Import the correct file and restore again.
• Where to keep them — a password manager entry, an encrypted copy in cloud storage, and a printed copy in a safe covers most failure modes.

Security notes

• Never store the keys file alongside the archives it opens.
• Limit who holds elevated access; backups and keys are downloadable by anyone who does.
• Prefer SSH key authentication for SFTP, and application-scoped credentials (not root or master keys) for S3.
• Watch who touches this page — see Request History and Logging.
• Test the restore path on a schedule, not just before you need it.

Related Documentation

• Data Retention — the per-store retention settings the nightly maintenance actually uses
• Attributes — how secure attribute values are encrypted
• Updates — the update flow to back up before
• Server — service, SSL/TLS, and host-level settings
• Logging — backup, restore, and maintenance log output