Skip to main content

Backup & Restore

The Backup & Restore page provides comprehensive data protection for your GEM installation, including local backups, automatic scheduling, offsite replication, and database maintenance.

Overview​

GEM backups include:

  • Complete database (configuration, programming, history)
  • Device scripts and drivers
  • UI configurations and themes
  • User accounts and permissions
  • System settings and attributes

Critical: Backups are encrypted. Always download and securely store your encryption keys.

Encryption Keys​

Why Encryption Keys Matter​

GEM uses AES-256-GCM encryption to protect sensitive data in backups:

  • Passwords and API keys
  • User credentials
  • Private keys and certificates
  • Custom secure attributes

Without the encryption keys, backups cannot be decrypted.

Downloading Encryption Keys​

The page prominently displays an encryption key warning banner:

First Time (Red Warning):

  • "Encryption Keys Required!" banner
  • Critical warning that backups cannot be decrypted without keys
  • Download Keys button (red)

After Download (Green Confirmation):

  • "Encryption Keys Backed Up" banner
  • Shows last download date
  • Download Again button (green) - get a fresh copy anytime
  • Collapse button (×) - Reduces the banner to a compact pill after confirming you've stored the keys. Click the expand icon on the pill to bring back the full banner at any time. The full warning always shows until keys have been downloaded at least once.

To Download Keys:​

  1. Click Download Keys button in the warning banner
  2. A JSON file (gem-encryption-keys-YYYY-MM-DD.json) downloads
  3. Store this file securely:
    • Password manager (1Password, LastPass, Bitwarden)
    • Encrypted cloud storage (encrypted Google Drive, Dropbox folder)
    • Physical safe or secure location
    • Never store with backups - keep separately for security

Importing Encryption Keys​

Required when restoring backups to new hardware or after system reinstall.

  1. Click "Need to restore a backup from a previous installation?"
  2. The import section expands
  3. Click Choose File and select your encryption keys JSON file
  4. Click Import Keys
  5. System updates to use the imported keys
  6. You can now restore backups from the previous installation

Important: Importing keys overwrites your current encryption keys. This is intentional - it allows your new system to decrypt backups from the old system.

Local Backups​

Viewing Backups​

The backup list shows all available local backups:

  • Filename - Auto-generated: gem-backup-YYYY-MM-DD-HHMMSS.tar.gz
  • Date - Creation timestamp
  • Size - Compressed backup size

Backups are sorted newest-first.

Creating a Backup​

  1. Click Create Backup button
  2. Progress modal shows creation status
  3. Backup appears in the list when complete

Creation Time: Typically 5-30 seconds depending on database size.

What's Included:

  • Full PostgreSQL database dump
  • Device scripts from lib/user_drivers/
  • Custom configuration files
  • Encryption metadata

What's Excluded:

  • System logs (separate retention policy)
  • Temporary files
  • Node modules and application code (reinstalled from source)

Downloading a Backup​

  1. Click Download button next to any backup
  2. Browser downloads the .tar.gz file
  3. Store offsite for disaster recovery

Use Cases:

  • Offsite storage (separate from GEM server)
  • Pre-upgrade snapshot
  • Archival before major changes
  • Migration to new hardware

Restoring a Backup​

  1. Click Restore button next to the backup
  2. Confirmation dialog warns this is irreversible
  3. Click Yes, Restore to proceed
  4. Progress modal shows restore status
  5. System automatically reloads when complete

Restore Process:

  1. System disconnects all clients
  2. Database is dropped and recreated
  3. Backup is decompressed and imported
  4. Services restart
  5. Browser reloads automatically

Important: Restore replaces ALL current data. Create a backup of current state first if needed.

Deleting a Backup​

  1. Click the Delete button (🗑️) next to any backup
  2. Confirm deletion
  3. Backup file is permanently removed

Auto-Backup​

Configure automatic scheduled backups:

Enable Auto-Backup​

Toggle Enable Auto-Backup to activate scheduled backups.

Schedule Options​

  • Hourly - Every hour on the hour
  • Daily - Every day at 2:00 AM
  • Weekly - Every Sunday at 2:00 AM
  • Monthly - First day of month at 2:00 AM

Times use server's local timezone.

Max Backups​

Set the maximum number of backups to retain (local and offsite):

  • Default: 10
  • Range: 1-100
  • Oldest backups are deleted when limit is exceeded

Example: With max_backups=10 and daily schedule:

  • Day 1-10: Accumulates backups
  • Day 11+: Deletes oldest, keeps newest 10

Saving Settings​

Click Save All Settings at the bottom of the page to apply changes.

Cloud Disaster Recovery​

In addition to user-configured offsite backups, GEM automatically pushes a single nightly backup to updates.mygem.us for disaster recovery. This is on by default and runs every night at 3:00 AM as an independent safety net — it does not require auto-backup to be enabled.

Toggle Cloud Disaster Recovery in the Auto-Backup section to opt in or out.

:::warning Turning it off deletes the cloud copy If you save settings with Cloud DR off after it was previously on, GEM asks updates.mygem.us to permanently delete your existing cloud backup. The page shows an inline warning before save so you don't lose this safety net by accident. If you ever need to restore from that copy in the future, leave the toggle on. :::

How it works​

  • Reuses recent backup when available — if a local encrypted backup was created within the last 24 hours (e.g. by the 2 AM auto-backup), the DR job uploads that file
  • Otherwise creates a temporary backup — when auto-backup is disabled or no recent backup exists, the DR job creates a backup, pushes it, then deletes the local copy so disk usage stays minimal for installs that haven't opted into local retention
  • Single slot per installation — overwritten on each push, sized to your most recent backup
  • Authenticated — uses the same Ed25519 client-signature scheme as license checks (signed with your installation's private key)
  • Server-side decryptable — the DR endpoint uses your installation_id as the key index so you can recover a backup even if you've lost the local copy and your encryption keys file
  • Skipped in development mode (NODE_ENV=development)

:::warning Recovery key The recovery key is your installation ID. Without it, the cloud copy cannot be restored. Keep it safe alongside your encryption keys. :::

tip

Cloud Disaster Recovery is a safety net, not a replacement for offsite backups you control. It exists so a fresh-install + DR pull can recover a lost site without third-party storage. For routine offsite redundancy use SFTP, S3, or SMB below.

Offsite Backup​

Protect against hardware failure by replicating backups to remote storage.

Provider Selection​

Choose offsite provider:

  • None (Local Only) - No offsite backup
  • SFTP - Secure FTP to your own server
  • S3-Compatible - Cloud object storage (AWS S3, Backblaze B2, MinIO, Wasabi, etc.)
  • SMB/CIFS - Windows/Samba network share

SFTP Backup​

Replicate backups to any SFTP server (NAS, VPS, dedicated server).

Configuration​

Enable SFTP Backup: Toggle to activate

Connection Settings:

  • Host - SFTP server hostname or IP
  • Port - SSH port (default: 22)
  • Username - SSH username
  • Remote Path - Destination directory (e.g., /backups/gem/)

Authentication Method (tabs):

Password Authentication:

  • Password - SSH password (encrypted)

SSH Key Authentication:

  • Private Key - Paste OpenSSH private key 
    -----BEGIN OPENSSH PRIVATE KEY-----
    b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAA...
    -----END OPENSSH PRIVATE KEY-----
  • Passphrase - Key passphrase if encrypted (optional)

Options:

  • Auto-upload - Automatically upload new backups after creation

Testing Connection​

  1. Fill in connection settings
  2. Click Test Connection
  3. Result shows: ✅ Connection successful! or ❌ Error message

Test before enabling to verify configuration.

Sync Now​

Click Sync Now to immediately:

  • Upload all local backups not on remote server
  • Download remote backups not on local server
  • Respect max_backups limit
  • Remove excess old backups

Remote Backups​

Click List Remote Backups to:

  • View all backups on SFTP server
  • Download specific backups
  • Restore directly from remote
  • Delete remote backups

Remote Backup Actions:

  • Download - Copy to local storage
  • Restore - Download and restore in one step
  • Delete - Permanently remove from remote server

S3-Compatible Backup​

Replicate backups to any S3-compatible cloud storage service.

Supported Providers​

ProviderEndpoint RequiredNotes
Amazon S3No (uses default)Standard AWS S3
Backblaze B2YesUse S3-compatible API endpoint; region must match bucket
WasabiYes (pre-filled)Hot storage, no egress fees
MinIOYesSelf-hosted; enable Force Path Style
OtherYesAny S3-compatible API

:::note Migrating from Backblaze B2 If you previously used the B2-specific integration, GEM automatically migrates your configuration to the S3-compatible format on startup. You will need to set the Endpoint and Region from your Backblaze bucket settings before auto-upload will work. :::

Configuration​

Enable S3 Backup: Toggle to activate

Provider: Select your storage provider. This pre-fills the endpoint and adjusts guidance.

Credentials:

  • Access Key ID - S3 access key (IAM user for AWS, Application Key ID for B2)
  • Secret Access Key - S3 secret key (encrypted)
  • Bucket - Target bucket name

Connection:

  • Endpoint - Provider's S3 API URL (not needed for AWS)
  • Region - Bucket region (e.g., us-east-1)
  • Remote Path Prefix - Organize by installation (e.g., backups/)
  • Force Path Style - Enable for MinIO and some self-hosted providers

Options:

  • Auto-upload - Automatically upload new backups after creation

Testing Connection​

  1. Fill in credentials and bucket info
  2. Click Test Connection
  3. Result shows: ✅ Connection successful with bucket details or ❌ Error

Sync Now​

Click Sync Now to upload/download backups and enforce max_backups limit.

Remote Backups​

Click List Remote Backups to view, download, restore, or delete S3-stored backups.

SMB/CIFS Backup​

Replicate backups to a Windows or Samba network share (NAS devices, Windows servers, etc.).

:::warning Requirement The smbclient package must be installed on the GEM server. Install with: sudo apt install smbclient :::

Configuration​

Enable SMB Backup: Toggle to activate

Connection Settings:

  • Host - Server hostname or IP address (e.g., 192.168.1.100 or nas.local)
  • Share Name - Network share name (e.g., Backups)
  • Username - Login username
  • Password - Login password (encrypted)
  • Domain - Windows domain or workgroup (optional, e.g., WORKGROUP)
  • Remote Path - Directory within the share (default: /gem/)

Options:

  • Auto-upload - Automatically upload new backups after creation

Testing Connection​

  1. Fill in connection settings
  2. Click Test Connection
  3. Result shows: ✅ Connection successful or ❌ Error with details

Sync Now​

Click Sync Now to upload local backups to the share and enforce max_backups limit.

Remote Backups​

Click List Remote Backups to view, download, restore, or delete SMB-stored backups.

Offsite Backup Best Practices​

  1. 3-2-1 Rule: 3 copies, 2 different media, 1 offsite

    • Local: On GEM server
    • SFTP or SMB: Network-attached storage
    • S3: Cloud storage

  2. Test Restores: Periodically verify backup integrity by restoring to a test system

  3. Multiple Destinations: Use both local network (SFTP/SMB) and cloud (S3) for redundancy

  4. Monitoring: Check sync success regularly (setup monitoring alerts)

  5. Encryption: Backups are encrypted before upload - keep keys separate

  6. Bandwidth: Consider bandwidth for initial upload and daily syncs

Database Maintenance​

Automatically clean old historical data to maintain performance.

Configuration​

Enable Auto-Maintenance: Toggle to activate

Retention Period: Days to keep data (default: 90)

What Gets Cleaned​

Database maintenance removes old records from:

  • Attribute History - Historical attribute value changes
  • Request History - API and command logs
  • Access Log - User login and access events

Not Affected:

  • Current configuration (devices, zones, macros, etc.)
  • User accounts
  • Programming
  • Current attribute values

Running Maintenance​

Automatic: Runs daily at 3:00 AM (server time) when enabled

Manual: Click Run Maintenance Now to execute immediately

Result Display:

✅ Maintenance complete! Deleted 15,234 old records:
• 10,456 attribute history
• 3,891 request history
• 887 access log

Retention Guidelines​

Recommended Settings:

  • Production Systems: 90-180 days
  • High-Traffic Systems: 30-60 days
  • Compliance Requirements: Per regulation (may be 1-7 years)
  • Development Systems: 7-30 days

Performance Impact:

  • 1M+ records: Noticeable slowdown on queries
  • 10M+ records: Significant performance degradation
  • Regular maintenance prevents accumulation

Backup Scenarios​

Pre-Upgrade Backup​

Before upgrading GEM:

  1. Create a manual backup
  2. Download the backup file
  3. Store separately from server
  4. Perform upgrade
  5. If issues occur, restore backup

Migration to New Hardware​

  1. On old system:

    • Create a backup
    • Download encryption keys
    • Download the backup file

  2. On new system:

    • Install GEM
    • Import encryption keys
    • Upload or restore backup
    • Verify all functionality

Disaster Recovery​

  1. Install GEM on new hardware
  2. Import encryption keys
  3. Restore backup from:
    • Downloaded backup file (upload manually)
    • SFTP server (list & restore)
    • S3 cloud storage (list & restore)
    • SMB network share (list & restore)
  4. Verify system operation
  5. Update network-dependent configuration (IP addresses, etc.)

Regular Archival​

For long-term archival:

  1. Enable auto-backup with appropriate schedule
  2. Configure offsite backup (SFTP or B2)
  3. Periodically download backups for offline storage
  4. Document backup dates and encryption key location
  5. Test restore process annually

Troubleshooting​

Backup Creation Fails​

Symptoms: Backup creation hangs or shows error

Causes:

  • Insufficient disk space
  • Database connection timeout
  • File permission issues

Solutions:

  1. Check free disk space: df -h
  2. Verify PostgreSQL is running: systemctl status postgresql
  3. Check backup directory permissions
  4. Review logs: Insights > Logging

Restore Fails​

Symptoms: Restore process errors or hangs

Causes:

  • Corrupted backup file
  • Mismatched encryption keys
  • PostgreSQL version mismatch
  • Insufficient disk space

Solutions:

  1. Verify backup file integrity (try downloading again)
  2. Confirm encryption keys are correct
  3. Check PostgreSQL version compatibility
  4. Ensure adequate free space
  5. Check logs for specific error messages

SFTP Connection Fails​

Symptoms: ❌ Connection failed error

Common Issues:

  1. Wrong Credentials: Double-check username/password
  2. Key Format: Ensure private key is OpenSSH format (not PuTTY .ppk)
  3. Firewall: Verify port 22 (or custom port) is open
  4. Permissions: Remote path must be writable
  5. Host Key: Some servers require host key verification

Debugging:

  • Test connection from command line: sftp username@host
  • Verify path exists and is writable
  • Check SSH server logs on remote system

S3 Connection Fails​

Symptoms: ❌ Connection failed error

Common Issues:

  1. Invalid Credentials: Verify access key ID and secret access key
  2. Bucket Not Found: Confirm the bucket name is correct and exists
  3. Wrong Endpoint/Region: For non-AWS providers, the endpoint and region must match your bucket configuration
  4. Permission Denied: Ensure the credentials have read/write access to the bucket
  5. Network Issues: Verify internet connectivity

Provider-Specific Tips:

  • Backblaze B2: The endpoint region (e.g., us-west-004) must match the bucket's region. Find this in your B2 bucket settings.
  • MinIO: Enable Force Path Style in configuration
  • AWS: Ensure IAM user has s3:GetObject, s3:PutObject, s3:DeleteObject, and s3:ListBucket permissions

SMB Connection Fails​

Symptoms: ❌ Connection failed error

Common Issues:

  1. smbclient Not Installed: Install with sudo apt install smbclient
  2. Login Failed: Double-check username and password; try including the domain
  3. Share Not Found: Verify the share name matches exactly (case-sensitive)
  4. Host Unreachable: Verify the host address and that SMB ports (445) are accessible
  5. Remote Path Not Found: The directory within the share may not exist yet (GEM will try to create it)

Debugging:

  • Test from command line: smbclient '//host/share' -U 'username'
  • Verify share is accessible from the GEM server's network

Encryption Key Issues​

Lost Keys: Cannot decrypt backups without keys - prevention is critical

Wrong Keys: Importing wrong keys will cause restore to fail with decryption errors

Backup Keys: Always have multiple secure copies:

  • Primary: Password manager
  • Secondary: Encrypted cloud storage
  • Tertiary: Printed hardcopy in physical safe

Security Considerations​

  1. Key Storage: Never store encryption keys in the same location as backups

  2. Access Control: Limit who can download backups and keys

  3. Network Security: Use SSH key authentication for SFTP when possible

  4. Cloud Security: Use application-specific keys for S3 providers, not master/root keys

  5. Audit Trail: Monitor access to Backup & Restore page

  6. Testing: Regularly test restore process to ensure business continuity

  7. Offsite: Always maintain offsite backups for disaster recovery