Updates

GEM includes an over-the-air update system that distributes new versions from a central update server to registered installations. The update server also hosts the public installer and distributable artifacts (command sets, web services, database artifacts).

Overview

The update architecture has three components:

Update Server (gem_updater) -- Central server at updates.mygem.us that stores and distributes updates, installers, and artifacts
GEM Installations -- Production systems that check for and download updates
Publish Scripts -- Developer tools that build and push new versions to the update server

How Updates Work

Update Flow

A GEM installation sends its installation_id to the update server (POST /check_update)
The server verifies the installation is registered and enabled
If a newer version exists, it returns the version, checksum, and encryption key
The installation downloads the encrypted archive (POST /pull_update)
The archive is decrypted and extracted to update the application files

Security

Each installation has a unique installation_id (32-character MD5 hash)
Updates are encrypted with AES -- each version has a unique encryption key
Only enabled installations can download updates
The update server authenticates publish requests with an update_key

Installation Registration

Auto-Discovery

When a new GEM installation checks for updates for the first time:

The update server creates a disabled entry for that installation_id
The check returns 401 Unauthorized (no update is served)
An administrator must enable the installation in the update server admin panel

Managing Installations

Access the update server admin panel at https://updates.mygem.us/admin.

For each installation you can:

Enable/Disable -- Control whether the installation receives updates
Name -- Assign a friendly name for identification
View creation and last-updated timestamps

Duplicate installations (same installation_id) are automatically cleaned up on server startup.

Update Server

The update server (gem_updater) is a standalone Express.js application running on port 3131.

Architecture

Component	Purpose
`app.js`	Express server, routes, middleware
`lib/updates.js`	Update/installer upload handling, distribution routes
`lib/admin.js`	Admin panel authentication and management routes
`lib/artifacts.js`	Artifact storage, listing, search, and distribution
`lib/data.js`	JSON file database with in-memory indexes and LRU cache
`data/database.json`	Persistent storage (installations, updates, artifacts, admin config)

Data Storage

The update server uses a JSON file database (data/database.json) rather than PostgreSQL. It features:

In-memory indexes for O(1) lookups by ID, version, and installation_id
LRU cache for frequently accessed installations
Debounced writes to minimize disk I/O
File watcher that reloads data on external changes
Automatic deduplication of installation records on startup

Admin Panel

Password-protected web interface at /admin:

Installations -- View, enable/disable, and name registered installations
Updates -- View and delete published update versions
Artifacts -- View uploaded artifacts (command sets, web services, database artifacts)
Password -- Change the admin password

The admin password is auto-generated on first run and printed to the console. Sessions expire after 24 hours.

API Endpoints

Public:

Method	Path	Description
`GET`	`/install.sh`	Dynamic install script (downloads latest installer)
`GET`	`/installers/:version`	Download installer by version
`GET`	`/artifacts/types`	List available artifact types

Authenticated (installation_id required):

Method	Path	Description
`POST`	`/check_update`	Check if a new version is available
`POST`	`/pull_update`	Download an update archive
`POST`	`/:type/list`	List artifacts of a given type
`POST`	`/:type/download/:filename`	Download a specific artifact
`POST`	`/:type/search`	Search artifacts by tags or text query

Authenticated (update_key required):

Method	Path	Description
`POST`	`/push_update`	Upload a new update (`.tar.gz.enc`, max 500 MB)
`POST`	`/push_installer`	Upload a new installer (`.run`, max 500 MB)

Artifacts

The update server distributes three types of artifacts to GEM installations:

Type	Directory	Description
`command_set`	`data/command_sets/`	Device command set JSON files
`web_service`	`data/web_services/`	Web service integration JSON files
`database_artifact`	`data/database_artifacts/`	Database seed/migration JSON files

Artifacts are JSON files with metadata stored in the database (name, description, version, tags, user, rating). Files on disk are auto-discovered and registered on server startup.

Artifact Management

Upload (admin panel):

Upload .json files with type, metadata, and optional overwrite flag
Maximum 10 MB per file

Search (API):

Filter by tags (OR matching)
Text search across name, description, and filename
Results include file size and modification date

Publishing Updates

Prerequisites

Your gem.json must contain:

{
    "update_url": "https://updates.mygem.us",
    "update_key": "your-secret-key"
}

The update_key must match the key stored in the update server's data/database.json.

Publish an Update

node scripts/publish-update.js

This:

Runs npm run build (version bump + full build)
Creates an encrypted .tar.gz.enc archive of dist/
Uploads to /push_update with the encryption key in headers
The server stores the archive and records the version, checksum, and encryption key

Publish an Installer

node scripts/publish-installer.js

This:

Runs npm run build:installer (creates .run file via makeself)
Uploads to /push_installer with the version in headers
The server stores the installer and records the version and checksum

The public install.sh endpoint automatically serves the latest installer version.

Configuration

GEM Installation Settings

In gem.json:

Key	Description
`update_url`	URL of the update server (e.g., `https://updates.mygem.us`)
`update_key`	Secret key for publishing updates

Update Server Settings

In data/database.json:

Key	Description
`update_key`	Secret key that must match publish requests
`admin.password`	Admin panel password (auto-generated on first run)
`installations[]`	Registered GEM installations
`updates[]`	Published update versions with checksums and encryption keys
`installers[]`	Published installer versions with checksums
`artifacts[]`	Artifact metadata records

Troubleshooting

Installation Not Receiving Updates

Verify the installation_id is registered on the update server
Check that the installation is enabled in the admin panel
Confirm update_url in gem.json is correct
Test network connectivity to the update server

Publish Fails

Error	Solution
`Missing update_url or update_key`	Add both to `gem.json`
`401 Unauthorized`	`update_key` doesn't match the server's stored key
`Upload timeout`	Large file or slow connection -- timeout is 5-10 minutes
`dist/ not found`	Run `npm run build` before publishing

Admin Panel Access

If you've lost the admin password:

Check the server console output (password is printed on startup)
Or read data/database.json directly: admin.password

Installation -- Installing GEM from scratch
Backup & Restore -- Data protection during updates
Command Sets -- Managing command set artifacts

Overview​

How Updates Work​

Update Flow​

Security​

Installation Registration​

Auto-Discovery​

Managing Installations​

Update Server​

Architecture​

Data Storage​

Admin Panel​

API Endpoints​

Artifacts​

Artifact Management​

Publishing Updates​

Prerequisites​

Publish an Update​

Publish an Installer​

Configuration​

GEM Installation Settings​

Update Server Settings​

Troubleshooting​

Installation Not Receiving Updates​

Publish Fails​

Admin Panel Access​

Related Documentation​