Updates

GEM includes an over-the-air update system that distributes new versions from a central update server to registered installations. The update server also hosts the public installer and distributable artifacts (command sets, web services, database artifacts).

Overview

The update architecture has three components:

1. Update Server (gem_updater) -- Central server at updates.mygem.us that stores and distributes updates, installers, and artifacts
2. GEM Installations -- Production systems that check for and download updates
3. Publish Scripts -- Developer tools that build and push new versions to the update server

How Updates Work

Update Flow

1. A GEM installation sends its installation_id to the update server (POST /check_update)
2. The server verifies the installation is registered and enabled
3. If a newer version exists, it returns the version, checksum, and encryption key
4. The installation downloads the encrypted archive (POST /pull_update)
5. The archive is decrypted and extracted to update the application files

Security

• Each installation has a unique installation_id (32-character MD5 hash)
• Updates are encrypted with AES -- each version has a unique encryption key
• Only enabled installations can download updates
• The update server authenticates publish requests with an update_key
• Each published version carries a bundle integrity hash that the update server registers as the known-good fingerprint for that release (see Bundle Integrity Registry)

Bundle Integrity Registry

The server build's embed-bundle-hash step writes a hash of the compiled controller bundle to build-artifacts/bundle-hashes.json, keyed by version. When publishing, publish-update.js reads the hash for the version being pushed and sends it to the update server in the X-Gem-Bundle-Hash header. The update server records it as the known-good fingerprint for that release, letting it flag controllers that report in with a tampered or unrecognized bundle.

The registry is sparse by design -- if no hash is found for a version (the build artifact is missing), the upload still succeeds and that release is simply treated as unverified. The publish log notes which case applied:

[PUBLISH] bundle hash for 2.0.4602: a1b2c3d4e5f6...

or, when no hash is available:

[PUBLISH] no bundle hash found for 2.0.4602 — integrity registry will be sparse for this release

Installation Registration

Auto-Discovery

When a new GEM installation checks for updates for the first time:

1. The update server creates a disabled entry for that installation_id
2. The check returns 401 Unauthorized (no update is served)
3. An administrator must enable the installation in the update server admin panel

Managing Installations

Access the update server admin panel at https://updates.mygem.us/admin.

For each installation you can:

• Enable/Disable -- Control whether the installation receives updates
• Name -- Assign a friendly name for identification
• View creation and last-updated timestamps

Duplicate installations (same installation_id) are automatically cleaned up on server startup.

Update Server

The update server (gem_updater) is a standalone Express.js application running on port 3131.

Architecture

Component	Purpose
app.js	Express server, routes, middleware
lib/updates.js	Update/installer upload handling, distribution routes
lib/admin.js	Admin panel authentication and management routes
lib/artifacts.js	Artifact storage, listing, search, and distribution
lib/data.js	JSON file database with in-memory indexes and LRU cache
data/database.json	Persistent storage (installations, updates, artifacts, admin config)

Data Storage

The update server uses a JSON file database (data/database.json) rather than PostgreSQL. It features:

• In-memory indexes for O(1) lookups by ID, version, and installation_id
• LRU cache for frequently accessed installations
• Debounced writes to minimize disk I/O
• File watcher that reloads data on external changes
• Automatic deduplication of installation records on startup

Admin Panel

Password-protected web interface at /admin:

• Installations -- View, enable/disable, and name registered installations
• Updates -- View and delete published update versions
• Artifacts -- View uploaded artifacts (command sets, web services, database artifacts)
• Password -- Change the admin password

The admin password is auto-generated on first run and printed to the console. Sessions expire after 24 hours.

API Endpoints

Public:

Method	Path	Description
GET	/install.sh	Dynamic install script (downloads latest installer)
GET	/installers/:version	Download installer by version
GET	/artifacts/types	List available artifact types

Authenticated (installation_id required):

Method	Path	Description
POST	/check_update	Check if a new version is available
POST	/pull_update	Download an update archive
POST	/:type/list	List artifacts of a given type
POST	/:type/download/:filename	Download a specific artifact
POST	/:type/search	Search artifacts by tags or text query

Authenticated (update_key required):

Method	Path	Description
POST	/push_update	Upload a new update (.tar.gz.enc, max 500 MB). Optional X-Gem-Bundle-Hash header registers the bundle integrity fingerprint for the version
POST	/push_installer	Upload a new installer (.run, max 500 MB)

Artifacts

The update server distributes three types of artifacts to GEM installations:

Type	Directory	Description
command_set	data/command_sets/	Device command set JSON files
web_service	data/web_services/	Web service integration JSON files
database_artifact	data/database_artifacts/	Database seed/migration JSON files

Artifacts are JSON files with metadata stored in the database (name, description, version, tags, user, rating). Files on disk are auto-discovered and registered on server startup.

Artifact Management

Upload (admin panel):

• Upload .json files with type, metadata, and optional overwrite flag
• Maximum 10 MB per file

Search (API):

• Filter by tags (OR matching)
• Text search across name, description, and filename
• Results include file size and modification date

Publishing Updates

Prerequisites

Your gem.json must contain:

{
    "update_url": "https://updates.mygem.us",
    "update_key": "your-secret-key"
}

The update_key must match the key stored in the update server's data/database.json.

Publish an Update

node scripts/publish-update.js

This:

1. Requires a clean working tree, then runs npm run lint and the unit suite as a release gate — a failure aborts the publish before anything is built, versioned, or uploaded, leaving no version commit, tag, or published artifact
2. Runs npm run build (version bump + full build)
3. Creates an encrypted .tar.gz.enc archive of dist/
4. Reads the bundle integrity hash for this version from build-artifacts/bundle-hashes.json (if present)
5. Uploads to /push_update with the encryption key and bundle hash in headers
6. The server stores the archive and records the version, checksum, encryption key, and bundle integrity hash
7. Pushes the version commit and tag to origin (the pre-push hook is skipped here — the release gate already validated this exact tree, and a flaky hook failure at this point would leave the release published but master unpushed)

Publish an Installer

node scripts/publish-installer.js

This:

1. Runs npm run build:installer (creates .run file via makeself)
2. Uploads to /push_installer with the version in headers
3. The server stores the installer and records the version and checksum

The public install.sh endpoint automatically serves the latest installer version.

Configuration

GEM Installation Settings

In gem.json:

Key	Description
update_url	URL of the update server (e.g., https://updates.mygem.us)
update_key	Secret key for publishing updates

Update Server Settings

In data/database.json:

Key	Description
update_key	Secret key that must match publish requests
admin.password	Admin panel password (auto-generated on first run)
installations[]	Registered GEM installations
updates[]	Published update versions with checksums and encryption keys
installers[]	Published installer versions with checksums
artifacts[]	Artifact metadata records

Troubleshooting

Installation Not Receiving Updates

1. Verify the installation_id is registered on the update server
2. Check that the installation is enabled in the admin panel
3. Confirm update_url in gem.json is correct
4. Test network connectivity to the update server

Publish Fails

Error	Solution
Lint or unit tests fail (release gate)	Fix the failures and re-run — nothing was versioned or uploaded
working tree is not clean	Commit or stash local changes before publishing
Missing update_url or update_key	Add both to gem.json
401 Unauthorized	update_key doesn't match the server's stored key
Upload timeout	Large file or slow connection -- timeout is 5-10 minutes
dist/ not found	Run npm run build before publishing

Node Runtime Bump Skipped

Some releases also bump the bundled Node.js runtime. Fetching the new runtime requires reaching nodejs.org for the tarball and its SHASUMS256.txt checksum, which fails on sites that whitelist only the GEM update endpoint. This step is best-effort: when it can't be reached the update log shows a warning like

node runtime bump to vX.Y.Z skipped: <reason>; continuing update on current node vA.B.C

and the application update still lands and restarts on the current runtime — this is expected, not a failure. The bump retries automatically on the next update. A runtime tarball that is fetched once has its checksum cached so it can be re-verified offline on later updates.

Admin Panel Access

If you've lost the admin password:

1. Check the server console output (password is printed on startup)
2. Or read data/database.json directly: admin.password

Related Documentation

• Installation -- Installing GEM from scratch
• Backup & Restore -- Data protection during updates
• Command Sets -- Managing command set artifacts