Updates

GEM includes an over-the-air update system that distributes new versions from a central update server to registered installations. The update server also hosts the public installer and distributable artifacts (command sets, web services, database artifacts).

Overview

The update architecture has three components:

1. Update Server (gem_updater) -- Central server at updates.mygem.us that stores and distributes updates, installers, and artifacts
2. GEM Installations -- Production systems that check for and download updates
3. Publish Scripts -- Developer tools that build and push new versions to the update server

How Updates Work

Update Flow

1. A GEM installation sends its installation_id to the update server (POST /check_update)
2. The server verifies the installation is registered and enabled
3. If a newer version exists, it returns the version, checksum, and encryption key
4. The installation downloads the encrypted archive (POST /pull_update)
5. The archive is decrypted and extracted to update the application files

One-time re-login after upgrading past 2.0.4827

Socket sessions are now bound to a bearer secret issued at login rather than to the client_id alone. Sessions established before the upgrade carry no such secret, so every browser and wall panel is signed out once on the first connection after the update and must log in again. Have panel PINs to hand before updating a site — a panel with a forgotten PIN cannot be recovered from the panel itself (reset it from Security → Users).

Never-expiring sessions are retired on upgrade

The Session Timeout field on the user editor defaulted to 0 — "never expires" — so any account created without touching it held a session that never ended. Updating sets those accounts to 43200 minutes (30 days) and names every changed account in the update log.

The built-in ui wall-panel account is deliberately excluded and keeps whatever it is set to, because a screen on a wall should not stop working and demand a PIN. Read the log line anyway: if any other listed account drives an in-wall panel, set it back to 0 in Security → Users now — otherwise that panel starts asking for a PIN 30 days later, long after anyone would connect it to this update.

Applying an update from the command line

Clicking the update button in the admin UI is one way to apply an available update. The other is to signal the running service:

sudo systemctl kill -s USR2 gem

The signal runs the same update the button runs — check, pre-update backup, stage, restart. There is deliberately no second update path, since one could pass while the real one was broken.

Two properties make it safe to send across a whole fleet in a loop:

• It checks first. A controller already on the latest build logs [UPDATE] signal: already current, nothing to do and stops there — no backup, no restart. Signalling every site is therefore idempotent.
• It will not overlap. A signal that arrives while an update is already running is logged and ignored.

Progress is written to the service log with an [UPDATE] signal: prefix, so journalctl -u gem -f shows what happened on that box. Nothing new is exposed on the network by this: delivering a signal already requires root on the machine, which is a level of access that could replace the application outright.

SIGUSR2, not SIGUSR1

Node reserves SIGUSR1 for its own debugger. Use USR2.

Applying an update from the admin UI requires an elevated session, as does Restart. Both are among the functions a by-name API grant cannot open — see Functions a by-name grant cannot open.

If dependency installation fails

After the new files are staged, the updater runs npm install. If that step fails — the package registry is unreachable, a native module won't build, the disk is full — the service is not restarted. It keeps running on the code and modules it already had loaded, so lighting, HVAC, access control and AV stay online while you deal with it.

The update log and the progress pane both say so explicitly and name the pre-update backup file:

Dependency install FAILED: <reason>
The service was NOT restarted — it is still running the previously loaded code.
Restore from the pre-update backup before restarting: <backup filename>

Restore that backup before restarting the service. Restarting without doing so brings the process up against a half-installed node_modules, which fails at startup and leaves systemd restarting it in a loop with the site offline. See Backup & Restore.

Security

• Dependency installation runs with --ignore-scripts, so no package's install lifecycle scripts execute. The service runs as root, so a lifecycle script anywhere in the dependency tree would be arbitrary root code on a live building controller. A dependency that genuinely needs a build step is rebuilt by name instead of re-enabling scripts tree-wide. Restore already installed this way
• Each installation has a unique installation_id (32-character MD5 hash)
• Updates are encrypted with AES -- each version has a unique encryption key, generated at publish time and held only by the update server
• Every package is verified at publish time to be openable with the key being published, so a release cannot be distributed that no controller can decrypt
• Only enabled installations can download updates
• The update server authenticates publish requests with an update_key
• Each published version carries a bundle integrity hash that the update server registers as the known-good fingerprint for that release (see Bundle Integrity Registry)

Bundle Integrity Registry

The server build's embed-bundle-hash step writes a hash of the compiled controller bundle to build-artifacts/bundle-hashes.json, keyed by version. When publishing, the release script reads the hash for the version being pushed and sends it to the update server in the X-Gem-Bundle-Hash header — on the installer upload as well as the update, so a controller installed fresh can verify on its first check-in. The update server records it as the known-good fingerprint for that release, letting it flag controllers that report in with a tampered or unrecognized bundle.

The registry is sparse by design -- if no hash is found for a version (the build artifact is missing), the upload still succeeds and that release is simply treated as unverified. The publish log notes which case applied:

[release] bundle hash for 2.0.4602: a1b2c3d4e5f6...

or, when no hash is available:

[release] no bundle hash recorded for 2.0.4602 — the integrity registry will be sparse for this release

Installation Registration

Auto-Discovery

When a new GEM installation checks for updates for the first time:

1. The update server creates a disabled entry for that installation_id
2. The check returns 401 Unauthorized (no update is served)
3. An administrator must enable the installation in the update server admin panel

Managing Installations

Access the update server admin panel at https://updates.mygem.us/admin.

For each installation you can:

• Enable/Disable -- Control whether the installation receives updates
• Name -- Assign a friendly name for identification
• View creation and last-updated timestamps

Duplicate installations (same installation_id) are automatically cleaned up on server startup.

Update Server

The update server (gem_updater) is a standalone Express.js application running on port 3131.

Architecture

Component	Purpose
app.js	Express server, routes, middleware
lib/updates.js	Update/installer upload handling, distribution routes
lib/admin.js	Admin panel authentication and management routes
lib/artifacts.js	Artifact storage, listing, search, and distribution
lib/data.js	JSON file database with in-memory indexes and LRU cache
data/database.json	Persistent storage (installations, updates, artifacts, admin config)

Data Storage

The update server uses a JSON file database (data/database.json) rather than PostgreSQL. It features:

• In-memory indexes for O(1) lookups by ID, version, and installation_id
• LRU cache for frequently accessed installations
• Debounced writes to minimize disk I/O
• File watcher that reloads data on external changes
• Automatic deduplication of installation records on startup

Admin Panel

Password-protected web interface at /admin:

• Installations -- View, enable/disable, and name registered installations
• Updates -- View and delete published update versions
• Artifacts -- View uploaded artifacts (command sets, web services, database artifacts)
• Password -- Change the admin password

The admin password is auto-generated on first run and printed to the console. Sessions expire after 24 hours.

API Endpoints

Public:

Method	Path	Description
GET	/install.sh	Dynamic install script (downloads latest installer)
GET	/installers/:version	Download installer by version
GET	/artifacts/types	List available artifact types

Authenticated (installation_id required):

Method	Path	Description
POST	/check_update	Check if a new version is available
POST	/pull_update	Download an update archive
POST	/:type/list	List artifacts of a given type
POST	/:type/download/:filename	Download a specific artifact
POST	/:type/search	Search artifacts by tags or text query

Authenticated (update_key required):

Method	Path	Description
POST	/push_update	Upload a new update (.tar.gz.enc, max 500 MB). Optional X-Gem-Bundle-Hash header registers the bundle integrity fingerprint for the version
POST	/push_installer	Upload a new installer (.run, max 500 MB)

Artifacts

The update server distributes three types of artifacts to GEM installations:

Type	Directory	Description
command_set	data/command_sets/	Device command set JSON files
web_service	data/web_services/	Web service integration JSON files
database_artifact	data/database_artifacts/	Database seed/migration JSON files

Artifacts are JSON files with metadata stored in the database (name, description, version, tags, user, rating). Files on disk are auto-discovered and registered on server startup.

Artifact Management

Upload (admin panel):

• Upload .json files with type, metadata, and optional overwrite flag
• Maximum 10 MB per file

Search (API):

• Filter by tags (OR matching)
• Text search across name, description, and filename
• Results include file size and modification date

Publishing Updates

Prerequisites

Your gem.json must contain:

{
    "update_url": "https://updates.mygem.us",
    "update_key": "your-secret-key"
}

The update_key must match the key stored in the update server's data/database.json.

Publish a release

npm run publish-release

One command, one version bump, one build, both artifacts: the encrypted update package the fleet applies, and the self-extracting .run installer a fresh site installs from. Both are published under the same version.

This:

1. Requires master and a clean working tree, then runs npm run lint and the unit suite as a release gate — a failure aborts the publish before anything is generated, versioned, built, or uploaded
2. Bumps the version with npm version patch --no-git-tag-version — no commit and no tag yet. The version lifecycle hook regenerates every tracked build input first (attribute registry, zone controls, the bin/create_*.sql schema dumps, and the documentation), so all of it lands in the single commit made at the end
3. Builds the client and server once, then builds the installer from that same output
4. Creates an encrypted .tar.gz.enc archive of dist/, sealed to a random per-version key and nothing else, in the legacy GEM_ENC_V1 container (see Update package format), and moves it out of dist/ immediately
5. Verifies the package a controller will actually receive: the container is GEM_ENC_V1, and the key about to be published really decrypts it. A package that fails either check is not uploaded
6. Reads the bundle integrity hash for this version from build-artifacts/bundle-hashes.json (if present)
7. Uploads the installer to /push_installer, then independently confirms GET /installers/<version> is being served
8. Uploads the update package to /push_update with the encryption key and bundle hash in headers
9. Checks what the server says it stored — both push routes reply with the filename they filed the artifact under, and a mismatch fails the publish instead of reporting a success the release does not have
10. Commits, tags v<version>, and pushes both to origin (the pre-push hook is skipped here — the release gate already validated this exact tree, and a flaky hook failure at this point would leave the release published but master unpushed)
11. Prunes local build output (see Local artifact retention)

Why the order matters

• Generators run before the bump. Every generated file is tracked, so regenerating them inside the version hook puts all of them in front of the single version commit instead of dirtying the tree afterwards.
• The installer is built before the update archive. build-installer.sh copies dist/* wholesale, so an archive sitting in dist/ would be packed inside the .run.
• The installer uploads before the update. The update is what moves the fleet, so it goes last: if the installer upload fails, nothing has shipped.
• Nothing is committed until every requested artifact is live. A failure before that point leaves a dirty tree and nothing else — no phantom version commit, no tag for an artifact that was never built, and no documentation published (the post-commit hook publishes docs/) for a release that does not exist. git reset --hard HEAD is a complete undo; git checkout -- . is not, because the version hook stages what it generated.

If a publish fails after an artifact is already live, the script names what shipped. No commit or tag was created, so that version is not recorded — re-running publishes the next patch version, which supersedes it.

Update package format

An update package and a site backup are made by the same archiver but have opposite requirements, and the difference matters:

	Site backup	Update package
Opened by	The controller that made it	Every other controller
Sealed to	That installation's own backup master key	Only the random key published alongside the package
Container	Current envelope format	Legacy GEM_ENC_V1

The legacy container is deliberate, not leftover. The build that applies an update is the old one already running on the site — old enough to predate the current container format — so a package written in the newer format would download and verify by checksum on every site and then fail to stage, after each of those controllers had already taken a full pre-update backup.

The publish-time verification in step 5 exists to make that failure impossible to ship. It reads the archive header and decrypts the first block using only the key being published, consulting nothing on the publishing machine, so a package sealed to the wrong key or written in the wrong container is caught before upload rather than by every site at once.

Publish a single artifact

npm run publish-update      # update package only
npm run publish-installer   # installer only

Both are thin wrappers over the same release flow — same guardrails, one version bump, one build, commit and tag only after the artifact is live — each skipping the artifact it doesn't ship. Use publish-update for a hotfix that only needs to move the fleet, and publish-installer when a fresh-install fix should go out without moving deployed sites. A normal release is npm run publish-release, which ships both.

The public install.sh endpoint automatically serves the latest installer version.

Local artifact retention

The update server keeps its own copy of every published release. install/ and build-artifacts/ on the build machine are working copies, and a successful release trims them to the newest 10 installers and the newest 100 sourcemaps. The sourcemap window is deliberately much wider: a map is the only way to read a crash report from a site still running that version. build-artifacts/bundle-hashes.json is never pruned — it is the integrity registry for every version ever published.

Run the same prune on demand, without cutting a release:

npm run clean:artifacts

Configuration

GEM Installation Settings

In gem.json:

Key	Description
update_url	URL of the update server (e.g., https://updates.mygem.us)
update_key	Secret key for publishing updates

Update Server Settings

In data/database.json:

Key	Description
update_key	Secret key that must match publish requests
admin.password	Admin panel password (auto-generated on first run)
installations[]	Registered GEM installations
updates[]	Published update versions with checksums and encryption keys
installers[]	Published installer versions with checksums
artifacts[]	Artifact metadata records

Troubleshooting

Installation Not Receiving Updates

1. Verify the installation_id is registered on the update server
2. Check that the installation is enabled in the admin panel
3. Confirm update_url in gem.json is correct
4. Test network connectivity to the update server

Publish Fails

Error	Solution
Lint or unit tests fail (release gate)	Fix the failures and re-run — nothing was versioned or uploaded
working tree is not clean	Commit or stash local changes before publishing
refusing to publish from '<branch>'	Releases go out from master only — switch branches and re-run
server stored '<name>', expected '<name>'	The update server filed the artifact under a different version than the one being released. Treat the release as not published and investigate the server before re-running
Missing update_url or update_key	Add both to gem.json
401 Unauthorized	update_key doesn't match the server's stored key
Upload timeout	Large file or slow connection -- timeout is 5-10 minutes
dist/ not found	Run npm run build:only before publishing
refusing to publish: archive is "…", not GEM_ENC_V1	The archive was written in the current container instead of the legacy one. Nothing was uploaded — the publish script must request the v1 container when creating the package
refusing to publish: the published key does not decrypt this archive / …did not produce a gzip stream	The package was sealed to something other than the key being published (typically the publishing machine's own key). Nothing was uploaded

Node Runtime Bump Skipped

Some releases also bump the bundled Node.js runtime. Fetching the new runtime requires reaching nodejs.org for the tarball and its SHASUMS256.txt checksum, which fails on sites that whitelist only the GEM update endpoint. This step is best-effort: when it can't be reached the update log shows a warning like

node runtime bump to vX.Y.Z skipped: <reason>; continuing update on current node vA.B.C

and the application update still lands and restarts on the current runtime — this is expected, not a failure. The bump retries automatically on the next update. A runtime tarball that is fetched once has its checksum cached so it can be re-verified offline on later updates.

Admin Panel Access

If you've lost the admin password:

1. Check the server console output (password is printed on startup)
2. Or read data/database.json directly: admin.password

Related Documentation

• Installation -- Installing GEM from scratch
• Backup & Restore -- Data protection during updates
• Command Sets -- Managing command set artifacts