Recovering an EverNVR recorder

Everything that goes wrong after the install, and what to do about it. For a fresh install, backups configuration and updates, see INSTALL.md.

Contents: what a backup holds · taking a backup by hand · restoring onto a replacement recorder · restoring across releases · pre-migration snapshots · owner account recovery · a failed update · an interrupted update · losing a storage location · logs and the support bundle · when the recorder is unreachable

What a backup holds

Every backup — the nightly one, Back up now, the one an update takes, the one before a migration, and the manual command — is a private directory containing:

• evernvr.db, a consistent SQLite snapshot including committed WAL contents,
• secret.key, the credential encryption key, if present,
• storage.json, the saved storage settings,
• service.env, only when a service environment file was explicitly supplied,
• a manifest with file sizes, SHA-256 checksums and the migrations applied.

The database holds accounts and password hashes, camera credentials, settings, events, the recording index, alert configuration and account history.

The recorder's own certificate authority is included, its key encrypted under the credential key, so a restored recorder stays trusted by the devices that trusted it. Footage, thumbnails, a certificate of the site's own, OS settings and application binaries are not included. Copy every media location, including its hidden .evernvr-storage marker, separately. Save mount, network and certificate configuration separately.

The directory contains the key that decrypts every stored camera credential. Keep it on access-controlled, encrypted backup storage. It is not a redacted support bundle. The checksums detect corruption; they are not signatures and do not make an untrusted backup authentic. Keep the trusted release and its version record alongside it.

Backup passphrase. Under Settings → Backups a permanent owner can set a backup passphrase. From then on every backup — nightly, on demand, before an update, before a migration, and the manual command — carries secret.key wrapped under a key derived from the passphrase (Argon2id, AES-256-GCM; the salt and parameters are in the manifest as keyEncryption), and a restore needs the passphrase. The passphrase itself is stored nowhere: lose it and no backup made under it can recover a stored credential, so keep it with the recovery material, not in the backup folder. It is typed twice, under Backup passphrase and Again, and must be at least 12 characters; Replace the passphrase changes it for backups from then on, and earlier backups keep needing the passphrase they were made under. Without a passphrase the key travels in clear beside the database, and the settings will not save until the owner ticks I understand on that page; Remove passphrase asks for the same acceptance. The acceptance is recorded in the account history, with who gave it and when. A backup made without a passphrase restores exactly as before.

Settings → Backups sets the folder, the hour of the nightly backup, and how many nightly and on-demand backups to keep; the recorder keeps recording while it copies. The folder must be inside the recorder's data folder or on a drive or share registered under Storage. The default, beside the database, protects against a failed update but not a failed disk; a second drive or a NAS share protects against both. A temporary installer can change the hour and the count but not the folder. A backup from an older release restores into a newer one, and the restore applies the database changes it is missing.

Nightly backups are named backup-YYYY-MM-DD-HHMMSS in the recorder's local time; pre-migration snapshots are pre-migration-… in the same format. The backup an update takes is under /opt/evernvr/backups/pre-update-<old version>-<timestamp>/: the database backup in evernvr/, and a copy of /etc/evernvr in config/. The first install over a recorder made by the old installer writes pre-adopt-<version>-<timestamp> there instead. The last three are kept.

Taking a backup by hand

The command-line procedure is for a stopped recorder: a replacement machine, a recorder that will not start, or a copy taken before physical work. It needs a planned recording outage. --offline is your assertion that you have stopped every process using this recorder's data; it does not stop them for you. A filesystem copy of a running SQLite database is not a backup.

sudo systemctl stop evernvr
sudo systemctl is-active evernvr    # must report inactive
sudo NODE_ENV=production /opt/evernvr/current/node/bin/node /opt/evernvr/current/maintenance.js backup \
  --data /var/lib/evernvr \
  --environment /etc/evernvr/evernvr.env \
  --output /mnt/backup/site1-2026-09-15-config --offline
sudo NODE_ENV=production /opt/evernvr/current/node/bin/node /opt/evernvr/current/maintenance.js verify \
  --backup /mnt/backup/site1-2026-09-15-config

/opt/evernvr/current/maintenance.js is the installed release's tool, run with that release's own Node (releases up to 0.1.50 have none; use /usr/bin/node there); it finds that release's migrations only with NODE_ENV=production. From a source checkout the entry point is node scripts/maintenance.mjs instead. Both tools work only on the paths you name, connect to no cameras and never migrate the source database. A missing or mismatched key, a corrupt database, an unexpected schema, changed sidecar configuration or an existing output directory aborts the operation. A backup without a complete manifest cannot be restored; a crash can leave one, so keep it for diagnosis and use a different output name when retrying.

backup reports the active, legacy and registered media paths. If the recorder uses EVERNVR_MEDIA with no saved storage settings, pass the absolute path with --media. Copy every reported media root, whole, including thumbnails and hidden markers:

sudo rsync -a -- /srv/evernvr-media/ /mnt/backup/site1-2026-09-15-media/

Repeat for the other locations. Do not use --delete against an existing backup. Record which source directory went to which backup directory, and the configuration backup's timestamp. Verify the copy — rsync -anc — and play a sample. An offline footage copy can take a long time; plan and record the gap. Only then:

sudo systemctl start evernvr
sudo systemctl status evernvr

Restoring onto a replacement recorder

1. Install the same release or a newer one on the replacement using INSTALL.md. Stop the service at once and confirm it is inactive. Keep the old recorder stopped through the cutover so the two do not compete for camera sessions.

2. Attach the trusted backup and verify it with the installed maintenance tool. verify reports migrationsToApplyOnRestore: the database changes this release will apply to the copy, and none for a backup from the same release. It also reports keyEncrypted: a backup made with a backup passphrase set carries the key wrapped, and verify checks the key only when the passphrase is piped in with --passphrase-stdin.

3. Restore into a new directory. The tool never overwrites an existing data directory, even an empty one. The restored copy is brought up to the installed release's schema; the backup itself is unchanged and can be restored again.

   sudo NODE_ENV=production /opt/evernvr/current/node/bin/node /opt/evernvr/current/maintenance.js restore \
     --backup /mnt/backup/site1-2026-09-15-config \
     --output /var/lib/evernvr-restored --offline
   sudo chown -R root:root /var/lib/evernvr-restored

   For a backup made with a backup passphrase, pipe the passphrase on stdin; the restored directory then holds the plain key again, and a wrong passphrase is refused with The backup passphrase is wrong. and nothing written:

   printf '%s' "$BACKUP_PASSPHRASE" | sudo NODE_ENV=production /opt/evernvr/current/node/bin/node \
     /opt/evernvr/current/maintenance.js restore \
     --backup /mnt/backup/site1-2026-09-15-config \
     --output /var/lib/evernvr-restored --offline --passphrase-stdin

4. Put the replacement's own data directory aside, then move the restored one into place while the service stays stopped. Both names must be absent before the move; do not overwrite an earlier recovery.

   sudo mv -T /var/lib/evernvr /var/lib/evernvr-before-recovery
   sudo mv -T /var/lib/evernvr-restored /var/lib/evernvr

5. Restore the media to the original absolute paths the manifest reports. That includes default footage under /var/lib/evernvr/media if it was used, and every external recording drive. Preserve the hidden storage markers, mount the drives persistently, and give root read and write access to the restored recording folders. Do not fabricate a marker to silence a missing-drive warning. Remapping media to arbitrary new paths is not supported.

6. Review the restored /var/lib/evernvr/service.env if there is one and apply the appropriate settings to /etc/evernvr/evernvr.env. Check the LAN address, the public origin, the detector path and any media overrides. Do not source the backup file into a shell or reuse an old network address blindly. Reapply certificates, systemd overrides, mount and network configuration separately — none of them are in the backup.

7. Start the service and sign in with an existing permanent account. Every old session was removed, so every device signs in again. Each restored account gets a Recorder restored entry in its history. Temporary installer access is ended during the restore, including grants saved before a later revocation, so restoring an old backup cannot reopen installer access. Authorize a fresh grant if there is more work.

8. Check camera credentials, detector readiness, every storage location and recording continuity. Open a known historical event, look at its thumbnail, and play and export its recording. Confirm new footage records and survives a reboot. Record the recovery duration and anything missing.

9. Open Settings → License. A replacement is a new installation, so it may start as a trial or read Not yet licensed. Enter the site's license key and choose Activate; when the page says the key is already in use on the old recorder, choose Move the license here. Recording does not wait for this step.

Restoring configuration without footage keeps the accounts and the event records but cannot bring back lost video or thumbnails. A backup also cannot recover a forgotten password: it holds the hash exactly as it was saved.

An old backup rolls back MFA recovery-code consumption and time-step history. After any restore, replace the owner's recovery codes.

Restoring across releases

A backup restores into the release that made it or any newer one: restore applies the database changes the backup is missing. A backup made by a newer release than the one restoring it is refused — install that release first.

If a release starts against a database a newer release has already migrated, it refuses with the migration names that are not in it. Use that release or a newer one. If the migration ledger has gaps — later migrations applied but earlier ones never — the database needs looking at before it is used; do not edit _migrations by hand.

Pre-migration snapshots

A release migrates the database when its service first starts. Before it changes a database that already has history, it saves it as it was under /var/lib/evernvr/backups/pre-migration-… and says so in the journal. The last three are kept; each is a full copy of the database on the system disk.

If the migration fails, the service exits with the migration's error and the snapshot's path, and systemd restarts it straight back into the same failure. The snapshot is reused on every attempt, not retaken, so it stays the database as it was before the first try.

To recover: stop the service, and either go back to the previous release — an update whose new release failed this way has already done that by itself and restored the pre-update backup; otherwise sudo evernvr-update rollback --restore-backup /opt/evernvr/backups/pre-update-… does it — or restore the snapshot with the previous release's maintenance tool into a new directory, as in restoring onto a replacement recorder. Keep the failed database for diagnosis.

Owner account recovery

For a staff member who has lost their authenticator and recovery codes, a permanent owner resets that account's second factor from Settings → Access; the account, its role and its camera grants survive. The procedure below is for the owner who has lost their own password or both factors.

It changes one existing permanent administrator's password and MFA, revokes every account's sessions and pending enrollments, clears temporary attempt limits, ends all temporary installer access, and writes an owner_recovered entry to account history. It preserves cameras, the credential vault key, recordings, events, roles and camera permissions. It never creates an administrator or promotes another account, and a temporary installer cannot be recovered as an owner. Use the matching installed release: a schema mismatch is refused rather than migrated.

You need local shell access as root, which is also what the recorder runs as.

sudo systemctl stop evernvr.service
sudo systemctl is-active evernvr.service
# Expect inactive. Check that any manually started recorder is stopped too.
read -r -s -p 'New owner password: ' owner_password; printf '\n'
read -r -s -p 'Repeat password: ' owner_confirmation; printf '\n'
if [[ "$owner_password" == "$owner_confirmation" ]]; then
  printf '%s\n' "$owner_password" | sudo NODE_ENV=production /opt/evernvr/current/node/bin/node \
    /opt/evernvr/current/maintenance.js recover-owner --data /var/lib/evernvr \
    --username 'OWNER_USERNAME' --password-stdin --offline
else
  printf 'Passwords differ; recovery was not run.\n'
fi
unset owner_password owner_confirmation
sudo systemctl start evernvr.service

Use the existing administrator's exact username, and the configured data directory if it is not the default. The password is read from bounded stdin and never appears in an argument or a log; avoid set -x. From a source checkout, node scripts/maintenance.mjs.

Then sign in with the new password, enroll a fresh authenticator, save new recovery codes and review Account history.

A lost secret.key is a different failure. Account recovery cannot decrypt camera credentials or other accounts' MFA seeds without it; restore the matching key from a backup.

Recorder startup and recovery both take an exclusive flock on DATA/.recorder.lock, so recovery refuses while the recorder or another recovery holds it. Kernel locks are released after a crash or power loss; the harmless lock file may remain. Never delete or replace the lock file to get past a refusal — stop the process holding it and retry.

Do not delete account tables to reopen first-run setup.

A failed update

An update rolls itself back when anything fails before the new release has passed its health check: the host check, the detector environment, the backup, a unit that will not start, a process that is not the new release, an API that does not answer, or a restart while it is being watched. An interrupt (Ctrl-C, a lost terminal) takes the same path. Settings → Updates then reports the update as rolled back, with the reason.

The rollback stops the failed release, points /opt/evernvr/current back at the previous release, renames the failed tree to /opt/evernvr/releases/VERSION.failed-<timestamp>, and renders the previous release's units again — including its EVERNVR_PYTHON link into /opt/evernvr-venv, so its detector environment comes back with it. If the service had been running, it starts it again and checks its health.

What happens to the database depends on how far the update got. The new release opens and migrates the database only when its service starts, so a failure before that — the detector environment, the Hailo reinstall, the backup — leaves the database exactly as it was: nothing is restored and nobody's session is revoked. A failure after the service started copies the pre-update /etc/evernvr back and uses the previous release's maintenance tool to restore the pre-update backup; the failed evernvr.db, its WAL and shm, secret.key and storage.json are moved to /opt/evernvr/backups/pre-update-…/replaced-<timestamp>/ and the restored copies put in place. Sessions are revoked through the normal restore path, so everyone signs in again. Footage stays where it is.

Events created after the pre-update snapshot are not in the restored database. Keep the failed database for investigation.

Afterwards, check evernvr-update status and the service, sign in, and check live recording and historical playback. The full output of the failed run is in /var/log/evernvr/update-<timestamp>.log.

Going back after a successful update

A release that passed its checks but turns out to be wrong is undone by hand. The previous release stays installed until a later update replaces it:

evernvr-update status                  # installed and previous versions
sudo evernvr-update rollback           # the previous release, with the database as it is now
sudo evernvr-update rollback --restore-backup /opt/evernvr/backups/pre-update-OLDVERSION-TIMESTAMP

A plain rollback keeps the current database. That is only safe when the newer release did not migrate it; if it did, the previous release refuses to start and names the migrations it does not have (see restoring across releases). --restore-backup also restores the pre-update backup — the configuration and the database as they were before the update — so events recorded since are lost from the database; the footage stays. The rollback's output is in /var/log/evernvr/rollback-<timestamp>.log.

An interrupted update

A power cut during an update, or a rollback that could not finish, leaves the marker /opt/evernvr/state/update-in-progress, which names the release being left, the release being installed and the pre-update backup. evernvr-update status then warns, and a new update refuses to start until it is recovered.

1. Keep the service stopped if it is not running. Do not delete either release, the backup or any database.

2. Read the marker and evernvr-update status. The updater's own log is the newest /var/log/evernvr/update-*.log.

3. Put the previous release back, restoring the backup the marker names:

   sudo evernvr-update rollback

   With the marker present, rollback returns to the release the update was leaving and restores the backup it names; it removes the marker only when the recorder is running and healthy again.

4. Verify login, recording and historical playback.

If the rollback cannot complete — the backup was never finished, say — keep the marker and everything under /opt/evernvr, verify the backup with the previous release's maintenance tool, and restore it by hand into a new directory as in restoring onto a replacement recorder. Keep the footage at its original paths. Remove the marker only after that recovery is verified.

A recorder still on the old installer (a plain /opt/evernvr tree, 0.1.33 and earlier) marks an interrupted update with /etc/evernvr/update-in-progress instead. Recover it with the deploy/RECOVERY.md shipped in that release; the first harness .run refuses to adopt the recorder while that marker exists.

Losing a storage location

A registered recording location remembers its mount source, its filesystem and its hidden .evernvr-storage marker. A missing or replaced mount blocks recording on that location; EverNVR never silently falls back to a folder on the system disk, and it never fabricates a marker.

Each location is probed from its own worker thread, so a hard NFS mount that stops answering is reported as Not responding within a few seconds and skipped by indexing, retention and recorder restarts while the other locations keep recording and playing back. The recorder resumes on it by itself once it answers. Nothing is buffered or backfilled for the time it was away. A mounted status only describes the kernel mount table; restore network and NAS connectivity before trying anything else.

Bring the original drive or share back at the same path, with its marker intact, and recording resumes there. If it is gone for good, choose another location under Settings → Storage & retention and then remove the dead one with Remove…, saying what happens to the recordings that were on it: Move to another location, Keep files on disk (indexing and retention stop; entering the same folder again reconnects it) or Permanently delete files. A location holding registered or preserved footage cannot be removed through the NAS controls until its recording location has gone first.

Recording I/O errors restart the affected FFmpeg process with backoff even when its live view keeps working, so recording reopens by itself when disk space comes back. Footage in flight during the failure can still be incomplete; the index and the incident history show the gap, and past gaps stay visible in playback.

Deliberate incident holds are excluded from both age and disk-pressure retention, so holds can fill a disk and stop recording. Export what matters to separately managed storage and release holds when they are no longer needed.

A factory reset

A factory reset asked for in Settings → This recorder writes /var/lib/evernvr/factory-reset.json and stops the service; it is carried out when the service starts again, before the database is opened, and the file is removed last. A reset interrupted by a power cut runs again at the next start. Recordings are moved into a .evernvr-erased-… folder on each drive and deleted in the background; one left behind is deleted at the next start.

Pre-update snapshots under /opt/evernvr/backups are outside the recorder's data and are not erased; they hold the previous owner's database. Delete them by hand when the recorder changes hands.

Logs and the support bundle

Settings → This recorder → Download support diagnostics produces a JSON bundle assembled from an allowlist: runtime versions and counters, schema names, record counts, camera and detector health, incident timing. It excludes accounts, camera names, addresses, credentials, keys, cookies, raw logs, recordings and thumbnails. Review it before sending it through the site's support process.

Download the recorder log gives the structured JSONL log from /var/lib/evernvr/logs/recorder.jsonl, kept as five rotated files of at most 10 MiB each, so about 50 MiB in all — enough to still hold the days before a fault was reported. Credential fields and URL credentials are redacted, but the log still holds site-specific detail, which is why it is not in the support bundle. A logging disk error does not stop recording, and the same lines go to the service journal; configure the journal's retention separately.

The Alarms page opens on the faults still needing attention and keeps the searchable history of faults and recoveries under All history, and History keeps rule runs, deliveries, camera writes, account changes and past health incidents, with CSV export.

When the recorder is unreachable

If a dead-man check URL is configured, the monitoring service will already have raised the alarm: the recorder fetches it once a minute while no critical fault is open, so silence covers a dead recorder, a full disk and dark cameras alike.

From a shell on the host:

systemctl status evernvr
journalctl -u evernvr -n 200 --no-pager
systemctl status evernvr evernvr-disk-health.timer
evernvr-update status

Work through it in this order:

• The service is in a restart loop. Read the journal for the reason. A migration failure names its error and its snapshot path — see pre-migration snapshots. A database migrated by a newer release names the migrations it does not have — see restoring across releases.
• evernvr-update status warns of an interrupted update (/opt/evernvr/state/update-in-progress exists). Go to an interrupted update and change nothing else first. The last update's result and log are in evernvr-update status and /var/log/evernvr/.
• The service is running but nothing answers on the network. Check EVERNVR_BIND (or EVERNVR_INTERFACE) and PORT in /etc/evernvr/evernvr.env, then the host firewall. With EVERNVR_INTERFACE, sudo journalctl -u evernvr | grep interface shows the address it is listening on; while the interface has no address it listens only on 127.0.0.1. The port serves HTTPS, and plain HTTP on it only redirects (or hands out the recorder's CA at /trust/ca.crt), so try the https:// address. Unless EVERNVR_TLS_CERT and EVERNVR_TLS_KEY are set, the recorder uses a certificate from its own CA in /var/lib/evernvr/tls/; the journal names every address it covers. Setting one of the pair without the other, or a key that does not match the certificate, fails startup with that message in the journal. An expired certificate does not: the recorder serves it and raises an alarm, so a browser warning is not a reason to look here.
• It answers but recording has stopped. Check the recording location under Settings → Storage & retention and go to losing a storage location.
• It answers but nobody can sign in. See owner account recovery. Repeated failures from one address are throttled and the lockout is listed and clearable under Settings → Access → Sign-in lockouts.
• The host itself is gone. Restore onto a replacement using the procedure above. Keep the failed machine's disks until the replacement has been verified.