Skip to main content

Health and self-healing

Each board watches itself and restarts when something has hung, but avoids restarting for problems a restart won't fix.

What the page shows​

Current problems appear in a banner at the top of the board's page, and the reason for the last restart under Firmware → Last reset. On the matrix, a device's status dot lists its problems. /api/status has the same under health.

When a board restarts itself​

  • A hang: an audio, serial or USB/IP task that stops running, or a CPU that's starved, trips a watchdog after 5 seconds.

  • A task that stops responding. Background tasks check in regularly, and the board restarts with … task stopped responding if one goes quiet:

    • USB handling or PTP for 30 seconds
    • stream announcements (SAP) or mDNS discovery for a few minutes

    Last reset then adds what the task was doing, such as (blocked in device write). ready in place of blocked means the task could run, but busier tasks kept the processor from it. Include this text when you report the problem, with usb.errors from /api/status: the USB audio transfer errors since the board started. It may end with [pc … ra …]: code addresses that show the firmware's developers where the task was stuck.

    The web server is the exception: an unresponsive one is reported but not restarted for, because a flood of slow visitors can tie it up without it having actually failed, and a needless restart would drop audio.

  • A failed health check. Checks run every second, after 20 seconds of warm-up. The board restarts if:

    • the USB audio adapter stops moving audio for 10 seconds
    • the PTP clock can't be read for 10 seconds, or stays more than 1 ms off for 60 seconds
    • an encoder's network sends fail for 15 seconds
    • a decoder's playout stops for 10 seconds
    • internal memory runs low

When it doesn't​

  • A decoder that stops receiving its stream rejoins the multicast group every 30 seconds instead: the usual causes (the source is off, or a switch forgot the membership) aren't fixed by a restart.
  • In the serial and USB/IP roles, only hangs, unresponsive tasks and low memory cause a restart. A missing adapter or device, or a TCP port that can't be opened, is shown as a problem: a quiet line is normal, and neither is fixed by a restart.

Degraded mode​

A fault that a restart doesn't fix doesn't become a restart loop. After 3 watchdog restarts or crashes without 10 fault-free minutes between them, the board stays up and shows degraded: stopped rebooting after 3 tries. It stays degraded while the fault lasts, and goes back to normal after 10 minutes without it. Hangs still cause a restart.

Any restart you start yourself (Reboot, saving settings, or a firmware update) resets the count.

New firmware​

New firmware that fails a health check, crashes or hangs in its first 40 seconds or so is rolled back. See firmware updates.