Data Retention

System > Data Retention sets how long each kind of history is kept, shows how much of the database each one is currently using, and names the attributes filling the largest table.

Both directions matter at commissioning. The defaults are short — Attribute History keeps 7 days — so a site that wants year-over-year trends has to say so. And they are short for a reason: on a residence with a few fast-polling meters, even a 7-day window measured 2.7 million rows and over a gigabyte on disk. The writers panel further down this page is how you find out which attributes are responsible.

Open Data Retention

The page is a summary line at the top with total records and total disk, a Retention Periods section with one row per store, a collapsed Top Attribute History Writers panel, and a reminder box at the bottom.

The nine stores

Every history table in the system appears as one row. The label and description below are exactly what the row shows.

Row	What it holds	Default	Never pruned
Attribute History	Sensor readings, device states, zone levels	7 days	—
Request History	Commands, macro calls, REST and web service requests	7 days	—
Access Activity	Access control events, pictures, and facial-match results	30 days	—
Automation History	Trigger, schedule and macro runs — failures kept longer	14 / 180 days	Runs still in progress
Alarm History	Closed alarm events	90 days	Open events, and latched events not yet acknowledged
Monitor History	Per-check monitor results and tag rollup transitions	30 days	—
Site Mode History	Site and space mode changes	180 days	—
Report Runs	Report delivery history	180 days	Deliveries still in progress
Daily Energy	Per-day energy totals per meter	1095 days	—

Each row also shows what that store currently costs: rows · size, and oldest — the date of the earliest record still present. The size is the table's full footprint including its indexes, which is why a table with a few million small rows can report well over a gigabyte. Monitor History sums two tables (per-check results and tag rollups) into one row and one setting.

The summary line at the top of the page totals rows and disk across all of them.

note

Saved reports and their delivery schedules are configuration, not history — retention never touches them. Delete those from the Reports page. The same applies to alarm definitions, monitors, and macros: only their run records are pruned here.

Two macro tables prune themselves

The full kept runs a macro keeps for its step-by-step inspector, and its revision history, are not on this page and are not touched by the nightly job — each drops its own oldest row as a new one is written. A macro keeps its last 20 runs (the macro_run_retention system attribute changes the count, read at startup; 0 turns the history off) and its newest 50 revisions. The summary rows above therefore under-report those two tables; a site with thousands of macros should size for them separately.

Setting a retention window

1. Type the number of days in the box on the store's row. The field accepts 1 to 3650 days.
2. Automation History has two boxes instead of one — success and failure days. The first covers runs that finished, were skipped, or were debounced; the second covers runs that failed or were aborted, and defaults to far longer because a failure matters long after a success stops mattering.
3. Leave Automatic daily maintenance enabled checked (it sits in the grey strip under the section title).
4. Click Save Settings. A toast confirms Retention settings saved successfully. Nothing is deleted at this point — the new window takes effect on the next nightly run.

Reset re-reads the saved values and throws away anything you typed but did not save.

warning

Purge Now uses the number in the box, not the number you saved. Type 30 into Attribute History, click Purge Now without clicking Save Settings first, and it deletes everything older than 30 days immediately — while the nightly job keeps using whatever is still saved. Save first, then purge, or you will get one result today and a different one tonight.

What the nightly job does

Automatic cleanup runs daily at 3 AM server time, and works through the stores in a fixed order: Attribute History, Request History, Access Activity, Automation History, Alarm History, Site Mode History, Monitor History, Report Runs, Daily Energy. It finishes by sweeping orphaned attribute rows, then logs a single line with the number of records removed from each.

There is no reindex, vacuum, or table-optimization step — the job deletes rows and nothing else. Postgres reclaims the space on its own schedule, so a table's reported size can stay high for a while after a large purge.

Batching. Attribute History, Access Activity, Alarm History, Site Mode History, Report Runs, Daily Energy and both monitor tables are deleted 10,000 rows at a time, each batch its own short transaction, so an overnight sweep never blocks the live attribute writes coming in from the building. Request History and Automation History are not batched — each is removed in one sweeping statement, which is a long one on a table that has gone unpruned for months.

Automation History has a second limit. Beyond the two date windows, the nightly job keeps at most 10,000 runs per macro, trigger, or schedule — a noisy-neighbour cap, so one chatty trigger cannot drown the table no matter how short its runs are. Manual Purge Now applies the date window only.

If a single run has to remove more than about 10 million rows from one table it stops at that ceiling, logs a warning, and takes the rest on the following night. Nothing is lost; a site that ran for a long time with maintenance disabled simply needs a few nights to catch back up.

Orphaned attribute sweep. The job also deletes attribute rows whose owning entity — zone, device, site, site space, subsystem, UI, variable, remote, or user — no longer exists. Deleting an entity already removes its attributes immediately, including secure values, so this only catches rows left behind by older versions or by paths that deleted database rows directly. Global system settings are never swept.

warning

Unchecking Automatic daily maintenance enabled disables Run Maintenance Now as well. With the box unchecked and saved, the button reports Maintenance failed: Unknown error and deletes nothing — the server refuses the whole job, not just its schedule. Per-store Purge Now still works. Re-check the box and save before running maintenance by hand.

Manual operations

Run Maintenance Now

The button in the page header runs the full nightly job immediately, against the saved settings. It asks to confirm, then reports Maintenance complete: N total records deleted and refreshes the statistics.

Use it after shortening a window, when you want the space back now rather than at 3 AM.

Purge Now

The button on a store's row deletes only that store's records, using that row's number of days. It asks to confirm and names the store and the window first.

Two rows behave differently from the rest:

• Automation History purges against the failure window — the larger of the two numbers — for every status, so successes between the success window and the failure window survive a manual purge even though tonight's job would remove them. Runs still in progress are always kept.
• Alarm History keeps open events and latched events awaiting acknowledgement no matter how old they are, exactly as the nightly job does.

Unlike the nightly job, a manual purge is one statement per table. On a multi-million-row Attribute History that is a long-running delete in the middle of the day; prefer shortening the window and letting the batched nightly run do the work if the site is occupied.

Top Attribute History Writers

Attribute History is almost always the largest store, and it is rarely large for a diffuse reason — on most sites a handful of fast-polling analog attributes account for the bulk of it. On one residence, 52 breaker circuits writing power every 5–10 seconds produced 77.9% of a 2.7-million-row table.

Click the Top Attribute History Writers heading to expand the panel. It loads only when opened, because it scans the history table twice; visiting the page to change a retention number costs nothing. Buttons switch the window between 1h, 6h and 24h.

The summary line reports how many rows landed in that window, the resulting rows/hour, how many live series the controller is counting, and the share of offered samples the deadband is currently rejecting.

Each row of the main table is one attribute name for one kind of entity — power on zones and power on devices are separate rows, and the grey text after the name tells you which:

Column	Meaning
Series	How many entities write this attribute. Fifty breaker circuits writing the same name show as one row with 50 series
Rows	Rows that landed in the selected window
Share	Percentage of all history rows written in that window
Suppressed	Percentage of offered samples the History Deadband rejected, from the live counters. A dash means the controller has no live counter for that name — most often because it restarted recently, or the attribute stopped writing

Suppressed is the column to read. Rows alone cannot tell an eventful attribute from an unfiltered one — a busy meter at 80% suppression is working as designed, while one at 0% is writing every raw poll straight to disk. Only the write path knows this number: rejected samples never reach the table, so no query over the history can reconstruct it.

Two badges mark the problems:

• flooding — at least one series under that name is writing 60 rows/hour or more while the deadband rejects under 5% of its samples. Fast and unfiltered. The count also appears next to the section heading.
• no deadband — nothing is filtering this attribute at all. The deadband only applies to float attributes on zones and devices whose name is in the attribute registry's auto-history set; anything else writes every sample it is offered, and history_deadband_percent has no effect on it at any value. This is the more serious of the two, because nothing holds the attribute back if its poll rate ever increases.

Below the main table, Worst individual series lists the ten single entities writing the most rows — the drill-down for when one zone or device is responsible rather than a whole class of them. Where the entity is one GEM can render, its name is a link that opens it in place, so you can check a device's poll interval without losing the window you selected or the tables you just loaded.

note

The Suppressed column and the badges come from counters the controller keeps in memory, not from the database. They cover the current hour once it has at least five minutes of data, otherwise the previous full hour, and they start empty after a restart — the panel says Live window is still open … suppression rates are provisional while that is the case. The Rows and Share columns are read from the table itself and are unaffected.

The same findings appear on the Diagnostics page, and one summary line is written to the log at the end of each hour in which anything was flooding.

warning

None of the above pages anyone. If a device starts writing far faster than intended, this panel will show it and nobody will be looking — and attribute_history filling the disk takes the whole controller down with it.

The Attribute History Flooding alarm is the backstop. It watches the combined rate of everything nothing is filtering and raises once that passes attribute_history_flood_alarm_rate rows/hour (default 5000) — a deliberately far higher bar than the badge above, so it means "the database is in trouble" rather than "here is something worth tuning". It is seeded enabled; give it recipients on the Alarms page, because an alarm with none is dashboard-only.

The Disk Space Low alarm covers the other end — growth that never tripped a rate threshold but has filled the disk anyway. Its message names the database size and its largest table, and its source link lands on this page, because history is nearly always what consumed the space.

Fixing a flooding attribute

The panel fixes what it reports. Two controls sit with the table, and both take effect on the next sample — no restart, no waiting for the settings cache.

The system deadband

The strip above the table sets history_deadband_percent (default 2) and history_deadband_max_interval (default 300 seconds) for the whole site. The percentage is the minimum change required to record a sample; the heartbeat forces one through when a value has been flat that long, so a stalled series still has a pulse in the data. Setting the percentage to 0 records every sample.

This is the right lever when everything analog is too chatty. It has two limits worth knowing before you reach for it: it only reaches float attributes in the registry's auto-history set, so it does nothing at all for a no deadband row; and a relative threshold is weak on signals that spend much of their time near zero, where 2% of 5 W is 0.1 W and ordinary meter jitter clears it every sample.

Tune — per attribute

The Tune button on each row sets a policy for that row's attribute, applied across every entity writing it — one setting covers all fifty breaker circuits reporting the same name. It follows the row's scope, so power on zones and power on devices are tuned separately. Three choices:

Choice	Effect
Platform default	No override. The system deadband applies if it reaches this attribute; if the row is badged no deadband, nothing filters it
Record only meaningful changes	This attribute's own thresholds in place of the system ones. Applies even where the system deadband cannot reach — this is the fix for a no deadband row, and it needs no registry entry
Stop recording history	No new history rows for this attribute on any entity

The middle choice takes three numbers:

• % change — the relative threshold, as above, for this attribute only.
• Min change — a floor in the attribute's own units: watts, degrees, litres. The percentage can raise the threshold above it but never lower it. This is the number that fixes metering. On the residence above, mean absolute power was 55 W — where 2% is about 1 W — and over half the rows recorded a change smaller than 10 W. A percentage large enough to filter that would destroy the resolution at high load, where the same percentage is hundreds of watts; a 10 W floor removes it and leaves every real load step intact. Leave it blank for percentage-only behaviour.
• s heartbeat — this attribute's own forced-sample interval.

Whatever those three say, a deadband still records the first sample of a series and every crossing of zero, so "the circuit turned off" and "the circuit came back" are never lost. Stop recording history is the one choice that has no exceptions — it records nothing at all.

caution

An override reaches integer attributes, which the system deadband never touches. Temperature, setpoint, level, humidity and battery are quantized — a change of 1 is already the smallest step they can report — so a percentage on one of them silently swallows whole degrees. Tuning an integer attribute is a deliberate choice, not a default; prefer the heartbeat, or Stop recording history, over a percentage there.

That choice affects only the history table. Live values, triggers, alarms, macros and widgets read the current attribute value and are completely unaffected — what stops is the accumulation of rows, so charts and reports over that attribute simply gain no new data.

None of these delete anything. Existing rows go when their retention window expires, or immediately via Purge Now.

A tuned attribute carries a blue badge in the table, and every override is listed under Per-attribute settings below the tables with Edit and Remove. That list is the place to look for something you silenced days ago: once an attribute stops writing it eventually falls out of the window and off the table itself, and the overrides list is then the only way back to it.

The other two fixes

1. Increase the device's poll interval if the resolution is not being used. Nothing downstream needs a meter read every five seconds if the charts are hourly.
2. Give the attribute a registry entry with history enabled and value_type float — see Attributes. This is the durable fix when the attribute belongs to a whole driver rather than to one site's wiring, because it travels with the integration instead of living in this site's settings.

Choosing windows

The shipped defaults are deliberate, not placeholders. Attribute History defaults to 7 days because its volume is driven by poll rate rather than by events: a large residence can land several hundred thousand rows a day, and every extra day of window is that much more table.

Situation	Reasonable starting point
Typical residence or small commercial site	Leave the defaults. Extend Attribute History to 30 days if the owner looks at trend charts
Trend analysis or energy work	Attribute History 30–90 days, after widening the deadband — otherwise you are keeping three months of jitter
Access-control compliance	Access Activity 365 days or whatever the regulation requires; it is event-driven, so the cost is low
Heavy automation debugging	Automation History success 30, failure 365
Storage constrained	Attribute History 3–7 days, Request History 7, Automation History success 7 / failure 90

Two windows deserve special care:

• Daily Energy defaults to three years for a reason. It is an aggregate — one row per meter per calendar day, written at each local midnight because meter readings are cumulative lifetime counters and a day's usage exists only as the difference between two snapshots. It is the only source the energy reporting can read, and once pruned those days cannot be reconstructed from attribute history. Fifty meters accumulate roughly 18,000 rows a year, which is nothing.
• Alarm History carries the notification delivery record for each closed event. Shortening it below the site's incident-review period removes the evidence of what was sent and to whom.

Widening the deadband almost always beats shortening the window: it removes rows that carry no information, while a shorter window removes rows that do. Start with Tune on the two or three names at the top of the writers panel rather than with the system-wide setting — that keeps the resolution intact everywhere it is actually being used.

Before you purge

• Deleted history cannot be recovered. Take a backup first, particularly the first time you try a new window.
• Check the retention requirements that apply to the site — access-control and alarm records are the ones that tend to be regulated.
• The warning box at the bottom of the page recommends minimums of Attribute History 30 days, Request History 7 days, and Access Activity 90 days. Attribute History nonetheless ships at 7 days, deliberately — its size follows poll rate rather than site activity, so raise it only once the writers panel has told you what you would be keeping.

note

Attribute History carried a redundant second index on its row identifier — on a large site, several hundred megabytes of pure duplication maintained on every insert — and a row identifier narrow enough to eventually run out. Both are corrected automatically the first time an upgraded controller starts, before devices connect.

Widening the identifier rewrites the table, so on a site with a multi-gigabyte Attribute History that one startup can take a few minutes longer than usual. The controller is working, not hung, and it logs what it is doing; it also needs roughly the table's own size in free disk while it runs. Subsequent restarts are normal, and shortening retention or purging beforehand makes it proportionally faster.

Related Documentation

• Backup & Restore — take one before purging
• Attributes — history flags, the deadband, and the attribute registry
• Attribute History — viewing what was kept
• Request History — viewing the request log
• Diagnostics — the flooding check and other health findings
• Alarms — alarm events and acknowledgement
• Reports — report definitions, schedules, and delivery history