Aggregation reporter: select sensors by asset, name and unit

[Ahmad-Wahid]

Description

A site usually wants its sensors as one aggregated signal — all the PV on site, so it can be
forecast and used in flex scheduling. Until now the AggregatorReporter could do that only if every
sensor was named by hand in the report parameters, which does not survive an inverter being added.

The reporter now selects its own sensors, through four new config fields:

• asset — aggregate the sensors of this asset and of all of its offspring, so pointing at a site asset covers everything below it.
• sensors — aggregate these sensors, listed by ID.
• sensor-name-pattern — keep only the sensors whose name matches this regular expression.
• sensor-units — keep only the sensors recording in one of these units, or in a unit measuring the same quantity, so ["MW"] also keeps a sensor recording in kW, but not one recording in MWh.

method: sum
asset: 3
sensor-name-pattern: "(?i)pv"
sensor-units: ["MW"]

Sensors named in the input parameters still work, and are read as described there; the selected
sensors are added to them. input is therefore no longer required in the parameters — a selection in
the config is enough, and a reporter that selects nothing at all says so, rather than failing later
inside Pandas.

Units and resolutions are reconciled with the output sensor. A discovered sensor cannot be
hand-checked, so values are converted to the unit of the output sensor and read at its resolution.
That is what makes a roof recording 100 kW in quarter-hourly events and a carport recording 0.2 MW in
hourly events add up to 0.3 MW on an hourly MW sensor. Passing a resolution parameter still wins
over the output sensor's, convert-units: false aggregates the raw values, a sensor without a unit is
never converted (an empty unit says nothing about what its values mean, so it draws a warning), and a
sensor whose quantity the output sensor cannot express — a temperature onto a power sensor — is
reported as an error instead of being silently added up.

The output sensor is left out of the aggregation. A report is often recorded on a sensor of the
very asset being aggregated, and reading it back in would fold each run's own previous output into the
next one. input_sensors leaves it out for the same reason, so the property names the sensors the
reporter will actually read; a sensor named explicitly in input is still kept, because naming it
there asks for it to be read.

Accountability. The selection lives in the config, not in the parameters, so it is recorded on
the reporter's data source alongside the method and weights, and a changed selection yields a new data
source.

• Added changelog item in documentation/changelog.rst

How to test

pytest flexmeasures/data/models/reporting/tests/test_aggregator.py

Eight tests are added, on a fixture whose PV sensors sit on a child and a grandchild asset, record in
kW and MW at quarter-hourly and hourly resolutions, and whose output sensor already holds a previous
report of 99 MW.

Each was verified to fail with the code it covers disabled. The thirteen breaks used were: dropping
the name-pattern filter, dropping the units filter, including the output sensor in the aggregation,
listing the output sensor in input_sensors, skipping the unit conversion, ignoring convert-units,
not defaulting the resolution to the output sensor's, dropping the empty-selection error, not
overriding input_sensors, ignoring the sensors list, not walking asset.offspring, skipping the
regex validation, and taking asset as a plain int rather than an asset. Every new test went red
under at least one of them, and the whole set is green with the code in place.

The 15 pre-existing aggregator tests are untouched and still pass, as do the reporting, schema, CLI
and API report-trigger suites.

It was also run end to end against a real database, building a site by hand — an asset, a sub-asset,
four sensors in two units at two resolutions, a day of CSV data — and reporting onto a site-level
sensor. The roof's 400 kW at quarter-hourly and the carport's 0.15 MW hourly come out as 0.547 MW on
the hourly MW sensor, and a second run of the same command saves nothing, because the output sensor
is not read back in.

Notes for reviewers

The three multi-word config fields are spelled in kebab-case (sensor-name-pattern, sensor-units,
convert-units) since #2547 landed on main and c846835 renamed them here; the Python attribute names
are unchanged. asset and sensors are single words and were never affected.

Further improvements

Two items of #2520 depend on PRs that have not landed yet, so they are noted rather than done here:

• Running as an automation (Reports as jobs and as automations #2297). Nothing in this PR gets in the way: the selection sits in the
  config, which is exactly what Reports as jobs and as automations #2297 pins on a report automation's data source, and input_sensors
  resolves the selected sensors so the output-scope check has something to check. Worth re-testing
  end to end once Reports as jobs and as automations #2297 merges.
• A prepared template (Prepared report templates #2300). A site-aggregate template carrying this reporter, an asset/
  sensor-name-pattern config and the recommended rolling window would fit Prepared report templates #2300's shape directly,
  and is left to that PR.

sensor-units matches on dimension rather than on the exact spelling, which is what makes it useful
next to the unit conversion; a caller wanting an exact-spelling filter would need a further option.

One rough edge found while testing, not introduced here and not fixed here: when a reporter raises,
flexmeasures add report lets the exception escape, so the user gets a full Python traceback with the
message at the bottom rather than a clean CLI error. That applies to every reporter, so it deserves
its own issue.

Related items

Closes #2520.

---

Sign-off

• I agree to contribute to the project under Apache 2 License.
• To the best of my knowledge, the proposed patch is not based on code under GPL or another incompatible license.

🤖 Generated with Claude Code

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #34706378 | 📁 Comparing b1aa2d9 against latest (819b836)

  🔍 Preview build  

17 files changed · ± 17 modified

± Modified

• changelog.html
• genindex.html
• http-routingtable.html
• _autosummary/flexmeasures.api.v3_0.assets.html
• _autosummary/flexmeasures.data.models.automations.html
• _autosummary/flexmeasures.data.models.reporting.aggregator.html
• _autosummary/flexmeasures.data.schemas.automations.html
• _autosummary/flexmeasures.data.schemas.reporting.aggregation.html
• _autosummary/flexmeasures.data.services.automations.html
• _autosummary/flexmeasures.data.services.scheduling.html
• and 7 more...

[nhoening]

Can you check this comment about using the flex-config for lookup (as well), which probably expands the scope a bit?

[Flix6x]

Just commenting on the PR description:

It feels strange to have created this whole mechanism (i.e. the flex-config) for defining a site topology, including how devices are nested, which sensors record aggregates and which sign corresponds to consumption/production, and then make/extend a second mechanism (the AggregatorReporter extended in this PR) for also aggregating across power sensors. Users would then have to keep the two topologies in sync themselves. It will lead to problems.

I could imagine an AggregatorReporter being useful to aggregate sensor data other than power data, though. Or using it to subtract one sensor's power data from another's. But for the purpose of aggregating across a site, the flex-config should be the single source of truth imo.

Related question: does the AggregatorReporter work with the VariableQuantity field, so that it gets all its source-filtering features? The flex-config does have that, which would become one likely source of discrepancies.

convert-units: false aggregates the raw values, a sensor without a unit is
never converted (an empty unit says nothing about what its values mean, so it draws a warning)

1. I'm not in favour of letting users add mismatched units. It's a mistake imo. 1 kW + 1 MW is not 2 kW. If you need this for something, I would recommend letting users use the weights to do unit conversion, for instance: 1000 kW/MW, demanding that the units are explicitly part of the weight.
2. It draws a warning: toast or logs? It should be immediate feedback to the user that might be making a mistake.

[Flix6x]

Having now looked at this and #2579 together, I don't think we should keep this PR. The site topology already lives in the flex-config, and this PR adds a second, name/unit-based mechanism for describing the same thing, with nothing to keep the two in sync. The regex/unit filters are guesswork (e.g. (?i)pv also matches a "PV forecast" or "PV curtailment setpoint" sensor), walking the whole subtree double-counts any sub-site that has its own aggregate sensor, and convert-units: false adds up raw values in different units. For non-power aggregations, the explicit input list plus weights already does the job.

My proposal: close this one and carry the parts worth keeping into the reworked #2579:

• unit conversion to the output sensor's unit, with a fix: it currently converts after the belief search has mean-resampled to the output resolution, so energy sensors at a finer resolution come out too low (four quarter-hourly 1 kWh events become 0.001 MW instead of 0.004 MW). Convert at the sensor's own resolution, before resampling.
• never folding the output sensor back into its own aggregate
• the clear error when there is nothing to aggregate
• copying the input descriptions rather than consuming them

Also note that defaulting resolution to the output sensor's resolution, together with the version bump, changes values and data sources of existing aggregator reports, so that deserves a changelog mention wherever it lands.