Aggregate the portfolio the flex-context already describes

[Ahmad-Wahid]

Description

One more idea is to use the flex-config to look up the aggregated portfolio and possibly even the
aggregate-consumption/production sensor to save the aggregated profile on.

A site that is scheduled has already said what it is made of. Its flex-context lists the sensors of
its inflexible load under inflexible-consumption and inflexible-production, and it can name the
sensors those aggregates belong on, under aggregate-consumption and aggregate-production. A report
that aggregates the same portfolio should not have to restate it.

A new portfolio config field on the AggregatorReporter, taking consumption or production,
does both halves of that idea:

# config
method: sum
asset: 3
portfolio: consumption

# parameters
start: "2023-01-01T00:00:00+00:00"
end: "2023-01-03T00:00:00+00:00"

The parameters name no sensors at all. The sensors to aggregate come from the asset's
inflexible-consumption, and the result is recorded on the sensor its aggregate-consumption names.
An output named in the parameters still wins, for a one-off report that belongs elsewhere.

The flex-context resolves up the asset tree, exactly as it does for scheduling, so a portfolio
defined on a site also serves the assets below it.

Source filters are passed through. Each entry may carry source-types, exclude-source-types,
sources or source-account, and those reach the belief search unchanged, so a portfolio pointing at
a forecaster's values aggregates exactly those rather than everything on the sensor.

portfolio uses asset to find the flex-context, not as a subtree to walk. This is the one
design decision worth a second look: the two readings of asset are genuinely different, and walking
the subtree as well would silently aggregate every sensor that happens to sit below the site —
which is what my first draft did, and what a test caught. sensors still contributes alongside a
portfolio when you want to add something.

The deprecated inflexible-device-sensors is refused, with a message saying why: it is a flat
list that says nothing about whether a sensor records consumption or production, so it cannot answer
for either portfolio. Splitting it is the migration the schema already wants.

output_sensors reports a sensor taken from the flex-context, so an automation checks the report
against the sensor it will really be written to — which matters now that #2297 has landed.

• Added changelog item in documentation/changelog.rst

How to test

pytest flexmeasures/data/models/reporting/tests/test_aggregator.py flexmeasures/data/schemas/tests/test_reporting.py

Eight tests are added, on a fixture that describes the site's consumption portfolio in its
flex-context and points aggregate-consumption at a sensor that already holds a previous report.
They cover a report naming neither input nor output, an entry's source filters reaching the search, an
output override, a missing asset, a portfolio the flex-context does not describe, the refusal of the
deprecated field, an unknown direction, and having nowhere to record at all.

Each was verified to fail with the code it covers disabled. The nine breaks used were: ignoring the
portfolio's inputs, not resolving the output from the flex-context, letting portfolio also walk the
subtree, dropping an entry's source filters, not erroring on a missing asset, not flagging the
deprecated field, not overriding output_sensors, removing the portfolio value validator, and
removing the at-most-one-output validator.

A test that had to change

test_specialized_reporter_schemas_preserve_required_dataflow_fields asserted that the aggregator's
output stays required. That is no longer the contract — the config can supply it — so the guard now
holds the aggregator to recording on at most one sensor instead, and the "something must supply an
output" rule is enforced at compute time with its own message and its own test. The
ProfitOrLossReporter half of the guard is untouched. Both halves were verified to still fail when
the constraint they name is removed.

Related items

Closes #2573. Stacked on #2525 (issue #2520), whose asset / sensor-name-pattern / sensor-units
selection this builds on; review that one first.

Note that both branches are currently 14 commits behind main, which has since gained #2297 and
renamed the reporter parameter belief_time to prior. Neither touches the aggregator, so the stack
was left tight rather than merged; #2525 needs the merge first, and it could not be done here because
that branch is checked out elsewhere.

---

Sign-off

• I agree to contribute to the project under Apache 2 License.
• To the best of my knowledge, the proposed patch is not based on code under GPL or another incompatible license.

🤖 Generated with Claude Code

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #34708220 | 📁 Comparing 387991f against latest (1b6ea6b)

  🔍 Preview build  

46 files changed · + 3 added · ± 37 modified · - 6 deleted

+ Added

• _autosummary/flexmeasures.api.dev.html
• _autosummary/flexmeasures.api.dev.sensors.html
• api/dev.html

± Modified

• changelog.html
• genindex.html
• http-routingtable.html
• py-modindex.html
• _autosummary/flexmeasures.api.html
• _autosummary/flexmeasures.api.v3_0.assets.html
• _autosummary/flexmeasures.api.v3_0.html
• _autosummary/flexmeasures.cli.utils.html
• _autosummary/flexmeasures.data.models.automations.html
• _autosummary/flexmeasures.data.models.forecasting.pipelines.base.html
• and 27 more...

- Deleted

• _autosummary/flexmeasures.api.ui.html
• _autosummary/flexmeasures.api.ui.sensors.html
• _autosummary/flexmeasures.api.ui.session.html
• _autosummary/flexmeasures.data.models.registry.html
• _autosummary/flexmeasures.utils.bound_utils.html
• api/ui.html

[Flix6x]

This is the right direction, but I'd rework it in a few ways (and retarget it at main, see my comment on #2525).

Wrong quantity on aggregate-consumption/aggregate-production. Those sensors hold the scheduled aggregate of all flexible and inflexible devices (net, and clipped when both are defined). portfolio: consumption sums only inflexible-consumption and records it there, so the reporter's realized values and the scheduler's planned values would sit on the same sensor while meaning different things.

Per-commodity contexts are missed. The aggregate and inflexible fields can also sit inside the flex-context's commodities list, and this PR only reads the top-level keys. Worse, when an electricity entry exists under commodities, the scheduler ignores the top-level fields altogether (_get_commodity_contexts), so this PR would read exactly the fields the scheduler doesn't use. Inflexible devices declared in flex-model entries on child assets are also not picked up.

Use group rather than a new portfolio field. We already support arbitrary aggregations: any flex-model entry, including an inflexible one, can join group: {"asset": ...} or group: {"sensor": ...}, and the scheduler records the group's aggregate on the group sensor or on the group entry's consumption/production sensors. So "all PV, recorded on sensor X" is already expressible. I'd let the reporter take a group reference and resolve its members through the same device inventory the scheduler uses (inventory.group_to_devices), so membership, commodity and sign convention have a single source of truth. The site-level aggregate-* sensors then become the implicit root group. Two things to check: a group without power-capacity (used purely for aggregation) should be allowed and documented as such, and a group sensor should not be allowed to also be listed as an inflexible member (double counting).

Source filters. The flex-config's source filters mix two things. Provenance (e.g. source-account, excluding our own scheduler's writes) belongs to the topology and should be inherited. Forecast vs. realization is a matter of belief horizons: a realized aggregate is a report with belief horizon ≤ 0, a forecast aggregate uses a positive horizon or a prior, both set in the report parameters, so one group serves both. The conflict case is an entry pinned to source-types: ["forecaster"], which would make a realized report come out empty. For that, the parameters could override source filters per member, and an empty result should raise a clear error. I'd start with inheriting and selecting by horizon, and only add the override when a real site needs it.

Smaller things from reading the code:

• output_sensors only catches ValueError, but OutputSensorReferenceSchema().load raises a marshmallow ValidationError, e.g. when the flex-context references a deleted sensor, so the automation's output check crashes.
• Portfolio inputs aren't checked against the output sensor.
• Two entries for the same sensor with different source filters get deduplicated, silently dropping the second.
• An entry filtered only by source-types can still hit the "Missing attribute 'sources'" error when several sources remain, whereas the scheduler accepts that data.