Reports as jobs and as automations

[Flix6x]

Description

Automations can now compute reports on a recurring basis, alongside forecasts and schedules. The
automation still only decides when work is due and queues it; the reporting worker computes the
report and stores it. Nothing about how a report is computed changes.

Running a report as a background job is no longer part of this PR. The reporting queue,
flexmeasures add report --as-job and the job runner itself landed on main in #2298, so what is left
here is the recurring side, and the rolling window that makes recurrence worth having.

A reporter is a data generator, like a forecaster. The reporter class and its configuration live
on a steady data source, pinned by the automation's generator_id, while the automation's
parameters hold what resolves afresh on each run. The creation field is therefore data-generator, taking a forecaster or a reporter class, and
config now describes both. The instance it resolves to is reported as the automation's source.
Schedules still assemble their generator per run and take neither.

Task names. reporting joins forecasting and scheduling in the supported types, which is
what the placeholder comment on SUPPORTED_TYPES had been holding open, and report joins the
result nouns used in messages about a single result.

The report window rolls. An automation that always computed the same period would be pointless, so the start and end resolve on every run, in one of two ways.
start-offset and end-offset hold comma-separated Pandas offsets, such as "-1D,DB" for the start of the previous day, applied to the run time on the automation's own clock: the timezone its cron string is read in.
Without any timing fields, the window runs from the end of the last window this automation successfully covered.
On the first run, when no covered window is known, it falls back to the last cron period.
A fixed start or end is refused when the automation is created, because every run would then report on the same period.

Coverage is recorded only on success. The reporting job carries its automation_id and records
the end of its window once the report has been stored, so a failed run leaves no permanent gap: the
next run starts where the last successful one ended, rather than skipping the period the failure
lost. Coverage is kept in Redis and never moves backwards, so a late-arriving job cannot rewind an
automation into recomputing what it has already reported.

Where a report may be recorded. validate_forecast_output_scope becomes
validate_automation_output_scope, so a report's output sensors are held to the rule a forecast's
already are: they must sit within the asset's own subtree, and the creator must be allowed to record
on them. A report names sensors in its config as well as its parameters, so both are checked.
And as #2536 does for schedules, a report job created by an automation records only on the sensors that were checked:
the reporter decides at run time which sensors it returns results for, so a result for any other sensor refuses the whole report before anything is saved.
The helper that works those sensors out moves from the scheduling service to the automations service, as both job types now use it.

CLI. flexmeasures add automation --type reporting takes the reporter through --reporter, its
configuration through --config, and the report parameters through --parameters:

flexmeasures add automation --asset 3 --name "Daily self-consumption report" \
  --cron "0 1 * * *" --type reporting --reporter PandasReporter \
  --config reporter-config.yml --parameters report-parameters.yml

--source accepts a reporter's data source, reusing its class and configuration the way it already
does for a forecaster.

Unset options. Click passes an empty tuple, not None, for a multiple-value option that was
never given, so unset options were leaking into the stored config and parameters as empty tuples.
They are now dropped along with the other unset values.

UI. The Reporting tab on an asset's Automations page is now populated, completing the three
tabs #2290 put in place. The New automation modal offers the data generator and its configuration.

• Added changelog item in documentation/changelog.rst

Look & Feel

The Reports tab, which #2290 and #2293 both showed as a disabled placeholder, is now populated. Here
a daily energy-costs report on a campus asset, recurring at 01:00 in Europe/Amsterdam, with the same
per-row actions the other two tabs offer:

Creating one asks for the reporter and its configuration through the same Data generator fields a
forecast automation uses, and takes the report parameters as JSON:

How to test

See the manual test walkthrough in the PR comments.

pytest \
  flexmeasures/cli/tests/test_automations.py \
  flexmeasures/data/tests/test_automations_fresh_db.py \
  flexmeasures/data/schemas/tests/test_reporting.py \
  flexmeasures/api/v3_0/tests/test_automations_api.py

Ten tests are added. Coverage includes the full roundtrip — create a report automation, let the
runner queue a reporting job with its provenance, let a worker compute and store it, then check the
stored values — and the window resolution on its own, a report automation whose config names
another organisation's sensor, one whose output falls outside the asset's subtree, the refusal to
create one without a generator, one that fixes its period, a reporter returning results for a sensor nobody checked,
offsets resolving on the automation's clock rather than the platform's, and the guarantee that recorded coverage cannot move backwards.
Each was verified to fail without the code it covers.

Further improvements

• Coverage lives in Redis, so a flushed cache sends an automation back to its cron-period fallback
  rather than to the true last covered window. Durable run records are tracked in Add durable automation-run records and safe retry semantics #2393.
• Other fixed moments go stale in an automation too: a fixed prior in any automation type, and a fixed start or end in a forecast or schedule automation. Refusing those for all three types is Refuse a fixed start, end or prior in an automation's parameters #2550.
• Report parameters cannot be edited after creation, for the reason forecast parameters cannot: it
  would decouple the automation from the access check its creator passed.
• The recommended rolling window has to be written out by hand for every report automation. Shipping
  ready-made report definitions that carry one follows in Prepared report templates #2300.

Related items

Part of the automations story #2334, and of #2288. Followed by #2300 and #2299, which are stacked on it.
#2298 and #2294, which this PR was originally written on top of, have since merged to main.

---

Sign-off

• I agree to contribute to the project under Apache 2 License.
• To the best of my knowledge, the proposed patch is not based on code under GPL or another incompatible license.

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #34598861 | 📁 Comparing 64936ff against latest (4a5892f)

  🔍 Preview build  

223 files changed · + 27 added · ± 196 modified

+ Added

• _autodoc/COUPLING.html
• _autodoc/COUPLING_COEFFICIENT.html
• _autodoc/GROUP.html
• _autodoc/INFLEXIBLE_CONSUMPTION.html
• _autodoc/INFLEXIBLE_PRODUCTION.html
• _autodoc/OPERATION_MODES.html
• _autosummary/flexmeasures.api.common.rate_limiting.html
• _autosummary/flexmeasures.api.v3_0.utils.html
• _autosummary/flexmeasures.data.models.automations.html
• _autosummary/flexmeasures.data.models.planning.devices.html
• and 17 more...

± Modified

• changelog.html
• configuration.html
• genindex.html
• getting-started.html
• http-routingtable.html
• index.html
• py-modindex.html
• _autodoc/COMMITMENTS.html
• _autodoc/CONSUMPTION_CAPACITY.html
• _autodoc/INFLEXIBLE_DEVICE_SENSORS.html
• and 186 more...

[Copilot]

🟡 Changes recommended

A couple of concrete issues need addressing (notably coverage recording in run_report_job and a misleading UI timezone label) before it can be safely approved.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

flexmeasures/data/services/reporting.py:165

• run_report_job records report coverage based solely on the automation_id kwarg. That makes coverage updates possible even when the job isn’t actually automation-triggered (e.g. if get_current_job() is unavailable or meta lacks the expected trigger), which can desynchronize coverage tracking from the runtime sensor-guard logic. Consider deriving the automation id from rq_job.meta['trigger'] and only recording coverage when origin == 'automation', and also explicitly rejecting timezone-naive end values before writing them to Redis.

• Files reviewed: 20/20 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

A Redis failure while recording report-automation coverage can currently cause a successfully stored report job to be marked failed, impacting operational reliability.

Get a fresh assessment by requesting another Copilot review.

Review details

• Files reviewed: 20/20 changed files
• Comments generated: 1
• Review effort level: Lite