You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
An asset's Automations page filled its Jobs column by asking after each automation separately: one
request per row, on every page load, each one walking that automation's job caches. With the
O(automations × sensors × jobs) Redis fanout flagged in the #2290 review underneath it, that is a page
that gets slower with every automation an asset gains. The counts now arrive with the listing itself.
One pass over the caches.[GET] /assets/(id)/automations includes a job-stats per automation.
The cache references of all the asset's automations are collected and deduplicated first, and the jobs
found are then grouped by the automation id recorded on their trigger. An asset whose automations
share sensors — the normal case, since they sit on the same asset — reads each cache once rather than
once per automation.
The page asks once. The Jobs column is filled from the list response, so the page no longer
fires a request per row to populate a column that is visible immediately, and an automation's details
load when they are opened, once, rather than eagerly for every row.
Redis trouble is reported once. The listing carries redis_connection_err, as the jobs endpoint
already does, and reports it against the page rather than against a row. If Redis is unavailable the
listing still renders, with empty counts and one message, instead of every row failing separately. A RedisError is logged and left as empty counts, since a worker's cache being briefly unreachable is
not something to put in front of the reader; only a missing Redis configuration, which is a host
misconfiguration, is reported to the caller.
Added changelog item in documentation/changelog.rst
How to test
The page looks the same — this changes how it loads, not what it shows — so the thing to observe is
the network panel, not the screen. Open /assets/<id>/automations on an asset with several
automations: the Jobs column fills from the single listing request, with no details request per row,
and opening Details fetches that automation once.
The added test drives the listing with a Redis that times out, and asserts the response still arrives,
with empty counts and an error message, rather than failing. It was verified to fail without the
handler it covers. The existing job-statistics tests were moved onto the batched helper, so they now
cover the path the page actually takes.
Part of the automations story #2334, and of #2288. Stacked on #2297. The UI edit action this PR
originally also carried has since arrived by way of #2294.
Sign-off
I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on code under GPL or another incompatible license.
Automations are recurring tasks (for now: computing forecasts) defined per
asset. The recurrence is defined by a cron string, and the work to be done
is defined by a data generator (linked through a data source) together with
the parameters to call it with.
Includes a migration for the new table, and new dependencies on croniter
(cron matching/validation) and cron-descriptor (natural-language recurrence
descriptions).
Part of #2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
- `flexmeasures add automation` creates an automation (active by default),
validating the forecast parameters with the forecast parameter schema and
storing the forecaster config on a data source.
- `flexmeasures edit automation` edits the name, recurrence (cron string)
or activation status.
- `flexmeasures delete automation` deletes an automation.
- All three record their events in the asset's audit log.
- `flexmeasures jobs run-automations` queues jobs for all automations due
this minute (to be run once per minute, e.g. via cron), with a Redis-based
guard against duplicate runs within the same minute.
Part of #2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Data generators can now be told how their queued jobs got triggered (via the
CLI, the API or an automation), and the train-predict pipeline stores this
on the jobs as meta data. The asset's status page shows it in a new
'Created Via' column of the jobs table.
Part of #2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
GET /api/v3_0/assets/<id>/automations lists the automations defined on an
asset (without generator and parameters details). GET
/api/v3_0/assets/<id>/automations/<automation_id> additionally provides the
parameters, data generator info and counts of recently created jobs per job
status. Both are documented in the OpenAPI specs.
Part of #2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
/assets/<id>/automations shows the asset's automations in a tabbed view
(schedules and reports tabs are prepared but deactivated), with per-row
details (parameters, data generator, job counts) loaded asynchronously into
a modal. The page is linked in the breadcrumbs dropdown and links to the
status page, where recent jobs are listed.
Part of #2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
CI runners have no locale set (POSIX), which made cron-descriptor render
'At 06:00' while dev environments with an en_US-style locale rendered
'At 06:00 AM'. Request 24-hour format explicitly so the description is
deterministic across environments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxkeq64jtENY7fiWjwUsVS
- Escape automation names (and other user-controlled strings) in the
Automations page and the status page's jobs table, closing two stored
HTML/script injection sinks.
- Wipe parameter state on the (possibly shared) cached data generator before
each automation run, so automations sharing a generator data source don't
pollute each other's runs.
- Count automation job stats under the forecast target sensor(s) from the
automation's parameters, which may belong to a different asset.
- Release the per-minute Redis guard when a run fails, so a retry within the
same minute can still queue jobs.
- Return 404 (as documented) for nonexistent automation ids on the detail
endpoint, and check permissions on the asset, so automation ids can no
longer be enumerated across accounts via 403-vs-422 differences.
- Use ondelete=SET NULL for the generator FK: deleting a data source no
longer silently deletes automations.
- Delegate Automation ACL to the asset's ACL instead of duplicating it.
- Extract the config/parameters assembly shared by `add forecasts` and
`add automation` into a helper (which no longer drops falsy config values).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Completes the previous commit, whose staged files were dropped by an
interrupted pre-commit run: template escaping, shared-generator state reset,
job stats under target sensors, Redis guard release on failure, 404 for
nonexistent automations, SET NULL generator FK, ACL delegation, and the
shared CLI config/parameters assembly helper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
The scheduling job creators accept an optional trigger dict (stored as job
meta data), like the forecasting pipeline already does. The API trigger
endpoint records origin API; the CLI and automations follow in the next
commit. The status page's 'Created Via' column picks this up automatically.
Part of #2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Automations now also support the 'schedules' type:
- `flexmeasures add automation --type schedules` validates the parameters as
a schedule trigger message (per the AssetTriggerSchema, as accepted by the
API trigger endpoint, without the asset id). The schedule 'start' may be
omitted, in which case each run schedules from the run time (floored to the
message's resolution, if given) — a fixed start draws a warning.
- The runner dispatches schedules automations to the same job creators as the
API trigger endpoint (sequential or simultaneous), recording trigger meta
data (origin automation) on the queued jobs; `flexmeasures add schedule
--as-job` now records origin CLI.
- Job stats for schedules automations are counted from the scheduling job
cache (asset-level wrap-up jobs and per-sensor device jobs).
- The UI automations page's Schedules tab is now enabled, with automations
filtered by type per tab.
Part of #2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
- New endpoints on assets: POST /automations (create, validating parameters
by automation type), PATCH /automations/<id> (name, cron string, activation
status) and DELETE /automations/<id>. Managing automations requires the
same principals that may delete the asset (account admins and consultants).
- The UI automations page gets a 'New automation' modal and per-row
(de)activate and delete actions, shown to users with management rights.
- Creation, update and deletion logic (incl. audit log records) moved into
the automations service, shared by the CLI commands and the API endpoints.
Part of #2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
- Activate the reporting queue (it was prepared but commented out), including
worker help texts and queue cleanup.
- Reporters accept as_job: a job is queued (with trigger meta data) that
rebuilds the reporter from its data source, computes the report and saves
the results to the database.
- `flexmeasures add report --as-job` queues such a job; reporting jobs show
up in the asset's jobs overview (status page and API).
Part of #2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Automations can now compute reports on a recurring basis:
- `flexmeasures add automation --type reports --reporter <class>` stores the
reporter config on a data source (steady across runs, so all report results
attribute to the same source) and validates the report parameters.
- The report window resolves freshly on each run: 'start-offset'/'end-offset'
fields (comma-separated Pandas offsets, applied to the run time in the
first output sensor's timezone) express a rolling window, and without any
timing fields the window defaults to the last cron period (from the
previous cron fire time until the run time). Absolute start/end still work,
but draw a warning.
- The API creation field 'forecaster' is generalized to 'generator' (also
accepting reporter classes), and the UI's New automation modal gains data
generator and config fields; the Reports tab is now enabled.
Part of #2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Each automation run is recorded in Redis; a report automation without timing
fields then reports on the period since its actual last run, falling back to
the last cron period when no last run is known (e.g. on the first run, or
after a Redis flush). This gives gapless coverage even when runs are missed.
Part of #2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Consolidates the shared automations concept (model, lifecycle, runner
deployment, provenance) into documentation/features/automations.rst, with
the per-feature pages linking to it and keeping only their type-specific
parameter semantics.
Part of #2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
- The automations page gets an Edit action (name and cron string), using the
existing PATCH endpoint.
- The list endpoint now includes per-automation job stats, computed in a
single pass over the relevant job caches, instead of the UI firing one
detail request per automation on page load; the Details modal loads its
contents lazily, when first opened.
Part of #2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
- API automation creation now checks that the caller may read every sensor
referenced in the parameters/config and record data on the sensors the
automation writes to, closing a cross-account data read/write hole.
- `add report --as-job` implies --save-config (the worker rebuilds the
reporter from its data source, so jobs without stored config always crashed).
- Report automation parameters are validated with the chosen reporter's own
parameters schema, not the base schema.
- Invalid start-offset/end-offset strings are rejected at creation instead of
being silently skipped at run time (which yielded empty report windows).
- run_report_job wipes the shared cached reporter's parameter state, like the
automation runner already did, so consecutive jobs in one worker process
don't pollute each other.
- Default report windows now anchor to the end of the last *successfully*
covered window, recorded by the reporting job upon success — failed jobs no
longer create permanent reporting gaps, and the enqueue-time minute-rollover
gap is gone (the recorded anchor is the window end itself).
- The cron-period fallback window is computed in the platform timezone,
matching how the runner decides when automations fire.
- Job stats for schedules automations also scan flex-model device sensors
(which may belong to child assets), so failed per-device jobs show up.
- The trigger provenance kwarg is excluded from the job cache hash, so
identical schedule requests from different origins dedupe again.
Part of #2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
…essage format
PR #2303 makes click report the validation message rather than the offending
value, which changes the exact wording of this error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Merge current main, resolve the shared forecasting and documentation changes, regenerate the lockfile, and move the automation migration after the current migration head.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Reject cron expressions with seconds, year fields, or aliases because the automation runner executes once per minute.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Test valid five-field expressions and reject unsupported seconds, year, and alias formats.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Keep the per-minute Redis guard after failures because an attempt may already have queued some forecast jobs.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
This branch already held #2297's commits, so every conflict with their
squashed form is between two representations of the same content, and
this branch's side is taken (see feature-branch-sync.instructions.md).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Resolve the changelog conflict by keeping both entries.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC
Signed-off-by: F.N. Claessen <claessen@seita.nl>
…o fixed moments (#2551)
* feat: add Automation data model
Automations are recurring tasks (for now: computing forecasts) defined per
asset. The recurrence is defined by a cron string, and the work to be done
is defined by a data generator (linked through a data source) together with
the parameters to call it with.
Includes a migration for the new table, and new dependencies on croniter
(cron matching/validation) and cron-descriptor (natural-language recurrence
descriptions).
Part of FlexMeasures/flexmeasures#2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
* feat: CLI commands to manage and run automations
- `flexmeasures add automation` creates an automation (active by default),
validating the forecast parameters with the forecast parameter schema and
storing the forecaster config on a data source.
- `flexmeasures edit automation` edits the name, recurrence (cron string)
or activation status.
- `flexmeasures delete automation` deletes an automation.
- All three record their events in the asset's audit log.
- `flexmeasures jobs run-automations` queues jobs for all automations due
this minute (to be run once per minute, e.g. via cron), with a Redis-based
guard against duplicate runs within the same minute.
Part of FlexMeasures/flexmeasures#2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
* feat: record on forecasting jobs how they were created
Data generators can now be told how their queued jobs got triggered (via the
CLI, the API or an automation), and the train-predict pipeline stores this
on the jobs as meta data. The asset's status page shows it in a new
'Created Via' column of the jobs table.
Part of FlexMeasures/flexmeasures#2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
* feat: API endpoints to list an asset's automations
GET /api/v3_0/assets/<id>/automations lists the automations defined on an
asset (without generator and parameters details). GET
/api/v3_0/assets/<id>/automations/<automation_id> additionally provides the
parameters, data generator info and counts of recently created jobs per job
status. Both are documented in the OpenAPI specs.
Part of FlexMeasures/flexmeasures#2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
* feat: UI page listing an asset's automations
/assets/<id>/automations shows the asset's automations in a tabbed view
(schedules and reports tabs are prepared but deactivated), with per-row
details (parameters, data generator, job counts) loaded asynchronously into
a modal. The page is linked in the breadcrumbs dropdown and links to the
status page, where recent jobs are listed.
Part of FlexMeasures/flexmeasures#2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
* test: cover automations CLI, API and UI
Part of FlexMeasures/flexmeasures#2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
* docs: document automations
Part of FlexMeasures/flexmeasures#2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
* docs: changelog entry for automations
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
* fix: render cron descriptions in 24-hour format regardless of locale
CI runners have no locale set (POSIX), which made cron-descriptor render
'At 06:00' while dev environments with an en_US-style locale rendered
'At 06:00 AM'. Request 24-hour format explicitly so the description is
deterministic across environments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxkeq64jtENY7fiWjwUsVS
* fix: address code review findings for automations
- Escape automation names (and other user-controlled strings) in the
Automations page and the status page's jobs table, closing two stored
HTML/script injection sinks.
- Wipe parameter state on the (possibly shared) cached data generator before
each automation run, so automations sharing a generator data source don't
pollute each other's runs.
- Count automation job stats under the forecast target sensor(s) from the
automation's parameters, which may belong to a different asset.
- Release the per-minute Redis guard when a run fails, so a retry within the
same minute can still queue jobs.
- Return 404 (as documented) for nonexistent automation ids on the detail
endpoint, and check permissions on the asset, so automation ids can no
longer be enumerated across accounts via 403-vs-422 differences.
- Use ondelete=SET NULL for the generator FK: deleting a data source no
longer silently deletes automations.
- Delegate Automation ACL to the asset's ACL instead of duplicating it.
- Extract the config/parameters assembly shared by `add forecasts` and
`add automation` into a helper (which no longer drops falsy config values).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
* fix: address code review findings for automations (remaining files)
Completes the previous commit, whose staged files were dropped by an
interrupted pre-commit run: template escaping, shared-generator state reset,
job stats under target sensors, Redis guard release on failure, 404 for
nonexistent automations, SET NULL generator FK, ACL delegation, and the
shared CLI config/parameters assembly helper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
* feat: record on scheduling jobs how they were created
The scheduling job creators accept an optional trigger dict (stored as job
meta data), like the forecasting pipeline already does. The API trigger
endpoint records origin API; the CLI and automations follow in the next
commit. The status page's 'Created Via' column picks this up automatically.
Part of FlexMeasures/flexmeasures#2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
* feat: schedules as automations
Automations now also support the 'schedules' type:
- `flexmeasures add automation --type schedules` validates the parameters as
a schedule trigger message (per the AssetTriggerSchema, as accepted by the
API trigger endpoint, without the asset id). The schedule 'start' may be
omitted, in which case each run schedules from the run time (floored to the
message's resolution, if given) — a fixed start draws a warning.
- The runner dispatches schedules automations to the same job creators as the
API trigger endpoint (sequential or simultaneous), recording trigger meta
data (origin automation) on the queued jobs; `flexmeasures add schedule
--as-job` now records origin CLI.
- Job stats for schedules automations are counted from the scheduling job
cache (asset-level wrap-up jobs and per-sensor device jobs).
- The UI automations page's Schedules tab is now enabled, with automations
filtered by type per tab.
Part of FlexMeasures/flexmeasures#2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
* docs: changelog entry for schedule automations
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
* feat: CRUD for automations via API and UI
- New endpoints on assets: POST /automations (create, validating parameters
by automation type), PATCH /automations/<id> (name, cron string, activation
status) and DELETE /automations/<id>. Managing automations requires the
same principals that may delete the asset (account admins and consultants).
- The UI automations page gets a 'New automation' modal and per-row
(de)activate and delete actions, shown to users with management rights.
- Creation, update and deletion logic (incl. audit log records) moved into
the automations service, shared by the CLI commands and the API endpoints.
Part of FlexMeasures/flexmeasures#2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
* docs: changelog entry for automations CRUD
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
* feat: reports can run as background jobs
- Activate the reporting queue (it was prepared but commented out), including
worker help texts and queue cleanup.
- Reporters accept as_job: a job is queued (with trigger meta data) that
rebuilds the reporter from its data source, computes the report and saves
the results to the database.
- `flexmeasures add report --as-job` queues such a job; reporting jobs show
up in the asset's jobs overview (status page and API).
Part of FlexMeasures/flexmeasures#2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
* feat: reports as automations
Automations can now compute reports on a recurring basis:
- `flexmeasures add automation --type reports --reporter <class>` stores the
reporter config on a data source (steady across runs, so all report results
attribute to the same source) and validates the report parameters.
- The report window resolves freshly on each run: 'start-offset'/'end-offset'
fields (comma-separated Pandas offsets, applied to the run time in the
first output sensor's timezone) express a rolling window, and without any
timing fields the window defaults to the last cron period (from the
previous cron fire time until the run time). Absolute start/end still work,
but draw a warning.
- The API creation field 'forecaster' is generalized to 'generator' (also
accepting reporter classes), and the UI's New automation modal gains data
generator and config fields; the Reports tab is now enabled.
Part of FlexMeasures/flexmeasures#2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
* docs: changelog entry for reports as jobs and automations
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
* feat: anchor default report windows to the automation's actual last run
Each automation run is recorded in Redis; a report automation without timing
fields then reports on the period since its actual last run, falling back to
the last cron period when no last run is known (e.g. on the first run, or
after a Redis flush). This gives gapless coverage even when runs are missed.
Part of FlexMeasures/flexmeasures#2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
* docs: add an Automations concept page
Consolidates the shared automations concept (model, lifecycle, runner
deployment, provenance) into documentation/features/automations.rst, with
the per-feature pages linking to it and keeping only their type-specific
parameter semantics.
Part of FlexMeasures/flexmeasures#2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
* fix: address stack code review findings
- API automation creation now checks that the caller may read every sensor
referenced in the parameters/config and record data on the sensors the
automation writes to, closing a cross-account data read/write hole.
- `add report --as-job` implies --save-config (the worker rebuilds the
reporter from its data source, so jobs without stored config always crashed).
- Report automation parameters are validated with the chosen reporter's own
parameters schema, not the base schema.
- Invalid start-offset/end-offset strings are rejected at creation instead of
being silently skipped at run time (which yielded empty report windows).
- run_report_job wipes the shared cached reporter's parameter state, like the
automation runner already did, so consecutive jobs in one worker process
don't pollute each other.
- Default report windows now anchor to the end of the last *successfully*
covered window, recorded by the reporting job upon success — failed jobs no
longer create permanent reporting gaps, and the enqueue-time minute-rollover
gap is gone (the recorded anchor is the window end itself).
- The cron-period fallback window is computed in the platform timezone,
matching how the runner decides when automations fire.
- Job stats for schedules automations also scan flex-model device sensors
(which may belong to child assets), so failed per-device jobs show up.
- The trigger provenance kwarg is excluded from the job cache hash, so
identical schedule requests from different origins dedupe again.
Part of FlexMeasures/flexmeasures#2288
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
* test: assert on the cron validation failure without pinning click's message format
PR #2303 makes click report the validation message rather than the offending
value, which changes the exact wording of this error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
* data/schemas: restrict automations to five-field cron
Reject cron expressions with seconds, year fields, or aliases because the automation runner executes once per minute.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* data/schemas/tests: cover automation cron field count
Test valid five-field expressions and reject unsupported seconds, year, and alias formats.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* cli/jobs: retain automation guard after queueing failure
Keep the per-minute Redis guard after failures because an attempt may already have queued some forecast jobs.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* cli/tests: cover partial automation queue failure
Verify that retrying a failed partial queueing attempt does not create duplicate jobs.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* cli: normalize YAML forecasting option files
Convert YAML dates to ISO strings, accept empty files, and report non-object config or parameter files as usage errors.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* cli/tests: cover automation YAML option files
Test YAML dates and timestamps, empty files, and invalid top-level list values for automation options.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* data/services: redact inaccessible automation provenance
Hide automation names and IDs from asset job responses when the current user cannot read the source automation.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* api/v3_0/tests: cover automation provenance authorization
Verify inaccessible automation provenance is redacted while authorized callers still receive the full identity.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* ui/assets: distinguish automation load failures
Show a persistent API error instead of presenting failed automation requests as an empty list.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* ui/tests: cover automation load error state
Check that the automations page renders the warning target and hides the table when loading fails.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* utils/docs: preserve standalone asterisks in RST conversion
Avoid interpreting cron wildcard asterisks as RST italic markup when generating OpenAPI documentation.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* utils/tests: cover RST cron wildcard conversion
Verify cron wildcards remain unchanged while ordinary italic markup is still converted.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* docs/forecasting: clarify automation execution contract
Document five-field cron expressions, at-most-once queueing attempts, and the automations API endpoint.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* changelog: record automation API and runner contract
Record the automation endpoints, authorization-aware provenance, and five-field runner behavior.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* api/docs: show job creation provenance
Include the created_via field in the asset jobs OpenAPI example.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* test: keep forecast CLI stub compatible with job provenance
Add the trigger method required by the forecasting CLI to the regressor parsing test double.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* fix: require valid automation generators
Require every automation to reference a data generator and prevent deleting a data source while an automation still depends on it, matching the retention policy for belief and annotation sources.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* test: cover automation generator retention
Verify referenced generators cannot be deleted, generator references cannot be cleared, and automation API fixtures always use valid generators.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* fix: constrain forecast automation outputs
Allow forecast output only on the automation asset or its descendants and revalidate that relationship before every scheduled run, including explicit sensor-to-save targets.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* test: cover forecast automation output scope
Cover same-asset, child, grandchild, ancestor, and unrelated output targets, explicit sensor-to-save behavior, and runtime revalidation after an asset is moved.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* docs: explain forecast automation ownership rules
Document output-sensor scope, runtime relationship checks, and the requirement to retain a generator while its automation exists.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* fix: merge automation and main migration heads
Join the automation and main Alembic branches so installations have a single database upgrade target.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* data/models: allow schedule automations without generators
Context:
- Schedule automations do not use a data generator, but the reviewed forecast automation schema required one.
Change:
- Make the foreign key nullable while retaining a database check that forecasts always have a generator.
- Add a forward migration without weakening data-source deletion semantics.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* cli/tests: cover schedule automation validation
Context:
- Review uncovered untested forecast-only options, invalid durations, and DST start calculation.
Change:
- Add CLI and trigger-preparation regressions while retaining forecast sensor validation.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* data/tests: cover schedule automation dispatch
Context:
- Persistence, inherited flex configuration, descendant statistics, and job counts lacked realistic coverage.
Change:
- Move automation service tests under the fresh-database fixture and add end-to-end schedule regressions.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* api/v3_0/tests: cover schedule job provenance
Context:
- API provenance was not asserted across asset and sensor schedule jobs.
Change:
- Verify API trigger metadata on sequential descendants, wrap-up jobs, and sensor jobs.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* ui/tests: cover automation type tabs
Context:
- The asset automation page tests still targeted the former single table.
Change:
- Assert type-specific tables, error rendering, filters, and hidden-tab column adjustment.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* scheduling: harden automation dispatch
Context:
- Minimal asset schedules lost stored defaults, invalid timing could execute, provenance was incomplete, and descendant jobs were miscounted.
Change:
- Validate and floor fixed durations safely, inherit stored flex configuration, preserve provenance, and count the jobs actually dispatched.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* cli: reject forecast options for schedule automations
Context:
- Schedule automation creation silently accepted forecaster settings that could never affect scheduling.
Change:
- Detect supplied forecast-only options and return a user-facing usage error.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* ui/assets: resize automation tables on tab changes
Context:
- DataTables initialized in the hidden schedule tab could render with stale column widths.
Change:
- Add tab accessibility state and adjust initialized table columns when a tab becomes visible.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* docs: clarify schedule automation inputs
Context:
- The feature guide linked only to the API root and omitted fixed-start and duration constraints.
Change:
- Document canonical fields, runtime start behavior, timing validation, and generator-free schedule automations.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* cli/tests: cover malformed automation YAML
Context:
- Manual testing exposed raw parser exceptions for malformed automation files.
Change:
- Require a user-facing usage error for invalid YAML in both config and parameter files.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* cli: report malformed automation YAML
Context:
- PyYAML parser errors escaped automation creation without a useful CLI message.
Change:
- Translate malformed config and parameter files into a normal Click usage error.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* data/tests: cover stored schedule flex configuration
Context:
- Manual execution showed minimal automations failing for a one-device asset tree.
Change:
- Exercise both simultaneous and sequential dispatch using realistic flex configuration stored on the child asset.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* scheduling: load stored flex config for minimal triggers
Context:
- A one-device asset tree collapsed to sensor scheduling without a sensor, and sequential dispatch could not resolve its stored output.
Change:
- Preserve asset-triggered flex models as a list and resolve sequential device sensors from stored consumption or production outputs.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* docs/scheduling: describe trigger propagation
Context:
- Scheduling service docstrings did not distinguish single-job and sequential provenance behavior.
Change:
- Document where trigger metadata is stored and correct the simultaneous return description.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* data/models: let data generators report their input and output sensors
Context:
- Review of #2290 asked for a data generator property listing the sensors it
reads from and writes to, so that automations can link to those sensors
(and, later, check the creating user's permissions on them)
Change:
- Added input_sensors and output_sensors to DataGenerator (empty by default),
implemented for Forecaster from its regressors and target sensor
- Added the same properties to Automation, resolved from its data generator
configured with the automation's own parameters
- Added get_automations_feeding_sensor to look up automations by output sensor
Signed-off-by: F.N. Claessen <felix@seita.nl>
* data/models/forecasting: only announce a pipeline run when actually running it
Context:
- 'flexmeasures jobs run-automations' logged 'Starting Train-Predict Pipeline'
for every automation, while it only queues the cycles as jobs
Change:
- Log that line at debug level when running with as_job, where the workers
running the cycles log their own start
Signed-off-by: F.N. Claessen <felix@seita.nl>
* cli: default the automation recurrence to daily, and reject options that --source already determines
Context:
- Review of #2290: --cron should not be required, and --forecaster/--config are
redundant with --source, whose data generator attributes already hold both
(get_data_generator silently ignores them when a source is given)
Change:
- Default --cron to '0 0 * * *' (daily at midnight)
- Abort when --source is combined with --forecaster, --config or any of the
forecaster configuration options, naming the conflicting options
Signed-off-by: F.N. Claessen <felix@seita.nl>
* api/v3_0: report an automation's input and output sensors
Context:
- The automation details modal should link to the sensors an automation feeds
Change:
- Added input_sensors and output_sensors (id and name each) to
GET /assets/<id>/automations/<automation_id>
Signed-off-by: F.N. Claessen <felix@seita.nl>
* api/v3_0: add an endpoint for one data source
Context:
- The sensor page should be able to show the full record of a data source,
including the attributes in which data generators store their configuration
Change:
- Added GET /sources/<id>, with the same access rules as listing sources
- Let _serialize_source optionally include the attributes and unset fields
Signed-off-by: F.N. Claessen <felix@seita.nl>
* api/v3_0: regenerate the OpenAPI specs
Context:
- The specs are generated from the endpoint docstrings by a pre-commit hook
Change:
- Regenerated after adding the data source endpoint and the automation's
input and output sensors
Signed-off-by: F.N. Claessen <felix@seita.nl>
* ui: link an automation's details to its sensors, and make the listing sortable
Context:
- Review of #2290: the details modal should link to the sensors an automation
feeds, with its data source pre-selected there, and the listing was not sortable
Change:
- Show the input and output sensors in the details modal, linking to
/sensors/<id>?source=<generator id>
- Enabled ordering, sorting the rendered columns on separate values (the ISO
timestamp, the activation status and the cron string), newest first
Signed-off-by: F.N. Claessen <felix@seita.nl>
* ui: show a sensor's data source record and the automations feeding it
Context:
- Review of #2290: the sensor page should be able to show all details of a data
source, and list the automations that write data to the sensor
Change:
- Added an info button next to the source selector, opening a modal with the
full data source record
- Pre-select the source given in the source query parameter, so that links from
an automation land on its own source
- List the automations feeding the sensor (those the user may read), linking to
the automations page of their asset
- Added user_can_read to the UI's permission helpers
Signed-off-by: F.N. Claessen <felix@seita.nl>
* tests: cover the automation and data source review follow-ups
Context:
- New behaviour from the #2290 review needs regression coverage
Change:
- CLI: the daily default recurrence, the --source conflict, and an automation's
input and output sensors
- API: an automation without a data generator reports no sensors; the new data
source endpoint, its access rules and its 404
- UI: the source query parameter reaches the page, and automations feeding a
sensor are listed on it
Signed-off-by: F.N. Claessen <felix@seita.nl>
* docs: describe the automation and data source follow-ups
Context:
- The #2290 review changed user-facing CLI, API and UI behaviour
Change:
- Documented the daily default recurrence and reusing a forecaster via --source
- Documented the links between automations and the sensors they feed
- Added API change log entries for the automations and data source endpoints
- Extended the changelog entry of #2290
Signed-off-by: F.N. Claessen <felix@seita.nl>
* api/v3_0: regenerate the OpenAPI specs after merging
Context:
- The merge combined endpoint docstring changes from both sides
Change:
- Regenerated the specs
Signed-off-by: F.N. Claessen <felix@seita.nl>
* cli: only reject configuration options that were actually given with --source
Context:
- The guard added on this branch compared against the assembled config, which
always holds the schema defaults of the list-valued options, so any use of
--source was rejected
Change:
- Detect the conflicting options from click's parameter sources, so --source on
its own works again while explicitly given configuration options still abort
- Name the conflicting options in the error message
Signed-off-by: F.N. Claessen <felix@seita.nl>
* data/services: only consider automations that could feed a sensor
Context:
- Listing the automations feeding a sensor sets up a data generator per
candidate, which does not need to happen for every automation in the database
Change:
- Narrowed the candidates to automations on the sensor's asset or an ancestor,
which is where an automation writing to it must live
Signed-off-by: F.N. Claessen <felix@seita.nl>
* tests: follow the merged automation behaviour
Context:
- Automations now always have a data generator, and the --source guard also
covers the forecaster configuration options
Change:
- Assert the sensors an automation with a generator reads from and writes to
- Cover a configuration option conflicting with --source
Signed-off-by: F.N. Claessen <felix@seita.nl>
* data/services: only let a user automate sensors they can access themselves
Context:
- Review of #2290 asked that automations administered through the UI (and hence
the API) may only involve sensors the creating user has access to; account
admin rights on the asset should not grant access to another account's sensors
Change:
- Work out the sensors an automation would read from and write to (forecasts:
the sensor to forecast plus its regressors, and the sensor to save to;
schedules: the flex-model's device sensors, and whatever the parameters refer to)
- Require read access to the former and create-children (the permission for
recording data through the API) on the latter, when creating via the API
- The CLI creates automations without a user, and stays unrestricted
Signed-off-by: F.N. Claessen <felix@seita.nl>
* api/v3_0: regenerate the OpenAPI specs
Context:
- The endpoint description now states the sensor access rule
Change:
- Regenerated the specs
Signed-off-by: F.N. Claessen <felix@seita.nl>
* api/v3_0/tests: cover automating an inaccessible sensor
Context:
- The sensor access rule for created automations needs regression coverage
Change:
- An account admin creating an automation on another account's sensor gets a 403
naming that sensor, and no automation is created; the same request on their own
sensor still succeeds (verified to fail without the check)
Signed-off-by: F.N. Claessen <felix@seita.nl>
* docs: describe which sensors an automation may involve
Context:
- The sensor access rule is user-facing
Change:
- Documented it in the forecasting feature docs, the changelog entry of #2294
and the API change log
Signed-off-by: F.N. Claessen <felix@seita.nl>
* data/services: check every sensor a schedule would be recorded on
Context:
- Schedulers hand their results to make_schedule as (sensor, data) pairs, and
those sensors are not only the flex-model's device sensors: a schedule is also
recorded on a device's state-of-charge, consumption and production sensors, and
on the flex-context's aggregate-consumption and aggregate-production sensors
Change:
- Derive a schedule's output sensors from all the fields that name where generated
data goes, at any depth in the flex-model and flex-context (which schedulers
deserialize themselves, so their sensor references are still raw)
- Everything else the parameters refer to (e.g. price sensors and the sensors of
inflexible devices, which may also live on the flex-context) counts as an input
Signed-off-by: F.N. Claessen <felix@seita.nl>
* api/v3_0/tests: cover a schedule aggregated onto an inaccessible sensor
Context:
- The flex-context's aggregate-consumption sensor is written to, so it needs the
same check as the flex-model's own sensors
Change:
- Posting such an automation gets a 403 that names the sensor and the action
(verified to fail when only the flex-model's sensors are treated as outputs)
Signed-off-by: F.N. Claessen <felix@seita.nl>
* cli: keep mypy happy about click 8 attributes
types-Flask pins types-click 7.1, whose stubs shadow the inline types that click ships itself.
Those stubs predate ParameterSource and Context.get_parameter_source, both added in click 8.0,
so mypy rejected the --source conflict detection and pre-commit failed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* cli: keep the automation help focused on the automation
`add automation` reuses the forecast schemas, so Click rendered every forecaster and pipeline option in its help,
burying the options that describe the automation itself.
Accept those options still, but hide them, and let the parameters file supply a field the schema requires,
so --sensor no longer has to be repeated on the command line.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* data/models: count a source-filtered regressor as an input sensor
SensorIdOrReferenceField deserializes a regressor that filters on sources into a SensorReference rather than a Sensor,
which _resolve_sensors skipped, so those regressors were missing from a data generator's input sensors.
The source filters only narrow down which beliefs are read from a sensor, not which sensor is involved,
so the wrapped sensor counts as an input just like a plain sensor ID does.
This matters beyond the sensor links in the UI: the input and output sensors are meant to carry the access checks
for automations administered through the API, where a missing input sensor means a missing permission check.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* data/services: do not report no sensors when an automation's sensors are unknown
Working out an automation's sensors could fail for several reasons, and every one of them was reported as no sensors at all.
That is fine for the sensor links in the UI, but the same answer is meant to carry the access checks
for automations administered through the API, where no sensors reads as nothing to check,
so a broken automation would pass every check on the sensors it involves.
Split the two uses: resolve_automation_sensors raises AutomationSensorsUnknown,
while get_automation_sensors keeps reporting none for display.
The broad exception handler is narrowed to the failures that can actually occur here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Feat automation timezones catchup (#2396)
* feat: add timezone-aware automation catch-up
Store each automation's IANA timezone and a durable UTC scheduling watermark. Canonicalize daylight-saving transitions, coalesce missed forecasts, and claim occurrences before queueing while retaining the existing at-most-once failure policy.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* test: cover timezone-aware automation scheduling
Exercise timezone defaults and validation, independent timezone evaluation, normal and missed occurrences, DST gaps and folds, durable cursor claims, inactive automations, API/UI exposure, and the existing Redis failure guard.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* docs: explain automation timezone and catch-up semantics
Document per-automation timezone snapshots, scheduling watermarks, downtime coalescing, daylight-saving behavior, reconfiguration boundaries, and the at-most-once retry limitation. Refresh the published automation API examples.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* changelog: record automation timezone and catch-up support
Announce the new CLI options, additive automation response fields, persistent scheduling progress, and DST-aware catch-up behavior for issue #2392.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* docs: link automation catch-up changelog to PR
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
---------
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* data/migrations: rejoin the two automation migration branches
Merging the timezone and catch-up work alongside the schedule automations left two alembic heads,
one adding an automation's timezone and scheduling cursor and one allowing a schedule automation without a data generator,
so flexmeasures db upgrade refused to run and the Docker image build failed.
The two touch different columns, so the merge point has nothing of its own to do.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* data/services: let the forecaster say which sensors an automation involves
A regressor that filters on sources deserializes into a sensor reference rather than a sensor,
which collect_sensors skipped, so such a regressor was left out of the sensors an automation reads from.
The access check is built on that list, so a user could set up an automation reading a sensor they cannot read themselves.
Ask the forecaster instead, as it derives its input and output sensors from the same config and parameters it will run with,
and already resolves sensor references. Schedules keep their own collection, as they have no data generator to ask.
Displaying the sensors involved and checking access to them now share one implementation, so they cannot disagree.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* api/v3_0: let an automation's timezone be set and changed through the API
An automation carries the timezone its cron expression is interpreted in, and the CLI can set and change it,
but the API could do neither, so every automation created through the API or the UI was stuck on the server's timezone.
Both the creation and the update schema now accept a timezone, defaulting to FLEXMEASURES_TIMEZONE on creation.
Also restores the OpenAPI spec's version string, which a regeneration during the merge had replaced with the locally installed version.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* data/services: set up the forecaster's data source only once the automation is allowed
Creating an automation looked up or created the data source holding the forecaster configuration before checking
whether the user may involve the sensors at all, so a refused request still added a data source within that request.
Nothing committed in between, so this did not outlive the request, but it relied on that rather than on the order of events.
The data source is now set up after the access check, which makes a refused request leave nothing behind by construction.
Also records what the output sensor field list approximates, namely the sensors a scheduler returns results for at run time,
and therefore how it can drift away from them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* fix(data/schemas): reject cron expressions without dates
Validate that a syntactically correct five-field recurrence can produce an actual calendar occurrence, preventing impossible dates such as February 31 from entering the automation table through either CLI or API input.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* fix(data/services): isolate invalid recurrences and stale claims
Keep one legacy or corrupted recurrence from aborting global discovery, and make occurrence claiming an atomic comparison against the active state, recurrence, timezone, and cursor observed by the runner so concurrent edits cannot queue stale work.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* fix(api/v3_0): protect automation sensor details
Require read access to every resolved input and output sensor before returning full automation details, ensuring the derived metadata and raw parameters cannot reveal cross-organisation sensor information to an asset reader.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* test(cli): cover impossible recurrence input
Exercise the CLI validation path with a syntactically valid recurrence that can never match, while updating the runner fixture to carry the scheduling snapshot required by atomic occurrence claims.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* test(data/services): cover resilient automation claims
Verify that corrupt recurrences are isolated and that deactivation, deletion, recurrence edits, timezone edits, or cursor movement after discovery prevent a stale claim, while non-execution name edits remain harmless.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* test(api/v3_0): cover private automation dependencies
Build an automation with a supplier-owned regressor and confirm that a plain member who may read the automation asset receives a generic denial without the inaccessible sensor name.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* fix(data/services): resolve schedule automation sensors
Prepare the scheduling configuration through the same scheduler collection path used before queueing, then derive declared input and output sensors so generator-free schedule automations expose their stored flex dependencies and participate in sensor relationships.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* test(data/services): cover schedule sensor resolution
Exercise a minimal schedule that inherits its flex model and context from the asset tree, confirming that price inputs, schedule outputs, and the sensor-to-automation relationship are all reported from stored configuration.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* test(api/v3_0): expose schedule dependency details
Create a generator-free schedule with stored price and output sensors and assert that the details endpoint returns both dependency sets instead of reporting empty arrays.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* fix(data/services): hide inaccessible sensor names
Permission failures now identify an inaccessible automation dependency only by the sensor ID supplied in the request. This preserves a useful reference for the caller without confirming private sensor names across organisation boundaries.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* test(api/v3_0): isolate automation endpoint tests
Automation endpoint tests now use function-scoped fresh database fixtures because they create, update, and delete automations and related sensors. The permission cases also assert that forbidden responses retain the submitted sensor ID without disclosing its private name.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* feat(ui/views): provide automation timezones
The asset automations view now supplies the canonical IANA timezone choices accepted by the automation schema. Keeping the options server-side ensures the create and edit controls offer the same vocabulary that the API validates.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* feat(ui): edit automation recurrence timezones
Managers can now choose an IANA timezone when creating an automation and edit its name, recurrence, timezone, and active state from the asset page. New automations default to the asset timezone, while the API remains responsible for validating every submitted value.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* test(ui): cover automation timezone controls
The asset page regression test verifies that managers receive create and edit timezone fields, that creation starts from the asset timezone, and that both forms include their selected timezone in the corresponding API request.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* docs(changelog): mention automation timezones
The automation CRUD entry now records that recurrence timezones are selectable in the user interface and uses the established organisation terminology for the people allowed to manage them.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* fix(data/services): preserve schedule validation errors
Schedule sensor discovery now leaves creation-time schema and scheduler errors intact so the CLI and API can render their established validation responses. Stored automation resolution still wraps those failures as unknown dependencies for strict permission checks and lenient displays.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* fix(automations): authorize reporter sensor dependencies
Make reporters declare the sensors they read from and write to so automation creation, detail rendering, and sensor links use one authoritative dependency model. Profit and loss reports now include price sensors stored in reporter configuration, preventing cross-organisation report automations from bypassing access checks, and the incomplete duplicate scanner in the API is removed.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* fix(reporting): require report inputs and outputs
Require every report payload to declare at least one output and preserve the inherited required constraints when specialized profit and aggregation schemas narrow list lengths. Invalid automations now fail during API or CLI validation instead of being accepted and later crashing a reporting worker with missing method arguments.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* fix(automations): constrain report output scope
Apply the existing automation subtree invariant to report outputs during both creation and execution, after access checks have established that sensor metadata may be discussed. Reporter parameters are prepared before dependency resolution so recurring reports with relative windows expose and validate their sensors consistently on API details and worker runs.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* fix(automations): require generators for reports
Extend the database invariant that protects executable automations so report rows, like forecast rows, cannot exist without a data generator. The migration replaces the forecast-only check constraint while preserving generator-free schedule automations and provides a reversible downgrade to the previous rule.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* fix(automations): anchor reports to claimed occurrences
Build recurring report windows from the canonical cron occurrence claimed by the runner and interpret prior occurrences in each automation's own IANA timezone. Delayed catch-up runs now produce stable boundaries, including across daylight-saving gaps, instead of using the platform timezone and the worker's later wall-clock time.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* fix(automations): advance report coverage monotonically
Update the Redis coverage anchor with an optimistic transaction that only accepts a later report end. Concurrent or out-of-order reporting workers can no longer let an older completion rewind the next default window and cause already reported periods to be processed again.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* test(ui): cover report automation listings
Assert that the asset automations page exposes the reports tab and its dedicated table alongside forecasts and schedules. This protects the report UI added by the stacked branch from disappearing during future template reconciliations.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* docs(automations): document report occurrence semantics
Describe per-automation timezone selection and the successful-coverage model used by recurring reports. The documentation now distinguishes the claimed cron occurrence from delayed runner time and explains how first runs, daylight-saving transitions, and out-of-order worker completions determine report windows.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* test(api): close rejected report transactions
Commit the fixture-owned sensor setup after rejected report automation requests so the shared API test database does not retain an idle transaction during teardown. This keeps the complete automation API module deterministic while preserving assertions that no unauthorized automation was created.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
* cli: refuse forecaster options that were given, not merely ones that differ from their default
The guard against combining forecaster options with --type schedules compared the forecaster against its default,
so naming the default forecaster explicitly passed silently and the automation was created,
leaving the impression that the option had applied to a schedule automation.
Ask which options were actually given on the command line instead, the way the --source conflict check already does,
and name the offending options in the error rather than listing every option it could have been.
Signed-off-by: Mohamed Belhsan Hmida mohamedbelhsanhmida@gmail.com
* data/models: name an automation's cursor after what it points at
Context:
- Review of #2396 asked what the "scheduling cursor" is, how an automation is "watermarked" (watermarks do not update), and why the field is needed at all.
- "scheduling" also collides with FlexMeasures' scheduling machinery (the "scheduling" queue, StorageScheduler), which this field has nothing to do with.
- The feature is unreleased, so the column, the API field and the migration can still be renamed without a compatibility burden.
Change:
- Renamed Automation.scheduling_cursor to Automation.cursor, and get_initial_scheduling_cursor to get_initial_cursor. As a column on the automation table, it reads as an automation's cursor without further qualification, like the neighbouring timezone column.
- Replaced the "watermark" wording everywhere with what the field holds: the scheduled time of the most recent run the automation committed to, advanced just before queueing, and therefore not a record of success.
- Said "run" instead of "occurrence" throughout the automation code, matching the vocabulary already used for run time, run-automations and the automation-run guard key.
- Explained in the migration why both columns are added nullable and backfilled before NOT NULL, and why the backfilled cursor is one minute before the upgrade.
- Regenerated the OpenAPI specs.
Signed-off-by: F.N. Claessen <felix@seita.nl>
* tests: follow the automation cursor rename
Context:
- Automation.scheduling_cursor became Automation.cursor, and automation "occurrences" became "runs".
Change:
- Updated the field name in the automation fixtures and assertions, and the UI assertion on the "Cursor (UTC)" heading.
- Renamed the coalescing and spring-forward test cases to speak of runs.
Signed-off-by: F.N. Claessen <felix@seita.nl>
* docs: explain the automation cursor, and say "run" instead of "occurrence"
Context:
- Review of #2396 found "the cursor is a watermark", "migration watermark", "the cursor is committed" and "durable run records" unclear, and asked why the cursor is needed at all.
- The docs already spoke of a run time, run records and run-automations, so "occurrence" was a second word for the same thing.
Change:
- Introduced the cursor by the problem it solves: the runner is a stateless once-a-minute command, so it needs a durable record of how far each automation has got.
- Stated what it holds, that it advances before queueing (so it is not a record of success), and that keeping one moving timestamp instead of a record per run is what produces the catch-up and concurrency behaviour described below it.
- Replaced the "migration watermark" sentence with what an upgrade actually does to existing automations.
- Linked issue #2393 where the docs referred to "durable run records".
- Applied the suggested wording for the "add automation" command summary.
Signed-off-by: F.N. Claessen <felix@seita.nl>
* docs/changelog: give the automation API changes their own version section
Context:
- Review of #2290 asked to move the automation API entries to a new v3.0-33 section, and noted that a revision to a CLI command introduced in the same version does not warrant its own entry.
Change:
- Moved the automation and data source entries from v3.0-32 to a new "v3.0-33 | September 1, 2026" section.
- Folded the timezone and cursor entry into the entry introducing the automation endpoints, applying the same reasoning as for the CLI changelog, and described the cursor in terms of the run it points at.
- Folded the --timezone and catch-up entry into the two CLI entries introducing the commands it revises.
- Folded the #2396 entry in the main changelog into the #2290 entry it refines, listing both PRs.
Signed-off-by: F.N. Claessen <felix@seita.nl>
* docs/changelog: restore the v3.0-32 underline to full length
Context:
- This branch had shortened the underline of "v3.0-32 | August 11, 2026" from 26 to 24 characters, one short of the 25-character title, which makes docutils warn that the title underline is too short.
Change:
- Set the underline to exactly the title length.
Signed-off-by: F.N. Claessen <felix@seita.nl>
* data/services: address review findings on the automations service
Context:
- Reviewing #2290 turned up three issues in how the automations service handles shared state and asset trees.
Change:
- run_automation now works on a copy of the data generator, like resolve_automation_sensors already did. The generator is cached on the data source, which several automations may share, so setting the job trigger on the shared instance would attribute jobs to the wrong automation as soon as anything runs concurrently.
- Moved the upward tree walk to asset_and_ancestor_ids in data/queries/generic_assets, and expressed asset_is_in_subtree in terms of it, so the two copies of that walk introduced by this branch became one.
- Added get_automations_involving_sensor, which considers every automation rather than only those on the sensor's asset and its ancestors, because a regressor may live anywhere in the tree.
Signed-off-by: F.N. Claessen <felix@seita.nl>
* data/models: index the automation asset foreign key
Context:
- PostgreSQL does not index a foreign key by itself, and automations are looked up by asset on an asset's automations page and when finding the automations that feed a sensor.
- Reviewing #2290 also showed that a bool would be read as a sensor ID, as bool is a subclass of int.
Change:
- Added an index on automation.asset_id, in a new migration rather than in the migration that creates the table, so a database that already ran that one still gets the index.
- Excluded bools from the integer branch of DataGenerator._resolve_sensors.
Signed-off-by: F.N. Claessen <felix@seita.nl>
* api/v3_0: work out an automation's sensors once when they cannot be resolved
Context:
- On the error path, the automation details endpoint called resolve_automation_sensors and then get_automation_sensors, which calls resolve_automation_sensors again and swallows the error, so a broken automation set up its data generator and loaded its parameters twice.
Change:
- Log the reason and fall back to empty sensor lists directly, which is what the second call amounted to.
Signed-off-by: F.N. Claessen <felix@seita.nl>
* cli: warn which automations a sensor deletion would break
Context:
- An automation refers to its sensors by ID inside its parameters, which no foreign key protects. Deleting such a sensor left the automation looking healthy while failing on its next run, with the reason visible only in the runner's output.
- A data source is protected from this by a foreign key, so the sensors were the remaining gap.
Change:
- flexmeasures delete sensor now lists the automations that read from or write to each sensor before asking for confirmation. The deletion is still allowed, as the host may well intend it.
Signed-off-by: F.N. Claessen <felix@seita.nl>
* tests: cover the sensor deletion warning, and stop depending on caplog
Context:
- test_invalid_cron_does_not_hide_other_due_automations failed whenever an earlier test in the session had built an app: creating one reconfigures logging and replaces the root handlers, after which pytest's caplog captures nothing. Reproduced with utils/tests/test_job_utils.py::test_app_queues_use_custom_global_and_queue_job_timeout running first.
- The condition is pre-existing and hits any test that reads caplog afterwards, including data/tests/test_utils.py::test_schema_mismatch_log_record_is_deduplicated on main, which already uses caplog.at_level. So at_level is not a workaround: the handler is gone, not merely filtered.
- The test's behavioural assertion passed throughout; only the log assertion failed.
Change:
- Assert on the logger itself rather than on caplog, which makes the test independent of what ran before it.
- Cover that deleting a sensor names the automations using it, including a regressor-only sensor, which get_automations_feeding_sensor does not find.
Signed-off-by: F.N. Claessen <felix@seita.nl>
* docs/changelog: record the sensor deletion warning
Context:
- flexmeasures delete sensor now warns which automations use a sensor.
Change:
- Added a CLI changelog entry, as delete sensor is a pre-existing command rather than one introduced in this version.
- Folded the behaviour into the automations entry in the main changelog, which already covers this feature.
Signed-off-by: F.N. Claessen <felix@seita.nl>
* docs: give automations their own page
Context:
- Review of #2290 found the automations section sitting under forecasting, although most of it will apply to scheduling and reporting automations too.
- The same review found the CLI example silent about what it automates, --forecaster and --config referred to before being introduced, the daylight-saving-time rules reading as a developer's note in the main body, and the pointer to issue #2393 reading as a todo.
Change:
- Moved the section to features/automations.rst, split into creating, running and viewing automations, and left a pointer in features/forecasting.rst. Wrote it in terms of automations in general, mentioning forecasts as today's only type.
- Made the example pass --type forecasts explicitly, which is the option the follow-up PRs use to distinguish schedules and reports.
- Introduced --forecaster and --config before the sentence that says --source makes them unnecessary.
- Moved the cursor and daylight-saving-time rules to an appendix, marked as bookkeeping you do not need in order to use automations.
- Dropped the sentence pointing at issue #2393, keeping the statement that a failed attempt is not retried, which is the part users need.
Signed-off-by: F.N. Claessen <felix@seita.nl>
* data/migrations: index the automation asset FK without adding a revision
Context:
- The separate revision added for this index branched off 9f2b6e1d4a73, but so does c63896a97a8e on the branches stacked on top of this one. Merging this branch down therefore left two alembic heads, and `flexmeasures db upgrade` fails on multiple heads. That broke the Docker build job on #2293, #2294, #2297 and #2299, while this PR itself stayed green with its single head.
- Adding the index in its own revision was meant to spare a database that had already run 9f2b6e1d4a73. Breaking the upgrade on four stacked PRs is the greater harm, so that trade-off no longer holds.
Change:
- Folded the index into 9f2b6e1d4a73, which every branch in the stack shares, and dropped the separate revision. No branch gains a head.
- Anyone whose database already ran 9f2b6e1d4a73 will not have the index; recreate the database or add the index by hand.
Signed-off-by: F.N. Claessen <felix@seita.nl>
* docs: document schedule automations in the automations chapter
The scheduling chapter now points to the automations chapter, the way the forecasting chapter
already does, and the details of what a schedule automation stores live next to the rest of the
automations documentation.
The changelog entry is merged with the one for forecasting automations, into a single entry on
automations. That entry had stayed in the v1.0.0 section although PR #2290 was merged after v1.0.0
was tagged, so the merged entry moves to v1.1.0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs
Signed-off-by: F.N. Claessen <claessen@seita.nl>
* Name automation types after the task, like the rest of the codebase
Queue names and job types call these tasks "forecasting" and "scheduling", so the automation types
now do too: 'forecasts' becomes 'forecasting' and 'schedules' becomes 'scheduling'. An automation's
type is now the name of the queue its jobs go to, so the runner no longer maps one to the other.
Automations were merged after v1.0.0 was tagged, so no released CLI or API surface changes. A
migration renames the values of existing rows, in both directions, and recreates the check
constraint that requires a data generator for forecast automations.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs
Signed-off-by: F.N. Claessen <claessen@seita.nl>
* docs/changelog: move the data source inspection entry to v1.1.0
Like the automations entry it accompanied, it was written while v1.0.0 was the open section, but PR
#2290 was merged on 2026-09-01, after the v1.0.0 tag of 2026-08-26. Neither the v1.0.0 nor the
v1.0.0rc5 tag contains it, and it is the last entry in that section that postdates the tag.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lp1bUhWjEQtyDbnvRZQgQs
Signed-off-by: F.N. Claessen <claessen@seita.nl>
* Schedulers as data generators, and a generator on every automation (#2464)
* Make the Scheduler a DataGenerator, and give every automation a generator
A scheduler's data source recorded only its class, version and author, so one source described every
schedule that scheduler ever made, whatever it computed. It now also records the flex config the
scheduler computed under, the way a reporter's and a forecaster's source records theirs, so a
schedule can be traced back to the configuration that produced it.
`Scheduler` therefore subclasses `DataGenerator`, with a config of the asset and its serialized
flex-model and flex-context. Timing stays out of it: start, end and resolution differ from run to
run, which is what `DataGenerator._clean_parameters` already says about parameters. The config is
snapshotted while still serialized, because a deserialized flex config holds sensors, quantities and
time series which do not survive a round trip. `resolve_flex_config` returns the config as passed,
and `StorageScheduler` overrides it to merge in what the asset tree stores, so a scheduler which
does not read the asset tree keeps describing exactly what it was given.
One scheduling request stays one data source: `create_sequential_scheduling_job` resolves the
request's source once and hands it to each device job, so a schedule can still be retrieved per
device from the request's job, rather than each device job resolving a source from its own slice of
the flex-model.
A schedule automation now points at such a source, so `generator_id` is required for every
automation and the constraint requiring it only for forecasts is gone. That generator is derived
rather than chosen: the scheduler follows from the asset and the config from the asset tree, so the
runner resolves it again on every run and moves the automation when either has changed. For the same
reason, a schedule automation's flex config may only describe the site and its devices: a field
fixing a moment, such as `soc-at-start` or a `soc-targets` entry with a `datetime`, is refused when
the autom…
The batched job stats and the list's Redis error follow main's kebab-case field names (#2545).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC
Signed-off-by: F.N. Claessen <claessen@seita.nl>
The listing now counts the jobs of every asset it spans in one pass, next to naming the asset each automation belongs to.
The page's own scripts are main's, which load an automation's Info panel as the table is built.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC
Signed-off-by: F.N. Claessen <claessen@seita.nl>
…d not be reached
The listing counts the jobs of every automation it holds, so a row reads them from the listing rather than from a request of its own,
and the listing is where the page says that Redis was unreachable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC
Signed-off-by: F.N. Claessen <claessen@seita.nl>
The automation detail response keeps this branch's Redis fallback for the job counts, next to the run statistics main added.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC
Signed-off-by: F.N. Claessen <claessen@seita.nl>
The job-cache lookup is main's, which asks the type's handler where its jobs are cached, and this branch's batched counting uses it.
The API changelog entry moves to a section of its own, where it sat under a released one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC
Signed-off-by: F.N. Claessen <claessen@seita.nl>
The reason will be displayed to describe this comment to others. Learn more.
Copilot review overview
🟡 Changes recommended
There is a real runtime bug in _job_cache_refs’s unknown-type fallback (returns {} instead of a set) and the UI can leave stale Redis error banners visible after recovery.
This PR improves the Automations listing flow by returning per-automation job-stats with the automations list response (instead of fetching per automation), and by surfacing a single redis-connection-err at the listing level when Redis job-cache access fails.
Changes:
Add batched job-stat counting utilities in flexmeasures/data/services/automations.py and expose results via GET /api/v3_0/assets/<id>/automations.
Update the Automations UI to render the Jobs column from the list response and show Redis connectivity issues at page level.
Extend tests and documentation/OpenAPI examples for the new response fields and Redis-timeout behavior.
…s away
The listing's job counts belong to the page rather than to the official API's contract, so the API changelog entry goes, and the page's own entry names this pull request too.
A type this server does not know contributed an empty dict where its caller unions sets.
The banner about an unreachable Redis stayed up once shown, although the page refreshes itself, so it now says what the last listing said.
Reading the counts for a row and for the panel is one helper again.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC
Signed-off-by: F.N. Claessen <claessen@seita.nl>
The reason will be displayed to describe this comment to others. Learn more.
Copilot review overview
🔵 Needs a closer look
The UI still appears to eagerly request per-automation details (which can recompute job-stats per automation), undermining the PR’s stated goal of eliminating per-row roundtrips and per-automation cache scans.
Avoid scanning overlapping asset cache refs multiple times
flexmeasures/api/v3_0/assets.py:1523
The job-stats batching still loops over assets and calls get_asset_automations_job_stats separately per asset. When include-child-assets=true, sub-asset cache refs (especially scheduling sensor refs) can overlap with the parent subtree, so Redis cache entries may be scanned multiple times in one request. To match the intended "one pass" behavior, consider collecting automations across all assets, unioning their cache refs (and precomputing scheduling sensor IDs per distinct subtree root), then counting via a single cache scan.
Remove per-automation detail fetches that recompute job stats
The new row comment says the Jobs column is filled “without a request of its own”, but the page still calls loadAutomationInfo for every automation when building/updating the tables (see automations.forEach(automation => loadAutomationInfo(automation.id));). Those per-row detail requests still hit [GET] /assets/<id>/automations/<automation_id>, which computes job-stats by scanning Redis caches per automation, so the original N-requests/N-cache-walk issue can still persist unless the eager prefetch is removed or the details endpoint stops recomputing job stats.
Sort job status counts to keep Jobs column order stable
jobStatsText formats job-status counts by iterating over object properties, but the insertion/order of keys in job-stats can vary between refreshes (Python dict insertion depends on iteration over set cache refs). This can cause the Jobs column text to flicker between different status orders. Sorting the entries before joining makes the UI stable.
The page asked the server about every automation it listed, each time it refreshed, which is what counting the jobs in the listing was meant to end.
A row now reads its counts from the listing, and opening an automation's Info panel is what asks about that one.
Running an automation refreshes the listing rather than that automation alone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Copilot's second round has no inline findings and one observation, which was right and went to this PR's point: the page still asked the server about every automation it listed, on every refresh, each of those requests counting that automation's jobs again. That is what counting them in the listing was meant to end — I had dropped the on-demand loading of the panel when I merged #2554, which introduced the eager loading.
Fixed in b75421e: a row reads its counts from the listing, opening an automation's Info panel is what asks about that one, and running an automation refreshes the listing rather than that automation alone.
The reason will be displayed to describe this comment to others. Learn more.
Copilot review overview
🟢 Approval recommended
The functional changes are consistent with the PR goals, include targeted test coverage, and the remaining feedback is limited to a minor type-annotation consistency improvement.
Review effort: Lite Findings: None
Previously missed (1)
In code that hasn't changed since last review
Allow descendants_cte to accept None for unlimited depth
flexmeasures/data/services/automations.py:1581
descendants_cte is called with max_depth=None, but its signature is descendants_cte(root_asset_id: int, max_depth: int). Even though the implementation supports a non-int to mean “no depth limit”, the type annotation and docstring don’t, which can confuse readers and static analysis. Consider updating descendants_cte to accept max_depth: int | None (and document None as unlimited depth), or provide a dedicated helper for an unbounded subtree query.
It has always meant every level below the root, and now says so.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Main asks the server about one automation only when its panel is opened (#2299), where this branch's copy button expected every row to have been read on page load, and would have said its details were still loading.
A copy now reads that automation itself when the panel has not been opened.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC
Signed-off-by: F.N. Claessen <claessen@seita.nl>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
An asset's Automations page filled its Jobs column by asking after each automation separately: one
request per row, on every page load, each one walking that automation's job caches. With the
O(automations × sensors × jobs) Redis fanout flagged in the #2290 review underneath it, that is a page
that gets slower with every automation an asset gains. The counts now arrive with the listing itself.
One pass over the caches.
[GET] /assets/(id)/automationsincludes ajob-statsper automation.The cache references of all the asset's automations are collected and deduplicated first, and the jobs
found are then grouped by the automation id recorded on their trigger. An asset whose automations
share sensors — the normal case, since they sit on the same asset — reads each cache once rather than
once per automation.
The page asks once. The Jobs column is filled from the list response, so the page no longer
fires a request per row to populate a column that is visible immediately, and an automation's details
load when they are opened, once, rather than eagerly for every row.
Redis trouble is reported once. The listing carries
redis_connection_err, as the jobs endpointalready does, and reports it against the page rather than against a row. If Redis is unavailable the
listing still renders, with empty counts and one message, instead of every row failing separately. A
RedisErroris logged and left as empty counts, since a worker's cache being briefly unreachable isnot something to put in front of the reader; only a missing Redis configuration, which is a host
misconfiguration, is reported to the caller.
documentation/changelog.rstHow to test
The page looks the same — this changes how it loads, not what it shows — so the thing to observe is
the network panel, not the screen. Open
/assets/<id>/automationson an asset with severalautomations: the Jobs column fills from the single listing request, with no details request per row,
and opening Details fetches that automation once.
The added test drives the listing with a Redis that times out, and asserts the response still arrives,
with empty counts and an error message, rather than failing. It was verified to fail without the
handler it covers. The existing job-statistics tests were moved onto the batched helper, so they now
cover the path the page actually takes.
Further improvements
jobs have expired shows none. Durable run records are tracked in Add durable automation-run records and safe retry semantics #2393.
Related items
Part of the automations story #2334, and of #2288. Stacked on #2297. The UI edit action this PR
originally also carried has since arrived by way of #2294.
Sign-off