Time every automation run the same way: offsets and a duration, and no fixed moments - #2551
Conversation
Automations are recurring tasks (for now: computing forecasts) defined per asset. The recurrence is defined by a cron string, and the work to be done is defined by a data generator (linked through a data source) together with the parameters to call it with. Includes a migration for the new table, and new dependencies on croniter (cron matching/validation) and cron-descriptor (natural-language recurrence descriptions). Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
- `flexmeasures add automation` creates an automation (active by default), validating the forecast parameters with the forecast parameter schema and storing the forecaster config on a data source. - `flexmeasures edit automation` edits the name, recurrence (cron string) or activation status. - `flexmeasures delete automation` deletes an automation. - All three record their events in the asset's audit log. - `flexmeasures jobs run-automations` queues jobs for all automations due this minute (to be run once per minute, e.g. via cron), with a Redis-based guard against duplicate runs within the same minute. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Data generators can now be told how their queued jobs got triggered (via the CLI, the API or an automation), and the train-predict pipeline stores this on the jobs as meta data. The asset's status page shows it in a new 'Created Via' column of the jobs table. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
GET /api/v3_0/assets/<id>/automations lists the automations defined on an asset (without generator and parameters details). GET /api/v3_0/assets/<id>/automations/<automation_id> additionally provides the parameters, data generator info and counts of recently created jobs per job status. Both are documented in the OpenAPI specs. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
/assets/<id>/automations shows the asset's automations in a tabbed view (schedules and reports tabs are prepared but deactivated), with per-row details (parameters, data generator, job counts) loaded asynchronously into a modal. The page is linked in the breadcrumbs dropdown and links to the status page, where recent jobs are listed. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
CI runners have no locale set (POSIX), which made cron-descriptor render 'At 06:00' while dev environments with an en_US-style locale rendered 'At 06:00 AM'. Request 24-hour format explicitly so the description is deterministic across environments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pxkeq64jtENY7fiWjwUsVS
- Escape automation names (and other user-controlled strings) in the Automations page and the status page's jobs table, closing two stored HTML/script injection sinks. - Wipe parameter state on the (possibly shared) cached data generator before each automation run, so automations sharing a generator data source don't pollute each other's runs. - Count automation job stats under the forecast target sensor(s) from the automation's parameters, which may belong to a different asset. - Release the per-minute Redis guard when a run fails, so a retry within the same minute can still queue jobs. - Return 404 (as documented) for nonexistent automation ids on the detail endpoint, and check permissions on the asset, so automation ids can no longer be enumerated across accounts via 403-vs-422 differences. - Use ondelete=SET NULL for the generator FK: deleting a data source no longer silently deletes automations. - Delegate Automation ACL to the asset's ACL instead of duplicating it. - Extract the config/parameters assembly shared by `add forecasts` and `add automation` into a helper (which no longer drops falsy config values). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Completes the previous commit, whose staged files were dropped by an interrupted pre-commit run: template escaping, shared-generator state reset, job stats under target sensors, Redis guard release on failure, 404 for nonexistent automations, SET NULL generator FK, ACL delegation, and the shared CLI config/parameters assembly helper. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
The scheduling job creators accept an optional trigger dict (stored as job meta data), like the forecasting pipeline already does. The API trigger endpoint records origin API; the CLI and automations follow in the next commit. The status page's 'Created Via' column picks this up automatically. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Automations now also support the 'schedules' type: - `flexmeasures add automation --type schedules` validates the parameters as a schedule trigger message (per the AssetTriggerSchema, as accepted by the API trigger endpoint, without the asset id). The schedule 'start' may be omitted, in which case each run schedules from the run time (floored to the message's resolution, if given) — a fixed start draws a warning. - The runner dispatches schedules automations to the same job creators as the API trigger endpoint (sequential or simultaneous), recording trigger meta data (origin automation) on the queued jobs; `flexmeasures add schedule --as-job` now records origin CLI. - Job stats for schedules automations are counted from the scheduling job cache (asset-level wrap-up jobs and per-sensor device jobs). - The UI automations page's Schedules tab is now enabled, with automations filtered by type per tab. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
- New endpoints on assets: POST /automations (create, validating parameters by automation type), PATCH /automations/<id> (name, cron string, activation status) and DELETE /automations/<id>. Managing automations requires the same principals that may delete the asset (account admins and consultants). - The UI automations page gets a 'New automation' modal and per-row (de)activate and delete actions, shown to users with management rights. - Creation, update and deletion logic (incl. audit log records) moved into the automations service, shared by the CLI commands and the API endpoints. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
- Activate the reporting queue (it was prepared but commented out), including worker help texts and queue cleanup. - Reporters accept as_job: a job is queued (with trigger meta data) that rebuilds the reporter from its data source, computes the report and saves the results to the database. - `flexmeasures add report --as-job` queues such a job; reporting jobs show up in the asset's jobs overview (status page and API). Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Automations can now compute reports on a recurring basis: - `flexmeasures add automation --type reports --reporter <class>` stores the reporter config on a data source (steady across runs, so all report results attribute to the same source) and validates the report parameters. - The report window resolves freshly on each run: 'start-offset'/'end-offset' fields (comma-separated Pandas offsets, applied to the run time in the first output sensor's timezone) express a rolling window, and without any timing fields the window defaults to the last cron period (from the previous cron fire time until the run time). Absolute start/end still work, but draw a warning. - The API creation field 'forecaster' is generalized to 'generator' (also accepting reporter classes), and the UI's New automation modal gains data generator and config fields; the Reports tab is now enabled. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Each automation run is recorded in Redis; a report automation without timing fields then reports on the period since its actual last run, falling back to the last cron period when no last run is known (e.g. on the first run, or after a Redis flush). This gives gapless coverage even when runs are missed. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Consolidates the shared automations concept (model, lifecycle, runner deployment, provenance) into documentation/features/automations.rst, with the per-feature pages linking to it and keeping only their type-specific parameter semantics. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
- API automation creation now checks that the caller may read every sensor referenced in the parameters/config and record data on the sensors the automation writes to, closing a cross-account data read/write hole. - `add report --as-job` implies --save-config (the worker rebuilds the reporter from its data source, so jobs without stored config always crashed). - Report automation parameters are validated with the chosen reporter's own parameters schema, not the base schema. - Invalid start-offset/end-offset strings are rejected at creation instead of being silently skipped at run time (which yielded empty report windows). - run_report_job wipes the shared cached reporter's parameter state, like the automation runner already did, so consecutive jobs in one worker process don't pollute each other. - Default report windows now anchor to the end of the last *successfully* covered window, recorded by the reporting job upon success — failed jobs no longer create permanent reporting gaps, and the enqueue-time minute-rollover gap is gone (the recorded anchor is the window end itself). - The cron-period fallback window is computed in the platform timezone, matching how the runner decides when automations fire. - Job stats for schedules automations also scan flex-model device sensors (which may belong to child assets), so failed per-device jobs show up. - The trigger provenance kwarg is excluded from the job cache hash, so identical schedule requests from different origins dedupe again. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
…essage format PR #2303 makes click report the validation message rather than the offending value, which changes the exact wording of this error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Merge current main, resolve the shared forecasting and documentation changes, regenerate the lockfile, and move the automation migration after the current migration head. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Reject cron expressions with seconds, year fields, or aliases because the automation runner executes once per minute. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Test valid five-field expressions and reject unsupported seconds, year, and alias formats. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Keep the per-minute Redis guard after failures because an attempt may already have queued some forecast jobs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Verify that retrying a failed partial queueing attempt does not create duplicate jobs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Convert YAML dates to ISO strings, accept empty files, and report non-object config or parameter files as usage errors. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Test YAML dates and timestamps, empty files, and invalid top-level list values for automation options. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Creating a forecast or report automation from a data source that stores another kind of data generator said "Could not set up reporter 'None'", as no class was named and the source was not mentioned. The error now names the data source. Without a source, a forecast automation's error names the class it actually tried, including the default. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
It changes core timing semantics across all automation types (including DST-sensitive window arithmetic), so it needs careful final human validation despite the strong test coverage.
Review details
- Files reviewed: 11/11 changed files
- Comments generated: 1
- Review effort level: Lite
The docstring and the docs said two of start-offset, end-offset and duration describe a window, and left out that a forecast or schedule automation also takes a duration alone, starting at the run time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC Signed-off-by: F.N. Claessen <claessen@seita.nl>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC Signed-off-by: F.N. Claessen <claessen@seita.nl>
…chains flexmeasures add automation gains --start-offset and --end-offset, which set the same parameters as a parameters file does, next to --duration, which it already accepted. The day-ahead schedule example no longer needs a file. The docs now say what an offset chain is: comma-separated steps applied from left to right, each a Pandas offset alias, linked to the Pandas docs, or DB or HB. A table lists the steps most useful for automations, with results checked against Pandas 2.3, including that W moves to the next Sunday rather than a week on, that boundary steps such as MS keep the time of day, and that H, M and T are deprecated. The command's help text no longer says only report automations refuse a fixed moment. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC Signed-off-by: F.N. Claessen <claessen@seita.nl>
…kes it --duration came in hidden, with the forecast schema's options, and its help spoke of forecasts only. It now describes the period each run covers for any automation type, next to --start-offset and --end-offset, so this command declares it itself, and add_cli_options_from_schema learns to exclude a field for that. The help test, which listed --duration among the forecast-only options kept out of the help, expects it and the offset options instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC Signed-off-by: F.N. Claessen <claessen@seita.nl>
…KeyError Creating an automation with an "end-offset" alone is refused, so `resolve_automation_window` only meets one in parameters stored before that check existed. It then indexed "duration" directly, so the dispatcher reported the run as `failed to queue jobs: 'duration'`, which names neither the automation's problem nor its fix. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
`resolve_schedule_generator` was the one caller left on the old signature, so it resolved any offsets on the platform clock, and against the current time rather than the run's. A day-ahead automation on an Amsterdam clock therefore described its data source by a window eight hours, and a whole run, away from the schedule that source records. Nothing breaks today, because a scheduler's data source records the flex config alone, and moment-fixing values are deliberately kept out of it (see #2444). The two calls now simply agree, so they cannot drift apart if the window ever reaches the source. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Offsets that fix both ends of a report's window are checked against each other, but a start carried over from the last successful report is not, because the automation does not fix it. An "end-offset" alone combines the two: it takes that carried-over start and an end that a backward offset can place before it, and the run then reported on an inverted window, which a reporter reads as neither the empty window it means nor an error. Such a run now reports on nothing and logs why, naming both moments and what to change. The alternative, refusing the run, would leave the automation stuck, since the start it cannot resolve is the one it did not choose. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
A duration with months or years is added as a `pd.DateOffset` built field by field, and that build took the days and the seconds of the remainder but not its microseconds, so "P1MT0.5S" silently landed half a second early. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
…e its parameters The CLI now explains the offsets and the refusal in its help, but the two surfaces a user is just as likely to meet did not: the create endpoint's description said nothing about either, so an API user met the rule only as a 422, and the UI's parameters field, a free-form JSON box, is the only place its form explains what belongs in it. Both now say it, and the endpoint gains a day-ahead example beside its daily-forecast one. The description also names `reporting`, a third automation type since #2297, which it had missed. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
…named Three corrections to what this branch already says. The comment on the Timestamp arithmetic hung its "so the day the clocks go forward lasts 23 hours" off the datetime clause, where it describes what a Timestamp does. The docs said a duration counts real time, which holds for "P1D" but not for "P1M", built as a `pd.DateOffset` and so calendar-aware: from midnight it ends at midnight. And the changelog read as though every belief time in report parameters was renamed, where the one inside an `input` entry, which selects beliefs rather than stamping the report, keeps its name. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
BelhsanHmida
left a comment
There was a problem hiding this comment.
Reviewed this one and tested it locally. The design holds up, so I pushed six commits.
Two of them I'd like you to look at properly:
A report could still resolve an inverted window, and fixing it needed a decision. Offsets that fix both ends are checked against each other, but a start carried over from the last successful report is not, since the automation does not fix it. An end-offset alone combines the two: it takes that carried-over start and an end a backward offset can place before it. The run then reported on an inverted window, which a reporter reads as neither the empty window it means nor an error.
I made such a run report on nothing and log why, naming both moments. The alternative — refusing the run — leaves the automation stuck, because the start it cannot resolve is the one it did not choose. If you would rather refuse, it is a three-line change.
resolve_schedule_generator was resolving a wider gap than I first thought. It was the one caller left on the old signature, so it passed neither the automation's timezone nor the run's scheduled_at. I wrote the test expecting an eight-hour drift between the platform clock and an Amsterdam one; the red run showed the generator was actually resolved for today's date, six months from the schedule it describes. Nothing breaks today, since a scheduler's data source records the flex config alone and #2444 keeps moment-fixing values out of it. The two calls now simply agree.
The rest:
resolve_automation_windowindexed"duration"directly on a path that creation refuses, so a stored automation from before that check made the dispatcher reportfailed to queue jobs: 'duration'. It now names the missing field.- A calendar duration is built as a
pd.DateOffsetfield by field, and the build took days and seconds but not microseconds, soP1MT0.5Slanded half a second early. - The CLI now explains the offsets and the refusal well, but the create endpoint's description said nothing about either, so an API user met the rule only as a 422 — and the UI's parameters field, a free-form JSON box, is the only place its form explains what belongs in it. Both now say it, and the endpoint gains a day-ahead example. The description also names
reporting, which it had missed since #2297. - Three corrections to wording already on the branch: the comment on the Timestamp arithmetic hung its "so the day the clocks go forward lasts 23 hours" off the datetime clause; the docs said a duration counts real time, which holds for
P1Dbut not for the calendar-awareP1M; and the changelog read as though every belief time in report parameters was renamed, where the one inside aninputentry keeps its name.
…on period along with its end-offset A report automation with only an end-offset fell back, on its first run, to a window starting at the previous cron fire time, which lies after the end whenever the offset reaches back further than one cron period (e.g. "-1D,DB" on a daily cron). The window then became empty, and so did every window of an hourly report up to midnight after its first run of the day. Queueing a job for an empty window is not safe: a PandasReporter whose transformations refer to its inputs raises a KeyError on one, and a failed job records no coverage, so the automation never got past it. Now, before the first successful report, the fallback cron period ends where the end-offset says, taken on the wall clock. A window its last successful report already covers queues no job, and running such an automation on demand says why. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC Signed-off-by: F.N. Claessen <claessen@seita.nl>
There was a problem hiding this comment.
🟡 Changes recommended
A newly added wrapped OpenAPI example description breaks mid-phrase and should be reflowed to match the repo’s line-break-after-punctuation documentation convention.
Get a fresh assessment by requesting another Copilot review.
Review details
- Files reviewed: 15/15 changed files
- Comments generated: 1
- Review effort level: Lite
…report does without timing Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC Signed-off-by: F.N. Claessen <claessen@seita.nl>
There was a problem hiding this comment.
🔵 Needs a closer look
It changes core automation timing semantics across all automation types (including DST-sensitive window arithmetic) and merits final human review despite no specific defects found in the reviewed diffs.
Review details
- Files reviewed: 15/15 changed files
- Comments generated: 0 new
- Review effort level: Lite
…kebab-case run stats in the example A fixed 'start' is refused since #2551, and a duration alone starts the forecast at the time of each run, which is what the fixture froze the clock to. The run-stats example still spelled its fields in snake_case, where the endpoint returns them in kebab-case. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC Signed-off-by: F.N. Claessen <claessen@seita.nl>
* feat(data/models): add durable automation run records An automation occurrence currently leaves no trace of its own: the only durable marker is the automation's scheduling cursor, which says an occurrence was attempted, not what happened to it. Record each occurrence a runner picks up as an AutomationRun, with the attempts made on it (AutomationRunAttempt) and the jobs it intends to create (AutomationRunJob). Dispatch progress and worker execution outcome are tracked separately, because 'queued everything' and 'the jobs succeeded' are different questions an operator needs answered. The database enforces one run per automation, occurrence and schedule revision, and one job intent per run and logical job key. The new schedule revision on Automation keeps runs of an edited or reactivated schedule apart from the runs of the schedule it replaced, even at the same scheduled UTC time. All run timestamps are validated to be timezone-aware and stored as UTC. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * feat(data/services): claim automation occurrences durably and resume partial dispatch The runner used to advance and commit the scheduling cursor before queueing anything, guarded only by a Redis key with a two-minute TTL. A failure before the first enqueue therefore lost the occurrence for good, while retrying a partial enqueue could have duplicated work. Claim each due occurrence into an AutomationRun instead, write down the plan for the run before the first enqueue, and give every intended job a logical key and, from it, a deterministic RQ job ID. A retry replays that stored plan: jobs whose IDs are already in Redis are recognised and left alone, and only the missing ones are queued. Because the plan holds the parameters and timings the occurrence was planned with, a retry hours later still dispatches the occurrence as originally intended, even if the automation has been edited since. Ownership is a database lease, not a Redis key. An occurrence is only picked up by another runner once the lease of the runner holding it has expired, which is how a runner that died mid-queueing hands its work over. A runner that fails releases its own lease, so its run is retryable at once. Dispatch is finished only when it is marked complete, so a crash between the last enqueue and that mark is finalized by the next runner rather than left hanging. Forecast cycle and wrap-up jobs now carry their run identity and report their own start, success and failure back to it, so the execution outcome outlives the Redis jobs. Queueing the jobs of a pipeline run moved out of the already long run() into its own methods, which also removes the duplicate queueing path. Editing an automation's cron string or timezone, or reactivating it, counts up its schedule revision, so runs of the old and new schedule stay distinct. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * feat(api/ui): show durable automation run status Job counts came from Redis alone, so once its jobs expired there was no way to tell whether an occurrence had failed before queueing anything, queued only part of its work, or queued everything and then failed while computing. Add a run_stats object to the automation detail response, summarizing the automation's durable runs and describing the recent ones: their occurrence, dispatch and execution state, attempt count, intended and queued job counts, timestamps, last error, latest attempt, and the jobs they created. Automation responses also expose schedule_revision. The automation details panel shows the latest run alongside the recent Redis jobs. Both additions are additive: no existing field changed. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs: describe durable automation runs and safe retries The forecasting docs and the CLI change log stated that a failed or partially completed queueing attempt is never retried, which no longer holds. Describe how an occurrence is claimed, planned, dispatched and retried instead, and what the dispatch and execution states mean, along with the changelog entries for the new run status in the API and the UI. Those two stale CLI change log lines are corrected rather than left standing: they describe an unreleased release, so leaving them would ship a change log that contradicts itself. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * refactor(data/models): keep the dispatched job statuses in one place The tuple of statuses that count a job intent as dispatched was written out in the model and, unused, in the service. Define it once next to the model that asks the question, and drop the copy nobody read. Also apply black to the migration and drop an unused import from the new test module. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix(data/services): keep recording a job outcome when the job broke the transaction Running an automation against a real database and a real Redis turned up two ways the durable execution record lied about what happened. A cycle job that failed on the database itself left the session in an aborted transaction, so recording its failure was refused and the job stayed 'running' forever. Roll back before recording, the way dispatch failures already do; the job's uncommitted work is lost either way, since it is failing. A later job succeeding also reset the run to 'running', burying an earlier failure: the wrap-up job succeeds whatever became of the cycle jobs it reports on. Derive the run's execution state from all of its jobs instead, so a failed job keeps the run failed. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs/changelog: link the durable automation runs PR The entry linked the issue, because no PR existed when it was written. Point it at PR #2457 instead, which is what the changelog convention asks for. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix(api/tests): stop the sensor-data error listener from outliving its test test_post_sensor_data_twice registers a 'handle_error' listener on the Engine class, which is global and process-wide, and never removes it. Every database error raised by any later test therefore ran its assertion that the error is an IntegrityError, so a test which provokes a different error fails inside SQLAlchemy rather than where it looks. Remove the listener in a finally block, so it only covers the posts it is about. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix(migrations): chain the durable automation runs migration after main's Merging main left the migration graph forked: main's 'drop obsolete tables' revision and this branch's own both named 84f268f5153c as their parent, so alembic saw two heads and 'flexmeasures db upgrade' refused to run, which is what failed the Docker image build. Migration files sit in separate files and never conflict textually, so the merge looked clean while the graph did not. Point this branch's revision at main's instead, making the chain linear again. The two migrations do not interact: main's drops nine obsolete pre-GenericAsset tables, this one only adds the automation run tables and a column on 'automation'. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * Update execution state condition for running intent Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Mohamed Belhsan Hmida <149331360+BelhsanHmida@users.noreply.github.com> * Reformat claim availability docstring Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Mohamed Belhsan Hmida <149331360+BelhsanHmida@users.noreply.github.com> * Fix formatting of docstring in test function Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Mohamed Belhsan Hmida <149331360+BelhsanHmida@users.noreply.github.com> * perf(data/services): summarize automation runs without loading their whole history An automation keeps one run record per scheduled run, so its history grows without bound, while the status summary only ever shows counts and the ten most recent runs. It nonetheless loaded every run to produce that, and each run drags its 'parameters' and 'plan' JSONB along, so an automation running by the minute would have the panel read a year of job payloads to render. Count per dispatch and execution state in the database instead, read only the recent runs the summary describes, and eager-load their attempts and jobs, which were costing a query each on top. The regression test asserts what the shape of the queries must be, rather than how long they take: no query may read run rows without a limit on how many. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs: say what a retry actually reuses The runs section claimed a retry reuses the parameters and timings the run was planned with. The parameters part holds, since the run stores its own copy. The timings part does not: only timings that were explicitly given are in those parameters, so one an automation left to the run time is resolved afresh on every attempt, and a resumed run's remaining jobs can cover a later window than the ones its first attempt queued. State what is actually guaranteed. Pinning the resolved start onto the run, so that the original sentence would hold, is a behaviour change and belongs in its own pull request. Reported by Felix in review of PR 2457. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * Name the durable run fields in kebab-case too `run-stats` had kebab on the outside and snake within: `latest_run`, `dispatch_state`, `attempt_count` and twenty more, all of them API output this branch adds. Automations have shipped in no release, so these are plain renames. The asset's Automations page and the tests read them by name, and follow. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WS6V8nZyRzMxsvqGnNpUTk Signed-off-by: F.N. Claessen <claessen@seita.nl> * Follow up on the merge: no fixed start in a durable-run fixture, and kebab-case run stats in the example A fixed 'start' is refused since #2551, and a duration alone starts the forecast at the time of each run, which is what the fixture froze the clock to. The run-stats example still spelled its fields in snake_case, where the endpoint returns them in kebab-case. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC Signed-off-by: F.N. Claessen <claessen@seita.nl> * Give up on a dispatch that keeps failing, instead of retrying it every minute A run whose dispatch fails is retryable again straight away, which is right for a first failure, as it usually passes. A dispatch that fails on every attempt, though, fails for a reason no retry fixes, such as a sensor that was deleted, and was retried every minute for as long as its automation stayed active, recording an attempt and an error each time. Each further attempt now waits twice as long as the last, up to an hour, and a run stops being dispatched after five of them. The run keeps its record of what went wrong on each attempt. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC Signed-off-by: F.N. Claessen <claessen@seita.nl> * Let a retry of an automation run pick up where the run's own record says it got to Three ways in which a retry could do the work of the attempt before it: Reconciling an intent asked Redis alone, where a job's hash is dropped once it has run, so a retry after that created the job again, under the same ID, and computed the same forecast twice. The run records which of its jobs were queued, which is what a retry now reads; Redis is asked only about a job that was never queued. A retry planned its jobs from the automation as it is now, while the run's intents describe the jobs it was planned with. Configuration that decides how many jobs a run has, such as a forecaster's retrain frequency, could therefore be edited between two attempts, and the retry raised a KeyError on a job the run never had. A retry now forecasts with the data generator the run recorded, and says what changed if its plan no longer matches the run's. A runner claims every run of its batch at once and dispatches them one after the other, so the claim on the last of them could expire before its turn came, and another runner could take it over. A runner now renews its claim just before dispatching, and leaves the run to whoever holds it if it cannot. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC Signed-off-by: F.N. Claessen <claessen@seita.nl> * Say what a run's record means, where it said more than it knew A run's execution ended as soon as one of its jobs failed, also while another was still computing, so the moment it recorded was not the one its name and the documentation promise. It now ends once nothing of the run is running any more. Execution state, and the jobs a run created, are recorded for forecast automations; a schedule or report run records its dispatch, while its execution state stays pending. The API changelog and the documentation now say so, rather than describing execution state as something every run reports. That changelog also named the two new fields in snake_case, where the API returns 'schedule-revision' and 'run-stats'. Removes claim_due_automation(), which lost its last caller to the durable runs: it is the cursor swap without the schedule revision, which is not what a new caller would want. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC Signed-off-by: F.N. Claessen <claessen@seita.nl> * Refuse a retry that plans fewer jobs, and record a job's failure once A retry planning a subset of the run's jobs was accepted, which would leave the jobs it left out pending for good while the run reported itself as fully queued. A failed job reports its failure itself, and the queue's exception handler reports it again for a job that could not. The first of them is now the one that counts, so that the moment recorded is the moment the job failed. Also drops a blank line that the merge with main left between two changelog entries. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC Signed-off-by: F.N. Claessen <claessen@seita.nl> * Break the durable runs' docstrings and comments after punctuation Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC Signed-off-by: F.N. Claessen <claessen@seita.nl> --------- Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> Signed-off-by: Mohamed Belhsan Hmida <149331360+BelhsanHmida@users.noreply.github.com> Signed-off-by: F.N. Claessen <claessen@seita.nl> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: F.N. Claessen <claessen@seita.nl> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Description
Closes #2550.
An automation runs again and again on the server clock, so a moment fixed in its parameters goes stale:
a fixed
startorendhas every run compute the same period, and a fixedpriorhas every run ignore the data recorded since then.Until now, each automation type handled its timing differently. Forecast automations accepted all three. Schedule automations warned about a fixed
start. Report automations (#2297) refused a fixedstartorend, and were the only type that could describe a window relative to the run.This makes the three types agree.
Fixed moments are refused for every type. A
start,endorpriorin an automation's parameters is refused when the automation is created, with one message and one exception (RecurringAutomationFixesAMoment, renamed fromRecurringScheduleFixesAMoment), so the API answers 422 and the CLI a usage error.A forecast automation can still fix
train-start, which belongs to the forecaster's config.Simulations are not affected: play mode leaves the server clock alone, and a simulation stepping through simulated time triggers each step through the API or CLI with an explicit
startandprior.Every type describes its window the same way. Any automation's parameters take two of
start-offset,end-offsetandduration, with offsets as the offset chainsflexmeasures add report --start-offsetalready uses (Pandas offsets, plusDBandHB).The offsets apply to the cron occurrence the run was claimed for, on the automation's own clock, so a run delayed past midnight still covers the day it was due for.
They resolve into what each data generator takes: a start and a duration for a schedule, a start and an end or duration for a forecast, a start and an end for a report.
That makes "every day at noon, schedule tomorrow" expressible, which it was not for schedules and forecasts:
A
start-offsetalone gives a forecast or schedule its default duration, and has a report end at the run time. Anend-offsetalone is for reports only: they continue from the last successful report, and before the first one, cover the cron period ending at that offset. A report with nothing new to cover (e.g. an hourly report up to midnight, after its first run of the day) queues no job, since not every reporter can compute an empty window, and a failed job would never record coverage.All three fields at once, and combinations a run could not resolve, are refused on creation.
Defaults without offsets are unchanged. Forecasts and schedules still start at the run time, rounded down, rather than at the claimed occurrence, so a caught-up run stays current as the docs promise. Reports still continue from the last successful report.
Two things worth a look:
startis believed at that start unless it gets aprior, so a resolved window also carries the run time asprior. Otherwise a day-ahead forecast would be stamped as believed tomorrow.pd.Timestamp: twodatetimes sharing aZoneInfosubtract in wall-clock time, which made the day the clocks go forward last 24 hours instead of 23.Report parameters name their belief time
prior, as schedule and forecast parameters do.belief_timeshipped in v1.0 and is still accepted.The report schema carries its own alias rather than
SupportsLegacyFieldAliases, becausepriorhas to win where both names appear:DataGenerator.computemerges a new run's parameters into those it loaded for its previous run, so a loadedbelief_timecan meet a newprior. The aggregator tests that compute twice on one reporter caught this.The first two commits are the schedule refusals first made in #2297, which that PR has since reverted, so they could be done together with forecasts here.
documentation/changelog.rstHow to test
New tests cover each window combination for each type, including the day the clocks go forward. They also cover a delayed run that must still cover the day it was due for, a forecast's belief time, the refused combinations, the refused fixed moments per type,
train-startstaying allowed, thepriordata key with its legacy alias and precedence, and day-ahead schedule and forecast runs end to end.Each new check was verified to fail without the code it covers.
Related items
Tracked in #2550, under the Automations v1 story (#2334). Stacked on #2297, from which the schedule refusals moved here.
Anchoring runs on the claimed occurrence is also the direction for #2537 (a retried run re-resolving its timings), which can build on this once durable runs (#2457) land.
Sign-off
🤖 Generated with Claude Code
https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC