You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Record runs on the rec and prov-extension vocabularies of secorolab/metamodels#73, on main's design.
What stays
The observer contract: one observer holds any number of runs and every call names its run. run_id is on every method, as before.
Run, Scenario, Feature, hooks, ingredients, the example.
MariaDBObserver with its table and per-category JSON columns. It gains run_id, state and verdict columns, added to a table from the previous release on first connect (status stays as it was); the auto-increment id is the short number to display and sort by.
What changes
BaseObserver implements the lifecycle and the log_*/add_* methods on the two primitives main already had, get_run and update_run_data, so a backend only stores columns.
rec/jsonld.py maps a run's columns to one JSON-LD document on the prov, prov-extension and rec contexts, and back. observer.document(run_id) returns it for any backend.
the run is a prov-ext:Execution with an OSLC Automation state and verdict: rec.State and rec.Verdict replace RunStatus, and a run stores both. A verdict other than unavailable needs a complete run, as the shapes say; DEAD is a reader's inference from a stale heartbeat, not a value.
host as rec:Host at prov:atLocation; sources and resources prov:used with qualified usage; artefacts as qualified generations with spdx:Checksum and dcat:byteSize; repositories (<run>/repository/<name>) and dependencies (<run>/dependency/<name>) as prov:SoftwareAgent; scalars as rec:Metric, one per name and step
run node https://secoro.uni-bremen.de/rec/run/<id>, base overridable per observer
New FileObserver(directory): one document per run at <dir>/<run_id>.ld.json; a run id must be a plain file name.
Run mints one id for every observer when the caller gives none, instead of adopting whatever run the database last had active. It records its result on completion and never closes its observers: whoever created an observer closes it once its runs are over.
Exceptions are logged with their trace and recorded as rec:fail-trace; prints are logging. One lock per observer, since a run's heartbeat thread writes the same row as the run.
click, dotenv and mariadb are no longer hard dependencies; MariaDB is an extra.
Tests
Conformance of a recorded run against the metamodels shapes, document round trip, six parallel runs on one observer, a queued run cancelled by id from a second observer, MariaDB behind REC_TEST_MARIADB_DATABASE. CI checks out secorolab/metamodels@main and runs against MariaDB 11 on Python 3.12 and 3.14.
A run owns one archive, so rec:run-id already identifies it. The archive's
location moves onto prov:atLocation as a rec:PathLocation, reusing the shape
the entity records already use: moving the archive changes rec:path and leaves
the identity alone. file_sources keys on run_id.
The lifecycle RDF type is now the only record of state. rec:status duplicated
it as a literal, and exec:ExecutionContext was asserted on every run without
anything requiring it; the metamodel constrains neither.
Reply to review on the run lifecycle and collected information.
Restore the queue timestamp dropped from log_queued_run, so a queued run is
timed and time-in-queue is measurable. Make queue and cancel public: a run
can only be cancelled out of the queue, while one stopped mid-flight is
interrupted, and a cancelled run refuses to start.
Keep both exceptions that end a run and store the stacktrace as
rec:fail-trace, so a failed archive says why it failed. Report progress
through the logging module rather than the prints that were removed.
Give every metric an auto-incrementing step when the caller omits one,
instead of silently overwriting the previous point, and record its name and
capture time. Collect host details at start from the standard library, so
rec:Host stops being an interface nobody calls.
Initialise the heartbeat handle, which raised AttributeError whenever the
interval was zero.
Restore the trigger and starter arguments dropped from _emit_started, and
pass them through Run.run. The entity that triggered the run rides on
prov:wasStartedBy, and the activity that generated that trigger on a
prov:Start, mirroring how usages and generations are already qualified.
A run started without either stays unqualified.
Runs are identified by a UUID, which no one wants to read or sort a
dashboard by. The runs table has always had an auto-increment db_id; keep it
on the observer so a dashboard can show and order by a short number while
rec:run-id stays the portable identity.
Archives imported with sync_files are numbered in start-time order, so the
number tracks when runs actually started.
The vocabularies are laid out as one .json and one .shacl.ttl each; the bare
prov.ttl and rec.ttl the consolidation looked for are gone, which also broke the
checkout probe that located the metamodels directory by rec/rec.ttl.
Claude-Session: https://claude.ai/code/session_017PYUidvZmUAqzoAYntn5sS
The run is a prov-ext:Execution whose lifecycle is oslc_auto:state and
oslc_auto:verdict (OSLC Automation 2.1) instead of six rec classes;
rec:run-id goes, the run id is the last segment of the run IRI. Queue time
and the latest heartbeat are rec:queued-time and rec:heartbeat-time.
Files carry spdx:Checksum and dcat:byteSize and sit at a prov:atLocation
relative to the document; the host is the run's other location with
schema:identifier. Repositories and dependencies are prov:SoftwareAgents
the run wasAssociatedWith, described with schema.org terms; metrics are
entities wasGeneratedBy the run. add_agent takes the name a software agent
must carry.
Consolidation reads the subclass axioms from prov-extension.shacl.ttl,
validates against the prov, prov-extension and rec shapes, takes the run
from rec.ld.json and treats runtime.ttl as optional.
Drop consolidation: its one caller, motion-spec's archive verification,
already validated rec.ld.json against the same shapes, wrote no
provenance.trig and never checked the joined documents. owlrl leaves the
dependencies, pyshacl is dev-only, click was unused.
Drop the API nobody called: log_sources (add_resource covers it), the
pre/post run hooks, elapsed_time, info, the row-key aliases (archivePath,
hasVersion, revision, tag, label) and the Run-to-observer parameter renames.
add_resource and add_artefact take used_by/used_at/generated_by/generated_at/
label end to end; log_repositories and log_dependencies are one
add_software(name, version=, commit=, repository=) minting
<run>/software/<name>. _iri no longer expands rec:, so instances are full
IRIs and only prov: remains as a type shorthand. The Scenario and Feature
stubs, the examples directory and the GitHub gitignore template go.
The archive is rec.ld.json everywhere; sync_files globs *.ld.json.
Every observer method holds one RLock: the heartbeat thread shares the graph
and the archive file with the run, and a fast beat crashed in
temporary.replace before this. Documents are compacted against the local
prefix map and name the published rec.json in @context without fetching it;
rdflib loads a URL context to compact even with auto_compact off, which
cost 165 ms per write against 1 ms.
Docs follow: the tutorial records a named agent and a used file so it
validates against prov-ext:ExecutionShape, the quick run is documented as
not validating, the stale file-ID text is gone and the MariaDB test module
skips without the driver. Version is a static 0.1.0.
request_cancel on an observer bound to a run id writes oslc_auto:canceling
into the store, from any process that reaches it. The run adopts the request
on its next write or heartbeat: every persist first reads the stored state,
with a JSON scan for files and a SELECT for MariaDB, since an rdflib parse
fetches the context and costs hundreds of milliseconds. It then sets
Run.cancel_requested for a cooperative main() and records canceled once
main() returns; a KeyboardInterrupt after the request ends as canceled too.
Run.cancel() on a running run does the same in process.
A queued run cancelled through the store never starts: run() checks the
observers before starting, since log_started_run would otherwise overwrite
canceling with inProgress. request_cancel on a finished run raises instead
of doing nothing, and a repeated request is not an error. Run.status turns
RUNNING only after every observer has recorded the start.
Start again from main. The observer contract stays as it was: one observer
holds any number of runs and every call names its run, which is what the
review asked to keep. BaseObserver now implements the lifecycle and the
log_*/add_* methods on the two primitives main already had, get_run and
update_run_data, so a backend only stores columns. MariaDBObserver keeps its
table and per-category JSON columns and gains a run_id column; the
auto-increment id is the short number to display and sort by. FileObserver
is new: one JSON-LD document per run in a directory.
rec/jsonld.py maps a run's columns to one document on the prov,
prov-extension and rec contexts of metamodels 73 and back: the run is a
prov-ext:Execution with an OSLC Automation state and verdict, the host a
rec:Host at prov:atLocation, sources and resources prov:used with qualified
usage, artefacts qualified generations with spdx:Checksum and dcat:byteSize,
repositories and dependencies prov:SoftwareAgents with schema.org terms,
scalars rec:Metric quantities. TIMED_OUT is complete/error with a trace;
DEAD is derived from a stale heartbeat, never recorded. The run node is
https://secoro.uni-bremen.de/rec/run/<id>, the base overridable per
observer. Metrics use the qudt: prefix directly: qudt.json declares unit as
a list, which rec:MetricShape rejects.
Run mints one id for every observer when the caller gives none, instead of
adopting whatever run the database last had active. Exceptions are logged
with their trace and recorded as rec:fail-trace; prints are logging. One lock
per observer, since a run's heartbeat thread writes the same row as the run.
click, dotenv and mariadb are no longer hard dependencies; MariaDB is an
extra. The Sphinx site is gone, the README describes the record. Tests:
conformance against the metamodels shapes, round trip, six parallel runs on
one observer, cancel by id from a second observer, MariaDB behind an
environment variable.
vamsikalagaturu
changed the title
Conform REC observers and add documentation
Record runs on the rec and prov-extension vocabularies
Sep 20, 2026
The reason will be displayed to describe this comment to others. Learn more.
Done: Run.run() no longer closes its observers. Whoever created an observer closes it once its runs are over; README, the example and test_one_observer_records_many_runs_at_once say so.
State and Verdict are the OSLC Automation terms of rec.json and replace
RunStatus: a run stores both, a verdict other than unavailable needs a
complete run, and TIMED_OUT and DEAD are no longer values. Repositories
and dependencies get distinct IRIs, a file run id must be a plain file
name, completion records the result, Run.run no longer closes shared
observers, and a MariaDB table from the previous release is migrated.
This lookup is keyed only by entity, so when the same resource is used by two activities, the later usage overwrites the earlier one. Both reconstructed resource rows then receive the later usage's timestamp. Key qualified usages by both activity and entity so each activity resolves its own usage.
Prevent slug collisions with injective path encoding
rec/jsonld.py:234
_slug is lossy: distinct values such as team/controller and team controller both become team_controller. Within one category this creates duplicate JSON-LD node IDs; record() then resolves both references to the last node, replacing one repository/resource with a duplicate of the other. Use an injective path-segment encoding instead of character replacement.
The reason will be displayed to describe this comment to others. Learn more.
Done: log_started_run reads the store under the observer's lock and refuses a run it holds as canceled; Run.run() calls the observers before changing its own state, so a run cancelled by id through another observer raises instead of starting. Tested at the end of test_a_queued_run_is_cancelled_by_id_from_the_store.
…inct IRIs
log_started_run refuses a run its store holds as canceled, before the Run
object's own state changes. MariaDBObserver.close closes the connection.
Qualified usages are keyed by activity and entity, and a slug is the
percent-encoded value, so two values never share a node.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
vamsikalagaturu
changed the title
Summary
Record runs on the
recandprov-extensionvocabularies of secorolab/metamodels#73, onmain's design.What stays
run_idis on every method, as before.Run,Scenario,Feature, hooks, ingredients, the example.MariaDBObserverwith its table and per-category JSON columns. It gainsrun_id,stateandverdictcolumns, added to a table from the previous release on first connect (statusstays as it was); the auto-incrementidis the short number to display and sort by.What changes
BaseObserverimplements the lifecycle and thelog_*/add_*methods on the two primitivesmainalready had,get_runandupdate_run_data, so a backend only stores columns.rec/jsonld.pymaps a run's columns to one JSON-LD document on theprov,prov-extensionandreccontexts, and back.observer.document(run_id)returns it for any backend.prov-ext:Executionwith an OSLC Automation state and verdict:rec.Stateandrec.VerdictreplaceRunStatus, and a run stores both. A verdict other thanunavailableneeds acompleterun, as the shapes say;DEADis a reader's inference from a stale heartbeat, not a value.rec:Hostatprov:atLocation; sources and resourcesprov:usedwith qualified usage; artefacts as qualified generations withspdx:Checksumanddcat:byteSize; repositories (<run>/repository/<name>) and dependencies (<run>/dependency/<name>) asprov:SoftwareAgent; scalars asrec:Metric, one per name and stephttps://secoro.uni-bremen.de/rec/run/<id>, base overridable per observerFileObserver(directory): one document per run at<dir>/<run_id>.ld.json; a run id must be a plain file name.Runmints one id for every observer when the caller gives none, instead of adopting whatever run the database last had active. It records its result on completion and never closes its observers: whoever created an observer closes it once its runs are over.rec:fail-trace; prints arelogging. One lock per observer, since a run's heartbeat thread writes the same row as the run.click,dotenvandmariadbare no longer hard dependencies; MariaDB is an extra.Tests
Conformance of a recorded run against the metamodels shapes, document round trip, six parallel runs on one observer, a queued run cancelled by id from a second observer, MariaDB behind
REC_TEST_MARIADB_DATABASE. CI checks outsecorolab/metamodels@mainand runs against MariaDB 11 on Python 3.12 and 3.14.