fix(dataset): find flaginput files by the time in the update ID - #217
Open
rikvl wants to merge 5 commits into
Open
fix(dataset): find flaginput files by the time in the update ID#217rikvl wants to merge 5 commits into
rikvl wants to merge 5 commits into
Conversation
validate_flag_updates collected the unique dataset IDs over every frequency, but the input flags it compares are not frequency dependent and the comparison only ever selects frames at freq_id. Dataset IDs from other frequencies select no frames, so their flags were never compared -- but they were still resolved against chimedb, and a failure to find their flags could abort the task. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The flags were kept as one FlagInputData per acquisition, and every dataset
looked up its update by scanning each acquisition's update_id list in turn.
Read them into a flat {update_id: (acquisition, flags)} mapping instead, which
makes the lookup a dict hit and gives a place to add flags read later.
Also fixes the mismatch warning naming the wrong acquisition: flg_acq was left
over from the search loop, so it reported whichever acquisition was iterated
last rather than the one the flags came from.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The flaginput files to validate against were selected from the timespan of the chimestack files being processed, minus a fixed 32h to allow for flag updates that get re-sent. When the flag broker restarts it re-sends the last update it knows about under that update's original ID, so after an outage longer than the allowance the chimestack file references an update that is outside the window, and the task fails with "Flag ID ... not found" even though the flaginput file holding it is right there on disk. Update IDs carry the time they were created, so use that instead of guessing: if an update isn't among the flags already loaded, ask the tracker for the flaginput file covering that time and read it. The 32h look-back stays as a pre-load for the common case, but no longer determines what can be found. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Raising DiasDataError left the chimestack file unregistered, so the next run selected it again, failed again, and never reached the files behind it. One unverifiable dataset blocked the task indefinitely. A missing update is not always someone's fault, either: the tracker only indexes staging, and files age out of it, so an update that was re-sent from long enough ago can be genuinely unreachable. Count it in the updateid_not_found metric, warn, and carry on with the next dataset. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Author
|
Deployed to The first run should work through the backlog that has accumulated since 2026-07-22. |
- The flaginput files are selected with the tracker, not with a ch_util Finder. This analyzer never used a Finder; the comment above the new_files call was left over from an earlier version. - Drop "Loop over contiguous periods within this acquisition" from a line that just takes the acquisition's file list. Its twin in the flaginput loop went away when that loop was replaced. - The keys of the flags mapping are acquisition directories, as returned by get_acquisitions, not acquisition names. Also reword the log line for the flaginput files being read. "Now processing acquisition <acq> (1 files)" reads as if the acquisition holds one file, when it is the number selected out of it -- which is exactly the count that mattered when the analyzer failed to find an update, and exactly the wrong thing to be ambiguous about. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
rikvl
force-pushed
the
rvl/dataset-fixes
branch
from
August 18, 2026 11:21
d9c0894 to
de08a08
Compare
Contributor
Author
|
This unblocked the dataset analyzer on |
rikvl
marked this pull request as ready for review
August 18, 2026 11:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #216
Five commits, reviewable in order (the last two commits are worth reading together with the third):
freq_idare resolved, matching the frequency the comparison is actually made at.{update_id: (acquisition, flags)}map instead of oneFlagInputDataper acquisition scanned linearly per dataset. No behaviour change; it gives the next commit somewhere to put flags read later.<type>_<YYYYmmddTHHMMSS.ffffffZ>_<sources>, sofind_flagsparses that and asks the tracker for the flaginput file covering that moment when an update isn't already loaded. The 32 h look-back stays as a pre-load for the common case, but no longer decides what can be found. Hits and misses are both cached, so each update is searched for at most once per acquisition.updateid_not_foundand skips that dataset.Notes:
flg_acqleft over from the search loop, i.e. whichever acquisition was iterated last rather than the one the flags came from.UPDATE_ID_SEARCH_MARGINof 1 h is needed because the tracker selects files withstart_time < endandend_time > start. An update that is the last one in its file (which is exactly the case after an outage) falls outside its own file's range with a zero-width window.