NOMADS GDAS Conv Obs Refactor#969
Draft
aayushg55 wants to merge 8 commits into
Draft
Conversation
Exercise the three-operation store seam through the synchronous __call__ path so cleanup delegation is verified, including that cleanup still runs when the request fails mid-flight.
Collaborator
Author
NoteOne public lexicon discrepancy remains
NNJA’s route-qualified form is more explicit and would ideally be the common contract. GDAS’s direct keys are established public behavior, so the shared base currently normalizes both forms to avoid an API break. Ideally we pick one and unify. Not done here to keep it a pure refactor. |
negin513
self-requested a review
July 15, 2026 04:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Stacked PR — draft until #966 merges.
This branch is stacked on the NCEP observation foundation in #966, so that
PR's changes appear here until it lands. Once merged, this branch will be
rebased onto
main, leaving only the NOMADS GDAS migration, shared taskplanning, tests, and validation. Until then, review commits starting with
8c054310(refactor: migrate GDAS observations to NCEP store).Summary
NomadsGDASObsConvduplicated the request, download, cache, and compilelifecycle now shared by NCEP observation sources. This PR migrates GDAS to that
foundation and removes the source-specific orchestration.
What changed
_NomadsObsStoreowns NOMADS HTTP initialization, bounded concurrentdownloads, retries, deterministic raw cache paths, and temporary cleanup.
NomadsGDASObsConvinherits input normalization, time and field validation,cycle-window unioning, URI deduplication, synchronous cleanup, ordered decode,
output projection, and frame assembly from
_NCEPObsSourceBase.and route-specific decoding adapters.
_NCEPObsTaskcontaining the URI,cycle window, and precomputed route-specific variable plan.
_NCEPObsSourceBasenormalizes prefixed and unprefixed lexicon keys,validates GPSRO descriptors, and creates PrepBUFR and GPSRO tasks.
Alignment with NNJA
GDAS and NNJA now use the same store protocol, source lifecycle, canonical
schema, field-resolution semantics, decode-error handling, cycle-window
implementation, and final
sourcemetadata convention.Both sources validate lexicon routes and build callable extraction plans before
fetching. Decode dispatch uses the task's explicit
prepbufrorgpsroroute.The remaining differences are product requirements:
historical six-hour archive cycles.
prepbufr.acft_profilesproductand normalizes its profile-stage report types. Default NNJA PrepBUFR and GDAS
require no such mapping.
Compatibility
Preserved behavior:
requests.
Deliberate alignment changes:
initialization;
resolve_fieldsreturns a validatedpa.Schema, matching NNJA and otherDataFrame sources;
attrs["source"] == "earth2studio.data.NomadsGDASObsConv".Validation
Earth2Studio source;
deselected;
gdas.py,nnja.py, andncep_obs.py;git diff --checkpassed.The live mixed-route smoke probe decoded 84,913 PrepBUFR temperature rows and
665,917 GPSRO rows from the 2026-07-14 12z NOMADS cycle with eight workers.
Out of scope
New observation products, variables, decoder semantics, schemas, QC, thinning,
selection, satellite support, and background processing.
Checklist
Dependencies
None.