Skip to content

NOMADS GDAS Conv Obs Refactor#969

Draft
aayushg55 wants to merge 8 commits into
NVIDIA:mainfrom
aayushg55:gdas-conv-refactor
Draft

NOMADS GDAS Conv Obs Refactor#969
aayushg55 wants to merge 8 commits into
NVIDIA:mainfrom
aayushg55:gdas-conv-refactor

Conversation

@aayushg55

@aayushg55 aayushg55 commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Note

Stacked PR — draft until #966 merges.
This branch is stacked on the NCEP observation foundation in #966, so that
PR's changes appear here until it lands. Once merged, this branch will be
rebased onto main, leaving only the NOMADS GDAS migration, shared task
planning, tests, and validation. Until then, review commits starting with
8c054310 (refactor: migrate GDAS observations to NCEP store).

Summary

NomadsGDASObsConv duplicated the request, download, cache, and compile
lifecycle now shared by NCEP observation sources. This PR migrates GDAS to that
foundation and removes the source-specific orchestration.

What changed

  • _NomadsObsStore owns NOMADS HTTP initialization, bounded concurrent
    downloads, retries, deterministic raw cache paths, and temporary cleanup.
  • NomadsGDASObsConv inherits input normalization, time and field validation,
    cycle-window unioning, URI deduplication, synchronous cleanup, ordered decode,
    output projection, and frame assembly from _NCEPObsSourceBase.
  • GDAS supplies its lexicon, NOMADS URL templates, two-day retention policy,
    and route-specific decoding adapters.
  • GDAS and NNJA use one shared route-tagged _NCEPObsTask containing the URI,
    cycle window, and precomputed route-specific variable plan.
  • _NCEPObsSourceBase normalizes prefixed and unprefixed lexicon keys,
    validates GPSRO descriptors, and creates PrepBUFR and GPSRO tasks.

Alignment with NNJA

GDAS and NNJA now use the same store protocol, source lifecycle, canonical
schema, field-resolution semantics, decode-error handling, cycle-window
implementation, and final source metadata convention.

Both sources validate lexicon routes and build callable extraction plans before
fetching. Decode dispatch uses the task's explicit prepbufr or gpsro route.

The remaining differences are product requirements:

  • NOMADS uses asynchronous HTTP; NNJA uses anonymous S3.
  • GDAS accepts recent observation times within NOMADS retention; NNJA validates
    historical six-hour archive cycles.
  • NOMADS fetch failures abort the request; NNJA can skip archive gaps.
  • NNJA additionally supports the archive-only prepbufr.acft_profiles product
    and normalizes its profile-stage report types. Default NNJA PrepBUFR and GDAS
    require no such mapping.

Compatibility

Preserved behavior:

  • constructor signature and defaults;
  • NOMADS PrepBUFR and GPSRO URL templates;
  • variable vocabulary and route ordering;
  • cache root, deterministic filenames, retry policy, and worker limits;
  • time-window task construction;
  • decoder adapters, output columns, values, dtypes, and null handling for valid
    requests.

Deliberate alignment changes:

  • invalid time, field, and variable requests are rejected before HTTP
    initialization;
  • resolve_fields returns a validated pa.Schema, matching NNJA and other
    DataFrame sources;
  • supplied PyArrow field types are validated instead of accepted by name only;
  • returned DataFrames use
    attrs["source"] == "earth2studio.data.NomadsGDASObsConv".

Validation

  • canonical workspace preflight passed with this worktree as the resolved
    Earth2Studio source;
  • focused GDAS, NNJA, and shared NCEP tests: 50 passed, 9 slow/live tests
    deselected;
  • Ruff passed on the changed production and test files;
  • mypy passed on gdas.py, nnja.py, and ncep_obs.py;
  • Black check passed on all changed files;
  • git diff --check passed.

The live mixed-route smoke probe decoded 84,913 PrepBUFR temperature rows and
665,917 GPSRO rows from the 2026-07-14 12z NOMADS cycle with eight workers.

Out of scope

New observation products, variables, decoder semantics, schemas, QC, thinning,
selection, satellite support, and background processing.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.
  • The CHANGELOG.md is up to date with these changes.
  • An issue is linked to this pull request.
  • Assess and address Greptile feedback.

Dependencies

None.

@aayushg55

Copy link
Copy Markdown
Collaborator Author

Note

One public lexicon discrepancy remains

GDASObsConvLexicon["v"][0]  # "wind::v"
NNJAObsConvLexicon["v"][0]  # "prepbufr::wind::v"

NNJA’s route-qualified form is more explicit and would ideally be the common contract. GDAS’s direct keys are established public behavior, so the shared base currently normalizes both forms to avoid an API break.

Ideally we pick one and unify. Not done here to keep it a pure refactor.

@negin513
negin513 self-requested a review July 15, 2026 04:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant