Skip to content

Add Solar Model to StormScope#943

Open
albertocarpentieri wants to merge 31 commits into
NVIDIA:mainfrom
albertocarpentieri:acarpentieri/solarstormcast
Open

Add Solar Model to StormScope#943
albertocarpentieri wants to merge 31 commits into
NVIDIA:mainfrom
albertocarpentieri:acarpentieri/solarstormcast

Conversation

@albertocarpentieri

Copy link
Copy Markdown
Collaborator

Earth2Studio Pull Request

Description

Adds StormScopeNSRDB, a StormScope-family diagnostic that estimates surface
Global Horizontal Irradiance (GHI) from StormScope GOES satellite imagery on the
3-km HRRR grid. It is a same-time estimator (given GOES at time T it estimates
GHI at T) and is designed to chain after StormScopeGOES to produce solar
irradiance nowcasts over the Continental US.
The model supports three inference modes:

  • Pure diffusion — sample GHI entirely from noise (default when no regression
    model is present).
  • Regression only — deterministic prediction from the regression model.
  • Regression + SDEdit — warm-start diffusion from the regression prediction by
    injecting noise at sdedit_sigma and denoising from there.
    It also supports clearness-index normalization (state_norm_mode="clearness_index"),
    where GHI is normalized by insolation (max(cos_zenith, 0) * solar_constant) rather
    than z-score, using the physicsnemo 1995 orbital insolation model to match training.

What's included

  • earth2studio/models/px/stormscope.py — new StormScopeNSRDB class
    (sharing StormScopeBase), including estimate_from_goes(...) as the intended
    entry point when chaining StormScopeGOES -> StormScopeNSRDB, plus regression +
    SDEdit support and clearness-index normalization.
  • earth2studio/models/nn/stormscope_util.pyMaskedModel denoiser wrapper
    that zeros outputs at invalid (off-grid / ocean) pixels after every sampling step
    while preserving the round_sigma interface used by the EDM sampler.
  • earth2studio/models/px/__init__.py — export StormScopeNSRDB.
  • examples/04_nowcasting/04_stormscope_solar_example.py — minimal example
    chaining StormScopeGOES -> StormScopeNSRDB to nowcast GHI from GOES-16 imagery
    and plot the result.
  • test/models/px/test_stormscope.py — unit tests for StormScopeNSRDB
    (estimate_from_goes output shape/coords/masking and output_coords handshakes),
    skipped when physicsnemo.utils.insolation is unavailable.
  • docs/modules/models_px.rst + CHANGELOG.md — documentation and changelog
    entries.
  • earth2studio/statistics/fss.py — minor unrelated fix: cast
    np.prod(...) to int before Tensor.view(...) (happy to drop if preferred).

Notes / follow-ups

  • StormScopeNSRDB currently has no load_default_package(); weights are loaded
    via an explicit Package(...). A default public Hugging Face package can be wired
    up once the official weights repo is available.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.
  • The CHANGELOG.md is up to date with these changes.
  • An issue is linked to this pull request.
  • Assess and address Greptile feedback (AI code review bot for guidance; use discretion, addressing all feedback is not required).

Dependencies

No new dependencies. StormScopeNSRDB relies on packages already required by the
stormscope extra (e.g. physicsnemo for the insolation model, earth2grid,
scipy).

root and others added 23 commits March 12, 2026 04:58
Signed-off-by: root <root@pool0-01737.cm.cluster>
* Update to physicsnemo v2
* lockfile and changelog
* Address feedback
* tox fix
* Serve fixes
* Fixes

---------

Co-authored-by: Nicholas Geneva <5533524+NickGeneva@users.noreply.github.com>
Co-authored-by: Nicholas Geneva <ngeneva@nvidia.com>
Signed-off-by: root <root@pool0-01737.cm.cluster>
Signed-off-by: Alberto Carpentieri <acarpentieri@cw-dfw-cs-001-vscode-01.cm.cluster>
Signed-off-by: root <root@pool0-01102.cm.cluster>
Signed-off-by: Alberto Carpentieri <acarpentieri@cw-dfw-cs-001-vscode-01.cm.cluster>
Signed-off-by: root <root@pool0-01101.cm.cluster>
Signed-off-by: root <root@pool0-01814.cm.cluster>
Signed-off-by: root <root@pool0-01814.cm.cluster>
Signed-off-by: Alberto Carpentieri <acarpentieri@cw-dfw-cs-001-vscode-01.cm.cluster>
Signed-off-by: Alberto Carpentieri <acarpentieri@cw-dfw-cs-001-vscode-01.cm.cluster>
…ntieri/earth2studio into acarpentieri/solarstormcast
Signed-off-by: Alberto Carpentieri <acarpentieri@cw-dfw-cs-001-vscode-01.cm.cluster>
Signed-off-by: acarpentieri <acarpentieri@nvidia.com>
Signed-off-by: acarpentieri <acarpentieri@nvidia.com>
Scope the branch to the StormScope NSRDB solar model contribution:

- Restore tox.ini and .gitlab-ci.yml to upstream/main (they contained
  committed merge-conflict markers) and revert unrelated test_fcn3.py /
  test_sfno.py physicsnemo-version skips and the 03 GOES example.
- Remove unrelated dev artifacts (test/serve/* and the parallel ensemble
  example script).
- Restore the dropped UCast import in models/px/__init__.py.
- Add StormScopeNSRDB unit tests, docs API entry, and CHANGELOG entry.
KDTree and cKDTree are equivalent in modern scipy (KDTree wraps cKDTree and
returns identical k=1 nearest-neighbor results), so drop the flag to reduce
API surface. Reverts utils/interp.py to upstream and removes the parameter
from StormScopeBase.build_conditioning_interpolator and the solar example.
@greptile-apps

greptile-apps Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds StormScopeNSRDB, a new StormScopeBase subclass that estimates surface GHI from StormScope GOES satellite imagery using a regression-then-SDEdit warm-start diffusion pipeline on the 3-km HRRR grid. It also adds a MaskedModel denoiser wrapper to zero invalid pixels after each sampling step, extends the existing GOES example to chain StormScopeGOES → StormScopeNSRDB, and includes a minor fss.py fix for NumPy integer type handling.

  • StormScopeNSRDB normalizes GHI via clearness index (insolation denominator), runs a single fixed regression + warm-start diffusion step, and uses physicsnemo's 1995 orbital insolation model to match training; the estimate_from_goes entry point builds a zero state and delegates to call_with_conditioning.
  • Test suite (test_stormscope.py): the new create_spoof_nsrdb_model helper passes diffusion_model= and sampler_args= to StormScopeNSRDB.__init__, which accepts neither, and omits the required model_spec and regression_model positional arguments — tests will raise TypeError in environments where physicsnemo.utils.insolation is installed.
  • fss.py fix (cast np.prod to int) is correct and uncontroversial.

Confidence Score: 4/5

Safe to merge once the test helper constructor call is corrected; the model implementation itself is sound.

The StormScopeNSRDB model logic, clearness-index normalization, MaskedModel wrapper, and SDEdit warm-start are all consistent with the described design. The only concrete defect is in the test file: create_spoof_nsrdb_model uses parameter names that don't exist in StormScopeNSRDB.__init__ and omits required arguments — every test guarded by @_needs_insolation will fail with a TypeError in any environment with physicsnemo.utils.insolation installed.

test/models/px/test_stormscope.py — the create_spoof_nsrdb_model helper needs its constructor call updated to match StormScopeNSRDB.__init__.

Important Files Changed

Filename Overview
earth2studio/models/px/stormscope.py Adds StormScopeNSRDB and MaskedModel; core logic appears sound (SDEdit warm-start, clearness-index de-normalization, valid-pixel masking) with no blocking logic errors found.
test/models/px/test_stormscope.py create_spoof_nsrdb_model passes unknown kwargs (diffusion_model, sampler_args) and omits required params (model_spec, regression_model); both new NSRDB tests will raise TypeError when insolation is available.
earth2studio/statistics/fss.py Minor fix: casts np.prod(...) result to int before passing to Tensor.view() to avoid dtype errors in newer NumPy.
examples/04_nowcasting/03_stormscope_goes_example.py Extends the GOES example to load StormScopeNSRDB, call estimate_from_goes in the rollout loop, and plot the resulting GHI field; straightforward additions.
earth2studio/models/px/init.py Adds StormScopeNSRDB to the public exports.
docs/modules/models_px.rst Adds StormScopeNSRDB to the model documentation index.
CHANGELOG.md Adds changelog entry for StormScopeNSRDB.

Comments Outside Diff (1)

  1. test/models/px/test_stormscope.py, line 661-677 (link)

    P1 Test constructor arguments don't match StormScopeNSRDB.__init__

    create_spoof_nsrdb_model passes diffusion_model=diffusion and sampler_args={"num_steps": 2}, but StormScopeNSRDB.__init__ accepts neither parameter. It also omits the two required positional parameters model_spec (a list of stage dicts) and regression_model (the regression network). Any test environment that has physicsnemo.utils.insolation installed will fail immediately with TypeError: __init__() got an unexpected keyword argument 'diffusion_model' — the @_needs_insolation guard only skips in environments without that package.

    To fix, replace diffusion_model=diffusion with a proper model_spec=[{"model": diffusion, "sigma_min": ..., "sigma_max": ...}], add a regression_model=<spoof regression model>, and drop sampler_args (NSRDB hardcodes {"num_steps": self._NUM_STEPS} internally).

Reviews (2): Last reviewed commit: "Merge branch 'NVIDIA:main' into acarpent..." | Re-trigger Greptile

Comment thread earth2studio/models/px/stormscope.py Outdated
Comment thread earth2studio/models/px/stormscope.py Outdated
Comment thread earth2studio/models/px/stormscope.py Outdated
Comment thread earth2studio/models/px/stormscope.py Outdated
- _load_grid: wrap longitudes to [0, 360) (restores the upstream GOES/MRMS
  convention dropped by the refactor; insolation now gets the [0, 360] values
  its lon_360 variable name implies).
- Revert StormScopeBase model_spec sort to upstream (descending sigma_max) so
  multi-stage GOES/MRMS expert selection is unchanged; NSRDB is single-stage
  and unaffected.
- Revert prep_input NaN/valid-mask narrowing to upstream to avoid permanently
  (and silently) shrinking valid_mask/conditioning_valid_mask across calls;
  StormScopeNSRDB already sanitizes NaN conditioning in _forward.
- Drop trailing whitespace in the optional-dependency import block.
@albertocarpentieri albertocarpentieri changed the title Acarpentieri/solarstormcast Add Solar Model to StormScope Jul 7, 2026
Comment thread earth2studio/models/nn/stormscope_util.py Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a separate file / wrapper for the NSRDB model might be better than adding it into this one, but maybe update the branch first and see how it looks with the newer stormscope base class/models. If in a separate file, can still import and subclass the base stormscope wrapper class if it makes sense and if a lot of the code would get reused.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated the PR, I believe that since this model works only on stormscope goes it is appropriate here, but also open to move it to its own py script

…arstormcast

Port StormScopeNSRDB onto the refactored StormScope base:
- Take upstream for StormScopeBase/GOES/MRMS and the new checkpoint loading.
- Re-add StormScopeNSRDB, inlining MaskedModel into stormscope.py (the old
  earth2studio/models/nn/stormscope_util.py was deleted upstream) and adding a
  private _load_grid staticmethod plus the physicsnemo insolation import.
- Re-add StormScopeNSRDB unit tests on top of upstream's rewritten test module.
- Use GOES model_name "3km_10min" (the alias in the updated nvidia/stormscope-goes-mrms package).
- The new GOES model ingests a multi-step sliding-window input, so keep only the
  last frame when estimating GHI for t=0 (StormScopeNSRDB is a same-time estimator
  expecting a single lead time). Verified end-to-end on GPU.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we want an example for this new wrapper. @NickGeneva to conform, but I think we've had this discussion before for another wrapper and the conclusion was "no example". However the Pr discussion section should have a code snippet + images/figures to help validate the model wrapper.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added the nsrdb model in the present example and removed the new one

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am just unsure if this is okay to do that. Please check with @NickGeneva

Comment on lines 53 to 55
from physicsnemo import Module
from physicsnemo.utils.insolation import insolation as pnm_insolation
from physicsnemo.utils.zenith_angle import cos_zenith_angle

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is using none of the physicsnemo diffusion module, which I am not approving at all

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is more a global dependency for the full stormscope model, maybe for a future update of the model? I do not think it is part of this PR

Comment thread earth2studio/models/px/stormscope.py Outdated
Comment thread earth2studio/models/px/stormscope.py Outdated
Comment thread earth2studio/models/px/stormscope.py Outdated

@CharlelieLrt CharlelieLrt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few comments. Main things:

  • Relies entirely on legacy APIs from physicnemo, that's a clear no-go for me
  • Over-engineered
  • No need for an example

My suggestion it's to create a wrapper class from scratch (don't inherit the StormScope one), because I am in any case not sure how much code you are really saving by composing the classes here. Also, keep the parameters from the model package that are absolutely necessary. You should have an idea of what works best, so we should stick to that in this wrapper. This will help simplify a lot of things.

I'll review again after these changes are implemented.

Comment thread earth2studio/models/px/stormscope.py Outdated
albertocarpentieri and others added 3 commits July 15, 2026 06:39
- Collapse to a single fixed pipeline (regression warm-start -> diffusion
  refinement in clearness-index space); drop the 3 inference modes.
- Load complete physicsnemo EDMPreconditioner + regression checkpoints via the
  base _load_checkpoints / _build_grid_and_times / _build_normalization and reuse
  the shared base EDM sampler (no more _use_forecast_checkpoint / custom sampling).
- Trim __init__ from ~25 args to essentials; hardcode what works best
  (sigma_max=0.25, num_steps=12, clearness-index eps/solar-constant, always mask).
- Rename sdedit_sigma -> sigma_max; remove AR(1) noise, reset_noise, disable_norm,
  lead-time labels, standard-norm path, and the _load_grid duplicate.
- Fold the solar example into 03_stormscope_goes_example.py and delete the
  dedicated 04 example.
Wrap the diffusion sampler call in torch.autocast(enabled=self.amp) to match the
base _forward (amp was previously a no-op for NSRDB). Drop MaskedModel.round_sigma
(the refactored base sampler no longer calls it) and fix its docstring.
@albertocarpentieri
albertocarpentieri marked this pull request as draft July 16, 2026 13:26
@albertocarpentieri
albertocarpentieri marked this pull request as ready for review July 16, 2026 13:28
create_spoof_nsrdb_model used the old signature (diffusion_model=, sampler_args=).
Use model_spec=[{model, sigma_min, sigma_max}] + a spoof regression_model, and set
sampler_args after construction (num_steps is hardcoded in __init__).
Comment on lines +2698 to +2706
# Regression first guess (masked), then warm-started diffusion refinement.
reg = self._sanitize(self.regression_model(condition))
reg = reg * self.valid_mask.reshape(1, 1, h, w).to(dtype=reg.dtype)
noise = torch.randn(
(b * t, len(self.variables), h, w),
device=device,
dtype=self._SAMPLER_DTYPE,
)
x_init = reg / self.sigma_max + noise

@CharlelieLrt CharlelieLrt Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you initializing the latent x_N with a noised prediction from the regression model? Does it work better than starting from plain noise? Just trying to understand the rationale for having a separate regresion model

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes correct, i tested using the diffusion model only but it gave worse results. The rationale is that the regression model can make you skip the high sigma diffusion steps, running the diffusion model only for the final part. It's an effort to make the model faster and temporally coherent.

- StormScopeNSRDB now reads the SDEdit warm-start noise level from the
  checkpoint's registry sigma_max (self.end_sigma) instead of a hardcoded
  constant, so the package is the single source of truth.
- Add StormScopeNSRDB.load_default_package() pointing at the solar package
  on Hugging Face (separate from the GOES/MRMS package).
- Update the NSRDB spoof test to mirror the real registry sigma_max (0.25).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants