Add RoutedObsSource composite observation data source#974
Conversation
Routes each requested variable to the backend DataFrameSource that owns it and concatenates the results on the backends' common schema. Lets a single logical observation stream span multiple archives (e.g. microwave sounders from NNJA, infrared sounders from UFS Replay) without consumers knowing about the split.
Greptile SummaryThis PR adds
|
| Filename | Overview |
|---|---|
| earth2studio/data/routed.py | New RoutedObsSource composite; crashes on empty variable lists and silently forwards caller-supplied fields that may not be in the common schema. |
| test/data/test_routed.py | Good coverage of routing, dispatch, schema intersection, and error cases; missing test for empty variable list input. |
| earth2studio/data/init.py | RoutedObsSource correctly exported from the package. |
| docs/modules/datasources_dataframe.rst | RoutedObsSource correctly added to the DataFrame data sources documentation. |
Reviews (1): Last reviewed commit: "Add RoutedObsSource to dataframe datasou..." | Re-trigger Greptile
| parts = [] | ||
| for source in self._sources: | ||
| source_variables = [v for v in variables if self._routes[v] is source] | ||
| if not source_variables: | ||
| continue | ||
| parts.append(source(time, source_variables, fields=fields)) | ||
|
|
||
| df = pd.concat(parts, ignore_index=True) |
There was a problem hiding this comment.
Empty variable list crashes with unhelpful error
When variable is an empty list or empty array, variables resolves to [], the unknown check passes silently, parts stays empty, and pd.concat([]) raises ValueError: No objects to concatenate. A caller passing np.array([]) through fetch_dataframe would hit this immediately. Adding a guard before the loop — returning an empty DataFrame or raising a clear ValueError — prevents the confusing traceback.
| if fields is None: | ||
| fields = self.SCHEMA.names |
There was a problem hiding this comment.
Caller-supplied
fields not validated against the common schema
When the caller passes explicit fields, those column names are forwarded verbatim to every involved backend with no check against self.SCHEMA. A field present in only one backend will cause the other to raise or silently drop it, producing inconsistent errors that are hard to attribute. Validating against the intersection upfront produces a clear, actionable message.
| if fields is None: | |
| fields = self.SCHEMA.names | |
| if fields is None: | |
| fields = self.SCHEMA.names | |
| else: | |
| # Normalise to a list for uniform handling below | |
| if isinstance(fields, pa.Schema): | |
| fields = fields.names | |
| elif isinstance(fields, str): | |
| fields = [fields] | |
| unknown_fields = [f for f in fields if f not in self.SCHEMA.names] | |
| if unknown_fields: | |
| raise ValueError( | |
| f"Field(s) {unknown_fields} are not in the common schema; " | |
| f"available: {self.SCHEMA.names}" | |
| ) |
Description
Adds
RoutedObsSource, a compositeDataFrameSourcethat routes each requested variable to the backend that owns it and concatenates the results on the backends' common schema (intersection of fields by name and type).Motivation: a single logical observation stream increasingly spans multiple archives — e.g. microwave sounders from NNJA (#968) while infrared sounders remain on the UFS Replay. This lets consumers (such as DA model pipelines) fetch through one source without knowing about the split:
DataFrameSourceprotocol (works withfetch_dataframeunchanged)Checklist
test/data/test_routed.py)datasources_dataframe.rst)🤖 Generated with Claude Code