Skip to content

fix: resolve dangling macro refs from fusion adapter-macro rediscovery - #16287

Open
vmaasalo wants to merge 2 commits into
dbt-labs:1.latestfrom
vmaasalo:fix/state-modified-missing-macro
Open

vmaasalo wants to merge 2 commits into
dbt-labs:1.latestfrom
vmaasalo:fix/state-modified-missing-macro

Conversation

@vmaasalo

@vmaasalo vmaasalo commented Sep 11, 2026

Copy link
Copy Markdown

Problem

state:modified+ raises an unhandled KeyError when a node's transitive macro dependency references a macro UID that is absent from manifest.macros.

This shows up with the v2 parser (dbt-fusion), which can omit certain adapter-dispatched macros from the manifest — e.g. macro.dbt_snowflake.snowflake__date_spine (added in dbt-snowflake 1.12.0). When dbt compile --use-v2-parser --select state:modified+ runs, recursively_check_macros_modified() walks each node's macro deps and does a direct dict index into manifest.macros, which raises if the parser didn't emit that macro.

Repro

dbt compile --use-v2-parser --select state:modified+
macro_node = self.manifest.macros[macro_uid]
KeyError: 'macro.dbt_snowflake.snowflake__date_spine'

Without --use-v2-parser this doesn't happen — the v1 parser emits all adapter macros.

Affected versions

  • dbt-core: 1.12.4
  • dbt-snowflake: 1.12.0
  • dbt-fusion (v2 parser): 2.0.0-rc.2

Root cause

Traced further: rediscover_adapter_macros (core/dbt/parser/fusion.py) evicts every macro whose package_name is one of the adapter's internal packages, then re-parses only what it can find as a .sql file on disk. snowflake__date_spine is a dispatch target bundled by fusion with no on-disk counterpart in dbt-snowflake 1.12.0, so it's evicted and never restored — a dangling reference that other macros' depends_on.macros still point to. This is what the selector hits.

Fix

Two parts, addressing both the crash and the underlying data gap:

1. Defensive guard in recursively_check_macros_modified (core/dbt/graph/selector_methods.py) — replace the direct dict index with .get() and skip the macro if it's absent:

macro_node = self.manifest.macros.get(macro_uid)
if macro_node is None:
    continue
if len(macro_node.depends_on.macros) > 0:

This is safe: by this point in the loop we've already checked that macro_uid is not in self.modified_macros, so a macro missing from the dict (but not flagged modified) means the parser simply did not emit it — skipping it is conservative and correct. Also guards against any future manifest/macro dict mismatch, not just this specific gap.

2. Root-cause fix in rediscover_adapter_macros (core/dbt/parser/fusion.py) — snapshot evicted adapter macros before popping them, and after the on-disk reparse passes, restore any that weren't replaced by a freshly reparsed one. This stops the dangling references from being created in the first place; the selector guard above is belt-and-suspenders for any case this doesn't cover.

Testing

  • tests/unit/graph/test_selector_methods.py: added test_select_state_changed_test_macros_missing_macro, reproducing the KeyError against pre-fix code (verified red) and passing after the fix (full file: 48 passed, 2 skipped).
  • tests/unit/parser/test_fusion.py: added test_keeps_adapter_macro_with_no_disk_source to TestRediscoverAdapterMacros, asserting a fusion-bundled macro with no on-disk source survives rediscovery unchanged. Adjusted test_replaces_stale_macros to actually simulate a successful reparse (it previously asserted eviction without exercising the "found on disk" path, which had masked this bug). Full file passes (36 passed; the TestRediscoverAdapterMacros class needs a registered adapter plugin, verified separately with register_adapter mocked).

@vmaasalo
vmaasalo requested a review from a team as a code owner September 11, 2026 05:46
@cla-bot

cla-bot Bot commented Sep 11, 2026

Copy link
Copy Markdown

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @vmaasalo

@cla-bot cla-bot Bot added the cla:yes label Sep 11, 2026
@vmaasalo vmaasalo changed the title fix: handle missing macros in state:modified selector fix: resolve dangling macro refs from fusion adapter-macro rediscovery Sep 11, 2026
@QMalcolm

Copy link
Copy Markdown
Contributor

As a heads up, currently this is targeting 1.12.latest. I'd recommend we actually target 1.latest and then backport it to 1.12.latest. Doing so ensures all future 1.x versions will also get the fix

@vmaasalo
vmaasalo changed the base branch from 1.12.latest to 1.latest September 14, 2026 08:24
@vmaasalo

Copy link
Copy Markdown
Author

Retargeted to 1.latest. I don't have permission to add the backport 1.12.latest label myself (403, need admin/maintainer) — could someone add it so the backport bot picks it up on merge?

vmaasalo and others added 2 commits September 14, 2026 11:27
recursively_check_macros_modified() indexed manifest.macros[macro_uid]
directly, raising KeyError if a node's transitive macro dependency was
absent from the manifest (e.g. a v2/fusion-parser manifest that omits
an adapter-dispatched macro like macro.dbt_snowflake.snowflake__date_spine).
The macro_uid was already confirmed not to be in modified_macros, so it's
safe to skip rather than raise.
…iscovery

rediscover_adapter_macros evicted every macro in the adapter's internal
packages, then re-parsed only what it could find on disk, silently
dropping fusion-bundled dispatch targets like snowflake__date_spine that
have no .sql file. Downstream macros still referenced them in
depends_on.macros, producing the dangling refs this branch's earlier
commit guards against. Snapshot evicted macros and restore any not
replaced by a reparsed one.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@vmaasalo
vmaasalo force-pushed the fix/state-modified-missing-macro branch from d5c92a8 to 267977e Compare September 14, 2026 08:30

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants