Conversation
|
Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA. In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR. CLA has not been signed by users: @vmaasalo |
|
As a heads up, currently this is targeting |
|
Retargeted to |
recursively_check_macros_modified() indexed manifest.macros[macro_uid] directly, raising KeyError if a node's transitive macro dependency was absent from the manifest (e.g. a v2/fusion-parser manifest that omits an adapter-dispatched macro like macro.dbt_snowflake.snowflake__date_spine). The macro_uid was already confirmed not to be in modified_macros, so it's safe to skip rather than raise.
…iscovery rediscover_adapter_macros evicted every macro in the adapter's internal packages, then re-parsed only what it could find on disk, silently dropping fusion-bundled dispatch targets like snowflake__date_spine that have no .sql file. Downstream macros still referenced them in depends_on.macros, producing the dangling refs this branch's earlier commit guards against. Snapshot evicted macros and restore any not replaced by a reparsed one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
d5c92a8 to
267977e
Compare
Problem
state:modified+raises an unhandledKeyErrorwhen a node's transitive macro dependency references a macro UID that is absent frommanifest.macros.This shows up with the v2 parser (dbt-fusion), which can omit certain adapter-dispatched macros from the manifest — e.g.
macro.dbt_snowflake.snowflake__date_spine(added in dbt-snowflake 1.12.0). Whendbt compile --use-v2-parser --select state:modified+runs,recursively_check_macros_modified()walks each node's macro deps and does a direct dict index intomanifest.macros, which raises if the parser didn't emit that macro.Repro
Without
--use-v2-parserthis doesn't happen — the v1 parser emits all adapter macros.Affected versions
Root cause
Traced further:
rediscover_adapter_macros(core/dbt/parser/fusion.py) evicts every macro whosepackage_nameis one of the adapter's internal packages, then re-parses only what it can find as a.sqlfile on disk.snowflake__date_spineis a dispatch target bundled by fusion with no on-disk counterpart in dbt-snowflake 1.12.0, so it's evicted and never restored — a dangling reference that other macros'depends_on.macrosstill point to. This is what the selector hits.Fix
Two parts, addressing both the crash and the underlying data gap:
1. Defensive guard in
recursively_check_macros_modified(core/dbt/graph/selector_methods.py) — replace the direct dict index with.get()and skip the macro if it's absent:This is safe: by this point in the loop we've already checked that
macro_uidis not inself.modified_macros, so a macro missing from the dict (but not flagged modified) means the parser simply did not emit it — skipping it is conservative and correct. Also guards against any future manifest/macro dict mismatch, not just this specific gap.2. Root-cause fix in
rediscover_adapter_macros(core/dbt/parser/fusion.py) — snapshot evicted adapter macros before popping them, and after the on-disk reparse passes, restore any that weren't replaced by a freshly reparsed one. This stops the dangling references from being created in the first place; the selector guard above is belt-and-suspenders for any case this doesn't cover.Testing
tests/unit/graph/test_selector_methods.py: addedtest_select_state_changed_test_macros_missing_macro, reproducing theKeyErroragainst pre-fix code (verified red) and passing after the fix (full file: 48 passed, 2 skipped).tests/unit/parser/test_fusion.py: addedtest_keeps_adapter_macro_with_no_disk_sourcetoTestRediscoverAdapterMacros, asserting a fusion-bundled macro with no on-disk source survives rediscovery unchanged. Adjustedtest_replaces_stale_macrosto actually simulate a successful reparse (it previously asserted eviction without exercising the "found on disk" path, which had masked this bug). Full file passes (36 passed; theTestRediscoverAdapterMacrosclass needs a registered adapter plugin, verified separately withregister_adaptermocked).