diff --git a/CHANGELOG.md b/CHANGELOG.md index a8fa14b..b8efa3d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -31,6 +31,10 @@ compatibility. - Root-project `selectors.yml` scalar string aliases for selector-backed commands through `--selector `, lowering supported alias definitions to the existing Zig selector engine. +- `state:new` selectors now read prior Manifest v12 `manifest.json` resource + IDs from `--state`, select current supported resources absent from that prior + manifest, and compose through existing graph expansion and `--exclude` + behavior. - Supported built-in DuckDB generic tests now parse and emit `where`, `limit`, `severity`, `warn_if`, and `error_if` configs for model, seed, and source tests, apply `where`/`limit` to failure-row SQL, and classify warning results diff --git a/PLAN.md b/PLAN.md index 6155b45..c381cc5 100644 --- a/PLAN.md +++ b/PLAN.md @@ -460,10 +460,11 @@ support root-project `selectors.yml` entries whose `definition` is a scalar string or a narrow composition of supported selector leaves through `union`, `intersection`, and `exclude`, then lower `--selector ` to the existing Zig selector and exclude expressions for commands sharing the selector engine. -They do not implement method: selector references, default selectors, -indirect-selection overrides, package/config YAML method broadening, -state/result/source-status matching, state artifact loading, or deferral -semantics. +They also support artifact-backed `result:*`, `source_status:*`, and first-slice +`state:new` matching through the common Zig selector engine. They do not +implement method: selector references, default selectors, indirect-selection +overrides, package/config YAML method broadening, broader state comparison, or +deferral semantics. Current `ls` output formats source note: `.agent/research/m2-ls-output-formats.md` maps dbt Core v1 `ListTask` output diff --git a/docs/COMPATIBILITY.md b/docs/COMPATIBILITY.md index 0d922a1..d22bdd2 100644 --- a/docs/COMPATIBILITY.md +++ b/docs/COMPATIBILITY.md @@ -8,7 +8,7 @@ behavior, with DuckDB as the first deterministic execution adapter. | Command | Status | Current support | Planned gaps | | --- | --- | --- | --- | | `dxt parse` | Partial | Loads supported project files and writes a deterministic Manifest v12-shaped slice, including analysis SQL resources, disabled SQL models from YAML properties, literal inline model `config(enabled=false)`, inline and YAML-disabled singular tests, and supported singular YAML patch/config fields. | Full dbt parser parity, saved queries, semantic resources, full package behavior. | -| `dxt ls` | Partial | Lists selected graph resources in text or JSON for supported selector syntax, including root-project `selectors.yml` scalar string aliases through `--selector `, `source_status:pass` / `source_status:warn` / `source_status:error` from a `--state` directory containing Sources v3 `sources.json`, and `result:error` / `result:fail` / `result:success` / `result:skipped` from Run Results v6 `run_results.json`. | Broader YAML selector composition, remaining state/result/defer selectors, full indirect-selection parity. | +| `dxt ls` | Partial | Lists selected graph resources in text or JSON for supported selector syntax, including root-project `selectors.yml` scalar string aliases through `--selector `, `state:new` from a `--state` directory containing Manifest v12 `manifest.json`, `source_status:pass` / `source_status:warn` / `source_status:error` from Sources v3 `sources.json`, and `result:error` / `result:fail` / `result:success` / `result:skipped` from Run Results v6 `run_results.json`. | Broader YAML selector composition, remaining state/result/defer selectors, full indirect-selection parity. | | `dxt clean` | Partial | Deletes configured project-relative `clean-targets`, defaulting to the effective target path; protects source directories, rejects outside-project deletion, skips missing paths and plain files, and does not require a profile. | `--no-clean-project-files-only`, Fusion positional file args, symlink/canonical-path parity, richer dbt event output. | | `dxt compile` | Partial | Compiles selected enabled SQL models, selected analyses, selected supported built-in generic tests, root-project model/source/seed column custom generic tests, installed-package model/source/seed column custom generic tests selected through `package.test_name`, and selected singular SQL tests through the supported render-only Jinja subset and writes compiled SQL plus manifest fields without opening DuckDB. | Non-column and table-level custom generic tests, full Jinja, general macro execution, adapter dispatch execution, arbitrary expressions, filters, hooks. | | `dxt run` | Partial | Executes selected enabled DuckDB SQL models with `table` and `view` materializations; writes completed prior rows plus a sanitized `error` run-result row when a selected model fails during DuckDB execution, records `skipped` rows for selected blocked model descendants, and continues later selected independent models. | Seeds, tests, snapshots, incremental, ephemeral, hooks, grants, full materialization macros. | @@ -30,7 +30,7 @@ behavior, with DuckDB as the first deterministic execution adapter. | `--target-path` | Supported | Overrides project target path for artifacts and default DuckDB file. | | `--vars` | Partial | Scalar CLI vars for narrow `ref()` / `source()` argument resolution, accepting strict JSON objects with stringified scalar values and the existing loose inline YAML-style scalar maps. | | `--select`, `--selector`, `--exclude` | Partial | Supported selector subset with graph expansion. `--selector ` reads root-project `selectors.yml` entries whose `definition` is a scalar string and lowers them to existing selector expressions. | -| `--state` | Partial | Read-only selector input for `source_status:pass` / `source_status:warn` / `source_status:error`; the path must contain dbt Sources v3 `sources.json`. It does not enable `state:`, `result:`, defer, or manifest comparison. | +| `--state` | Partial | Read-only selector input for `state:new` from dbt Manifest v12 `manifest.json`, `result:*` from Run Results v6 `run_results.json`, and `source_status:pass` / `source_status:warn` / `source_status:error` from Sources v3 `sources.json`. It does not enable `state:modified`, defer, or manifest diffing beyond prior unique-id membership. | | `--threads`, `--full-refresh` | Accepted/planned | Product semantics are not complete yet. | | `--output`, `--output-keys` | Partial | `ls` supports legacy `text`, compact `json`, dbt-style `name`, `path`, and `selector` formats. `--output-keys` filters compact JSON to `unique_id`, `resource_type`, `name`, `package_name`, `source_name`, `alias`, source-only `identifier`, `path`, `original_file_path`, `tags`, `config.materialized`, `config.tags`, `config.enabled`, `config.docs.show`, `depends_on.nodes`, `depends_on.macros`, and dxt's compact `selector` extension; full dbt node JSON and arbitrary nested-key traversal are not implemented. | | `--host`, `--port`, `--no-browser`, `--browser`, `--no-open` | Partial | `docs serve` parses these dbt Core/Fusion-shaped flags. Browser opening is intentionally unsupported in this slice; use `--no-browser`. | @@ -76,10 +76,12 @@ behavior, with DuckDB as the first deterministic execution adapter. | `--exclude` | Partial | | `tag:`, `path:`, `file:`, `package:`, `resource_type:`, `test_type:`, `config.materialized:` | Partial; wildcard matching includes `*`, `?`, and fnmatch-style bracket character classes in the supported selector methods. | | `source:`, `exposure:`, selected generic test names | Partial | +| `state:new` | Partial; reads prior Manifest v12 resource IDs from `--state/manifest.json` and participates in graph expansion such as `state:new+` and `+state:new`. | | `source_status:pass`, `source_status:warn`, `source_status:error` | Partial; reads prior Sources v3 status rows from `--state/sources.json` and participates in graph expansion such as `source_status:warn+`. | +| `result:success`, `result:error`, `result:fail`, `result:skipped` | Partial; reads prior Run Results v6 status rows from `--state/run_results.json` and participates in graph expansion such as `result:error+`. | | Wildcards | Partial; pinned to observed dbt Core behavior where tested. | | Root `selectors.yml` scalar string aliases | Partial; `--selector ` resolves root-project selector entries with scalar string `definition` values into existing selector expressions. | -| YAML selector composition, state/result/defer/access/group/version selectors | Planned | +| YAML selector composition, remaining state/defer/access/group/version selectors | Planned | ## Artifacts diff --git a/src/project.zig b/src/project.zig index d2241a9..6f26351 100644 --- a/src/project.zig +++ b/src/project.zig @@ -15,6 +15,7 @@ const manifest = @import("project/manifest.zig"); const run_results = @import("project/run_results.zig"); const selector = @import("project/selector.zig"); const source_freshness = @import("project/source_freshness.zig"); +const state_artifacts = @import("project/state.zig"); const types = @import("project/types.zig"); const util = @import("project/util.zig"); @@ -830,10 +831,12 @@ fn resolveSelection(runtime: Runtime, options: Options) !selector_config.Resolve const SelectionState = struct { source_status_index: ?source_freshness.SourceStatusIndex = null, result_status_index: ?run_results.ResultStatusIndex = null, + prior_manifest_index: ?state_artifacts.PriorManifestIndex = null, fn deinit(self: *SelectionState, allocator: std.mem.Allocator) void { if (self.source_status_index) |*index| index.deinit(allocator); if (self.result_status_index) |*index| index.deinit(allocator); + if (self.prior_manifest_index) |*index| index.deinit(allocator); self.* = .{}; } @@ -841,6 +844,7 @@ const SelectionState = struct { var ctx: selector.SelectionContext = .{}; if (self.source_status_index) |*index| ctx.source_status_index = index; if (self.result_status_index) |*index| ctx.result_status_index = index; + if (self.prior_manifest_index) |*index| ctx.prior_manifest_index = index; return ctx; } }; @@ -848,15 +852,18 @@ const SelectionState = struct { fn loadSelectionState(runtime: Runtime, options: Options, selection: selector_config.ResolvedSelection) !SelectionState { const needs_source_status = selector.usesSourceStatusSelector(selection.select, selection.exclude); const needs_result = selector.usesResultSelector(selection.select, selection.exclude); - if (!needs_source_status and !needs_result) return .{}; + const needs_state = selector.usesStateSelector(selection.select, selection.exclude); + if (!needs_source_status and !needs_result and !needs_state) return .{}; const state_dir = options.state orelse { + if (needs_state) return error.MissingStateManifestState; if (needs_result) return error.MissingResultState; return error.MissingSourceStatusState; }; var state: SelectionState = .{}; errdefer state.deinit(runtime.allocator); + if (needs_state) state.prior_manifest_index = try state_artifacts.loadPriorManifestIndex(runtime, state_dir); if (needs_source_status) state.source_status_index = try source_freshness.loadSourceStatusIndex(runtime, state_dir); if (needs_result) state.result_status_index = try run_results.loadResultStatusIndex(runtime, state_dir); return state; diff --git a/src/project/selector.zig b/src/project/selector.zig index 5c97f8d..c574d9b 100644 --- a/src/project/selector.zig +++ b/src/project/selector.zig @@ -1,6 +1,7 @@ const std = @import("std"); const run_results = @import("run_results.zig"); const source_freshness = @import("source_freshness.zig"); +const state = @import("state.zig"); const types = @import("types.zig"); const util = @import("util.zig"); @@ -15,6 +16,7 @@ const UnitTestDef = types.UnitTestDef; pub const SelectionContext = struct { source_status_index: ?*const source_freshness.SourceStatusIndex = null, result_status_index: ?*const run_results.ResultStatusIndex = null, + prior_manifest_index: ?*const state.PriorManifestIndex = null, }; pub const SelectedResource = struct { @@ -83,6 +85,16 @@ pub fn usesResultSelector(select: ?[]const u8, exclude: ?[]const u8) bool { return false; } +pub fn usesStateSelector(select: ?[]const u8, exclude: ?[]const u8) bool { + if (select) |value| { + if (selectorValueUsesState(value)) return true; + } + if (exclude) |value| { + if (selectorValueUsesState(value)) return true; + } + return false; +} + fn selectorValueUsesSourceStatus(value: []const u8) bool { var expressions = std.mem.tokenizeAny(u8, value, " \t\r\n"); while (expressions.next()) |expression| { @@ -107,6 +119,18 @@ fn selectorValueUsesResult(value: []const u8) bool { return false; } +fn selectorValueUsesState(value: []const u8) bool { + var expressions = std.mem.tokenizeAny(u8, value, " \t\r\n"); + while (expressions.next()) |expression| { + var terms = std.mem.splitScalar(u8, expression, ','); + while (terms.next()) |raw_term| { + const term = parseSelectorTerm(raw_term); + if (term.valid and std.mem.startsWith(u8, term.value, "state:")) return true; + } + } + return false; +} + fn validateSelectorExpression(value: []const u8) !void { var terms = std.mem.splitScalar(u8, value, ','); var matched_any = false; @@ -180,6 +204,7 @@ fn validateSelectorMethod(part: []const u8) !void { "config.materialized:", "source_status:", "result:", + "state:", }; for (prefixes) |prefix| { if (std.mem.startsWith(u8, part, prefix)) { @@ -189,6 +214,7 @@ fn validateSelectorMethod(part: []const u8) !void { if (std.mem.eql(u8, prefix, "test_type:") and !isSupportedTestType(value)) return error.UnsupportedSelector; if (std.mem.eql(u8, prefix, "source_status:") and !isSupportedSourceStatusSelector(value)) return error.UnsupportedSelector; if (std.mem.eql(u8, prefix, "result:") and !run_results.isSupportedResultSelectorStatus(value)) return error.UnsupportedSelector; + if (std.mem.eql(u8, prefix, "state:") and !isSupportedStateSelector(value)) return error.UnsupportedSelector; return; } } @@ -213,6 +239,10 @@ fn isSupportedSourceStatusSelector(value: []const u8) bool { return std.mem.eql(u8, value, "pass") or std.mem.eql(u8, value, "warn") or std.mem.eql(u8, value, "error"); } +fn isSupportedStateSelector(value: []const u8) bool { + return std.mem.eql(u8, value, "new"); +} + fn matchesResultSelector(unique_id: []const u8, value: []const u8, context: SelectionContext) bool { const requested = value["result:".len..]; const index = context.result_status_index orelse return false; @@ -220,6 +250,13 @@ fn matchesResultSelector(unique_id: []const u8, value: []const u8, context: Sele return std.mem.eql(u8, requested, status); } +fn matchesStateSelector(unique_id: []const u8, value: []const u8, context: SelectionContext) bool { + const requested = value["state:".len..]; + if (!std.mem.eql(u8, requested, "new")) return false; + const index = context.prior_manifest_index orelse return false; + return !index.contains(unique_id); +} + pub fn selectResources(allocator: std.mem.Allocator, graph: *const Graph, resource_type: ?[]const u8, select: ?[]const u8, exclude: ?[]const u8) ![]SelectedResource { return try selectResourcesWithContext(allocator, graph, resource_type, select, exclude, .{}); } @@ -415,6 +452,7 @@ fn matchesNodeSelectorIntersection(graph: *const Graph, node: *const Node, value } fn matchesNodeSelectorTerm(graph: *const Graph, node: *const Node, value: []const u8, context: SelectionContext) bool { + if (std.mem.startsWith(u8, value, "state:")) return matchesStateSelector(node.unique_id, value, context); if (std.mem.startsWith(u8, value, "result:")) return matchesResultSelector(node.unique_id, value, context); if (matchesSelectorPattern(value, node.name) or std.mem.eql(u8, value, node.unique_id) or matchesNodeFqnPattern(value, node)) return true; if (std.mem.startsWith(u8, value, "resource_type:")) { @@ -479,6 +517,7 @@ fn matchesTestSelectorIntersection(graph: *const Graph, test_node: *const Generi } fn matchesTestSelectorTerm(graph: *const Graph, test_node: *const GenericTestNode, value: []const u8, context: SelectionContext) bool { + if (std.mem.startsWith(u8, value, "state:")) return matchesStateSelector(test_node.unique_id, value, context); if (std.mem.startsWith(u8, value, "result:")) return matchesResultSelector(test_node.unique_id, value, context); if (matchesSelectorPattern(value, test_node.name) or std.mem.eql(u8, value, test_node.unique_id) or matchesGenericTestFqnPattern(value, test_node)) return true; if (matchesAttachedNodeNameOrFqnSelector(graph, test_node, value)) return true; @@ -532,6 +571,7 @@ fn matchesSingularTestSelectorIntersection(graph: *const Graph, test_node: *cons } fn matchesSingularTestSelectorTerm(graph: *const Graph, test_node: *const SingularTestNode, value: []const u8, context: SelectionContext) bool { + if (std.mem.startsWith(u8, value, "state:")) return matchesStateSelector(test_node.unique_id, value, context); if (std.mem.startsWith(u8, value, "result:")) return matchesResultSelector(test_node.unique_id, value, context); if (matchesSelectorPattern(value, test_node.name) or std.mem.eql(u8, value, test_node.unique_id) or matchesSingularTestFqnPattern(value, test_node)) return true; if (std.mem.startsWith(u8, value, "tag:")) { @@ -622,6 +662,7 @@ fn matchesSourceSelectorIntersection(graph: *const Graph, source: *const SourceD } fn matchesSourceSelectorTerm(graph: *const Graph, source: *const SourceDef, value: []const u8, context: SelectionContext) bool { + if (std.mem.startsWith(u8, value, "state:")) return matchesStateSelector(source.unique_id, value, context); if (std.mem.startsWith(u8, value, "result:")) return matchesResultSelector(source.unique_id, value, context); if (std.mem.startsWith(u8, value, "resource_type:")) { const resource_type = value["resource_type:".len..]; @@ -691,6 +732,7 @@ fn matchesExposureSelectorIntersection(graph: *const Graph, exposure: *const Exp } fn matchesExposureSelectorTerm(graph: *const Graph, exposure: *const ExposureDef, value: []const u8, context: SelectionContext) bool { + if (std.mem.startsWith(u8, value, "state:")) return matchesStateSelector(exposure.unique_id, value, context); if (std.mem.startsWith(u8, value, "result:")) return matchesResultSelector(exposure.unique_id, value, context); if (matchesSelectorPattern(value, exposure.name) or matchesUniqueIdFqnPattern(value, exposure.unique_id)) return true; if (std.mem.startsWith(u8, value, "resource_type:")) { @@ -752,6 +794,7 @@ fn matchesUnitTestSelectorIntersection(graph: *const Graph, unit_test: *const Un } fn matchesUnitTestSelectorTerm(graph: *const Graph, unit_test: *const UnitTestDef, value: []const u8, context: SelectionContext) bool { + if (std.mem.startsWith(u8, value, "state:")) return matchesStateSelector(unit_test.unique_id, value, context); if (std.mem.startsWith(u8, value, "result:")) return matchesResultSelector(unit_test.unique_id, value, context); if (matchesSelectorPattern(value, unit_test.name) or std.mem.eql(u8, value, unit_test.unique_id) or @@ -1436,6 +1479,67 @@ test "result selectors match run_results state and graph expansions" { try std.testing.expect(!usesResultSelector("source_status:warn", null)); } +test "state new selectors match prior manifest membership and graph expansions" { + var arena = std.heap.ArenaAllocator.init(std.testing.allocator); + defer arena.deinit(); + const allocator = arena.allocator(); + + var graph = Graph{ .allocator = allocator, .project_name = "demo" }; + defer graph.deinit(); + + try graph.nodes.append(allocator, .{ + .package_name = "demo", + .unique_id = "model.demo.stg_customers", + .name = "stg_customers", + .path = "stg_customers.sql", + .original_file_path = "models/stg_customers.sql", + .raw_code = "", + }); + try graph.nodes.append(allocator, .{ + .package_name = "demo", + .unique_id = "model.demo.customers", + .name = "customers", + .path = "customers.sql", + .original_file_path = "models/customers.sql", + .raw_code = "", + }); + try graph.nodes.items[1].depends_on.append(allocator, "model.demo.stg_customers"); + try graph.nodes.append(allocator, .{ + .package_name = "demo", + .unique_id = "model.demo.orders", + .name = "orders", + .path = "orders.sql", + .original_file_path = "models/orders.sql", + .raw_code = "", + }); + try graph.nodes.items[2].depends_on.append(allocator, "model.demo.customers"); + + var prior_unique_ids = [_][]const u8{ + "model.demo.stg_customers", + }; + const prior_manifest = state.PriorManifestIndex{ .unique_ids = &prior_unique_ids }; + const context = SelectionContext{ .prior_manifest_index = &prior_manifest }; + + const new_resources = try selectResourcesWithContext(allocator, &graph, null, "state:new", null, context); + try std.testing.expectEqual(@as(usize, 2), new_resources.len); + try std.testing.expectEqualStrings("model.demo.customers", new_resources[0].unique_id); + try std.testing.expectEqualStrings("model.demo.orders", new_resources[1].unique_id); + + const new_with_parents = try selectResourcesWithContext(allocator, &graph, null, "+state:new", null, context); + try std.testing.expectEqual(@as(usize, 3), new_with_parents.len); + try std.testing.expectEqualStrings("model.demo.customers", new_with_parents[0].unique_id); + try std.testing.expectEqualStrings("model.demo.orders", new_with_parents[1].unique_id); + try std.testing.expectEqualStrings("model.demo.stg_customers", new_with_parents[2].unique_id); + + const new_with_children = try selectResourcesWithContext(allocator, &graph, null, "state:new+", "orders", context); + try std.testing.expectEqual(@as(usize, 1), new_with_children.len); + try std.testing.expectEqualStrings("model.demo.customers", new_with_children[0].unique_id); + + try std.testing.expect(usesStateSelector("state:new+", null)); + try std.testing.expect(usesStateSelector(null, "+state:new")); + try std.testing.expect(!usesStateSelector("result:error", null)); +} + test "singular test selectors match type path file and graph expansion" { var arena = std.heap.ArenaAllocator.init(std.testing.allocator); defer arena.deinit(); diff --git a/src/project/state.zig b/src/project/state.zig new file mode 100644 index 0000000..0f1620a --- /dev/null +++ b/src/project/state.zig @@ -0,0 +1,118 @@ +const std = @import("std"); +const project_fs = @import("fs.zig"); +const types = @import("types.zig"); + +const Runtime = types.Runtime; + +pub const PriorManifestIndex = struct { + unique_ids: []const []const u8 = &.{}, + + pub fn deinit(self: *PriorManifestIndex, allocator: std.mem.Allocator) void { + for (self.unique_ids) |unique_id| allocator.free(unique_id); + allocator.free(self.unique_ids); + self.* = .{}; + } + + pub fn contains(self: *const PriorManifestIndex, unique_id: []const u8) bool { + for (self.unique_ids) |candidate| { + if (std.mem.eql(u8, candidate, unique_id)) return true; + } + return false; + } +}; + +pub fn loadPriorManifestIndex(runtime: Runtime, state_dir: []const u8) !PriorManifestIndex { + const path = try project_fs.pathJoin(runtime.allocator, &.{ state_dir, "manifest.json" }); + defer runtime.allocator.free(path); + const text = std.Io.Dir.cwd().readFileAlloc(runtime.io, path, runtime.allocator, .limited(64 * 1024 * 1024)) catch |err| switch (err) { + error.FileNotFound => return error.MissingStateManifestArtifact, + else => return err, + }; + defer runtime.allocator.free(text); + return try parsePriorManifestIndex(runtime.allocator, text); +} + +pub fn parsePriorManifestIndex(allocator: std.mem.Allocator, text: []const u8) !PriorManifestIndex { + var parsed = std.json.parseFromSlice(std.json.Value, allocator, text, .{}) catch return error.MalformedStateManifestArtifact; + defer parsed.deinit(); + + const root = if (parsed.value == .object) parsed.value.object else return error.MalformedStateManifestArtifact; + const metadata_value = root.get("metadata") orelse return error.MalformedStateManifestArtifact; + const metadata = if (metadata_value == .object) metadata_value.object else return error.MalformedStateManifestArtifact; + const schema_value = metadata.get("dbt_schema_version") orelse return error.MalformedStateManifestArtifact; + const schema_version = if (schema_value == .string) schema_value.string else return error.MalformedStateManifestArtifact; + if (!std.mem.eql(u8, schema_version, "https://schemas.getdbt.com/dbt/manifest/v12.json")) return error.UnsupportedStateManifestSchemaVersion; + + var unique_ids: std.ArrayList([]const u8) = .empty; + errdefer { + for (unique_ids.items) |unique_id| allocator.free(unique_id); + unique_ids.deinit(allocator); + } + + try appendManifestMapUniqueIds(allocator, &unique_ids, root, "nodes"); + try appendManifestMapUniqueIds(allocator, &unique_ids, root, "sources"); + try appendManifestMapUniqueIds(allocator, &unique_ids, root, "exposures"); + try appendManifestMapUniqueIds(allocator, &unique_ids, root, "unit_tests"); + + return .{ .unique_ids = try unique_ids.toOwnedSlice(allocator) }; +} + +fn appendManifestMapUniqueIds(allocator: std.mem.Allocator, unique_ids: *std.ArrayList([]const u8), root: std.json.ObjectMap, field: []const u8) !void { + const value = root.get(field) orelse return error.MalformedStateManifestArtifact; + const object = if (value == .object) value.object else return error.MalformedStateManifestArtifact; + var iterator = object.iterator(); + while (iterator.next()) |entry| { + const resource = if (entry.value_ptr.* == .object) entry.value_ptr.*.object else return error.MalformedStateManifestArtifact; + const unique_id_value = resource.get("unique_id") orelse return error.MalformedStateManifestArtifact; + const unique_id = if (unique_id_value == .string) unique_id_value.string else return error.MalformedStateManifestArtifact; + if (!std.mem.eql(u8, entry.key_ptr.*, unique_id)) return error.MalformedStateManifestArtifact; + if (!containsString(unique_ids.items, unique_id)) try unique_ids.append(allocator, try allocator.dupe(u8, unique_id)); + } +} + +fn containsString(values: []const []const u8, needle: []const u8) bool { + for (values) |value| { + if (std.mem.eql(u8, value, needle)) return true; + } + return false; +} + +test "prior manifest index reads supported resource maps" { + const text = + \\{ + \\ "metadata": {"dbt_schema_version": "https://schemas.getdbt.com/dbt/manifest/v12.json"}, + \\ "nodes": { + \\ "model.demo.customers": {"unique_id": "model.demo.customers"}, + \\ "test.demo.not_null_customers.abc": {"unique_id": "test.demo.not_null_customers.abc"} + \\ }, + \\ "sources": {"source.demo.raw.customers": {"unique_id": "source.demo.raw.customers"}}, + \\ "exposures": {"exposure.demo.dashboard": {"unique_id": "exposure.demo.dashboard"}}, + \\ "unit_tests": {"unit_test.demo.customers.assert_rows": {"unique_id": "unit_test.demo.customers.assert_rows"}} + \\} + ; + var index = try parsePriorManifestIndex(std.testing.allocator, text); + defer index.deinit(std.testing.allocator); + + try std.testing.expect(index.contains("model.demo.customers")); + try std.testing.expect(index.contains("test.demo.not_null_customers.abc")); + try std.testing.expect(index.contains("source.demo.raw.customers")); + try std.testing.expect(index.contains("exposure.demo.dashboard")); + try std.testing.expect(index.contains("unit_test.demo.customers.assert_rows")); + try std.testing.expect(!index.contains("model.demo.orders")); +} + +test "prior manifest index rejects malformed and unsupported manifests" { + try std.testing.expectError(error.MalformedStateManifestArtifact, parsePriorManifestIndex(std.testing.allocator, "{}")); + try std.testing.expectError( + error.UnsupportedStateManifestSchemaVersion, + parsePriorManifestIndex(std.testing.allocator, + \\{"metadata":{"dbt_schema_version":"https://schemas.getdbt.com/dbt/manifest/v11.json"},"nodes":{},"sources":{},"exposures":{},"unit_tests":{}} + ), + ); + try std.testing.expectError( + error.MalformedStateManifestArtifact, + parsePriorManifestIndex(std.testing.allocator, + \\{"metadata":{"dbt_schema_version":"https://schemas.getdbt.com/dbt/manifest/v12.json"},"nodes":{"model.demo.customers":{}},"sources":{},"exposures":{},"unit_tests":{}} + ), + ); +} diff --git a/src/root.zig b/src/root.zig index 8973ca7..6879752 100644 --- a/src/root.zig +++ b/src/root.zig @@ -263,6 +263,10 @@ fn commandError(err: anyerror, stderr: *Io.Writer) ExitCode { error.MissingRunResultsArtifact => stderr.writeAll("error: --state must point to a directory containing run_results.json for result selectors\n") catch {}, error.MalformedRunResultsArtifact => stderr.writeAll("error: run_results.json is malformed or missing required result fields\n") catch {}, error.UnsupportedRunResultsSchemaVersion => stderr.writeAll("error: run_results.json must use dbt Run Results v6 schema for result selectors\n") catch {}, + error.MissingStateManifestState => stderr.writeAll("error: state selectors require --state pointing to a directory containing manifest.json\n") catch {}, + error.MissingStateManifestArtifact => stderr.writeAll("error: --state must point to a directory containing manifest.json for state selectors\n") catch {}, + error.MalformedStateManifestArtifact => stderr.writeAll("error: manifest.json is malformed or missing required state fields\n") catch {}, + error.UnsupportedStateManifestSchemaVersion => stderr.writeAll("error: manifest.json must use dbt Manifest v12 schema for state selectors\n") catch {}, error.UnsupportedCompileSelection => stderr.writeAll("error: compile currently supports only selected SQL model or supported generic or singular SQL test resources\n") catch {}, error.UnsupportedCustomGenericTest => stderr.writeAll("error: custom generic test compilation currently supports only model, seed, or source column test blocks with static SQL plus {{ model }} and {{ column_name }}\n") catch {}, error.UnsupportedRunSelection => stderr.writeAll("error: run currently supports only selected SQL model resources\n") catch {}, diff --git a/tests/test_cli.py b/tests/test_cli.py index 79d64ba..de98610 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -6621,6 +6621,43 @@ def write_run_results_state(state_dir: Path, rows: dict[str, str]) -> Path: return path +def write_manifest_state(state_dir: Path, unique_ids: list[str]) -> Path: + state_dir.mkdir(parents=True, exist_ok=True) + artifact = { + "metadata": { + "dbt_schema_version": "https://schemas.getdbt.com/dbt/manifest/v12.json", + "dbt_version": "0.0.0", + "generated_at": "1970-01-01T00:00:00Z", + "invocation_id": None, + "invocation_started_at": None, + "env": {}, + }, + "nodes": { + unique_id: {"unique_id": unique_id} + for unique_id in unique_ids + if unique_id.startswith(("model.", "analysis.", "seed.", "test.")) + }, + "sources": { + unique_id: {"unique_id": unique_id} + for unique_id in unique_ids + if unique_id.startswith("source.") + }, + "exposures": { + unique_id: {"unique_id": unique_id} + for unique_id in unique_ids + if unique_id.startswith("exposure.") + }, + "unit_tests": { + unique_id: {"unique_id": unique_id} + for unique_id in unique_ids + if unique_id.startswith("unit_test.") + }, + } + path = state_dir / "manifest.json" + path.write_text(json.dumps(artifact), encoding="utf-8") + return path + + def test_manifest_schema_validator_rejects_missing_required_key(tmp_path: Path): manifest_path = tmp_path / "manifest.json" manifest_path.write_text(json.dumps({"metadata": {}}), encoding="utf-8") @@ -9169,6 +9206,162 @@ def test_result_selector_reports_missing_malformed_and_version_mismatch(tmp_path assert "dbt Run Results v6 schema" in version.stderr +def test_ls_state_new_selects_resources_from_prior_manifest_state(tmp_path: Path): + project = copy_fixture(tmp_path, "selector_graph") + state_dir = tmp_path / "state" + write_manifest_state( + state_dir, + [ + "model.selector_graph.stg_customers", + "model.selector_graph.orders", + ], + ) + + def selected_ids(selector: str) -> list[str]: + result = subprocess.run( + [ + DXT, + "ls", + "--project-dir", + str(project), + "--state", + str(state_dir), + "--select", + selector, + "--output", + "json", + ], + cwd=ROOT, + text=True, + capture_output=True, + ) + assert result.returncode == 0, result.stderr + return [item["unique_id"] for item in json.loads(result.stdout)] + + assert selected_ids("state:new") == ["model.selector_graph.customers"] + assert selected_ids("state:new+") == [ + "model.selector_graph.customers", + "model.selector_graph.orders", + ] + assert selected_ids("+state:new") == [ + "model.selector_graph.customers", + "model.selector_graph.stg_customers", + ] + + excluded = subprocess.run( + [ + DXT, + "ls", + "--project-dir", + str(project), + "--state", + str(state_dir), + "--select", + "+state:new", + "--exclude", + "stg_customers", + "--output", + "json", + ], + cwd=ROOT, + text=True, + capture_output=True, + ) + assert excluded.returncode == 0, excluded.stderr + assert [item["unique_id"] for item in json.loads(excluded.stdout)] == [ + "model.selector_graph.customers" + ] + + +def test_state_new_selector_reports_missing_malformed_and_version_mismatch(tmp_path: Path): + project = copy_fixture(tmp_path, "selector_graph") + + missing = subprocess.run( + [ + DXT, + "ls", + "--project-dir", + str(project), + "--state", + str(tmp_path / "missing-state"), + "--select", + "state:new", + ], + cwd=ROOT, + text=True, + capture_output=True, + ) + assert missing.returncode == 2 + assert "directory containing manifest.json" in missing.stderr + + no_state = subprocess.run( + [DXT, "ls", "--project-dir", str(project), "--select", "state:new"], + cwd=ROOT, + text=True, + capture_output=True, + ) + assert no_state.returncode == 2 + assert "state selectors require --state" in no_state.stderr + + malformed_state = tmp_path / "malformed" + malformed_state.mkdir() + (malformed_state / "manifest.json").write_text( + "{\"metadata\":{},\"nodes\":{}}", + encoding="utf-8", + ) + malformed = subprocess.run( + [ + DXT, + "ls", + "--project-dir", + str(project), + "--state", + str(malformed_state), + "--select", + "state:new", + ], + cwd=ROOT, + text=True, + capture_output=True, + ) + assert malformed.returncode == 2 + assert "manifest.json is malformed" in malformed.stderr + + version_state = tmp_path / "version" + version_state.mkdir() + (version_state / "manifest.json").write_text( + json.dumps( + { + "metadata": { + "dbt_schema_version": "https://schemas.getdbt.com/dbt/manifest/v11.json" + }, + "nodes": {}, + "sources": {}, + "exposures": {}, + "unit_tests": {}, + } + ), + encoding="utf-8", + ) + version = subprocess.run( + [ + DXT, + "ls", + "--project-dir", + str(project), + "--state", + str(version_state), + "--select", + "state:new", + ], + cwd=ROOT, + text=True, + capture_output=True, + ) + assert version.returncode == 2 + assert "dbt Manifest v12 schema" in version.stderr + + def test_dbt_core_result_selector_oracle(tmp_path: Path, capsys: pytest.CaptureFixture[str]): try: has_dbt_core = importlib.util.find_spec("dbt.cli.main") is not None @@ -9274,6 +9467,107 @@ def test_dbt_core_result_selector_oracle(tmp_path: Path, capsys: pytest.CaptureF assert dxt_ids == dbt_ids +def test_dbt_core_state_new_selector_oracle(tmp_path: Path, capsys: pytest.CaptureFixture[str]): + try: + has_dbt_core = importlib.util.find_spec("dbt.cli.main") is not None + has_dbt_duckdb = importlib.util.find_spec("dbt.adapters.duckdb") is not None + except ModuleNotFoundError: + has_dbt_core = False + has_dbt_duckdb = False + + if not has_dbt_core: + pytest.skip("dbt Core is not installed for the optional state:new oracle") + if not has_dbt_duckdb: + pytest.skip("dbt DuckDB adapter is not installed for the optional state:new oracle") + + from dbt.cli.main import dbtRunner + + project = copy_fixture(tmp_path, "selector_graph") + (project / "profiles.yml").write_text( + "\n".join( + [ + "default:", + " target: dev", + " outputs:", + " dev:", + " type: duckdb", + " path: oracle.duckdb", + " schema: main", + ] + ) + + "\n", + encoding="utf-8", + ) + state_dir = tmp_path / "state" + parse_result = dbtRunner().invoke( + [ + "parse", + "--project-dir", + str(project), + "--profiles-dir", + str(project), + "--target-path", + str(state_dir), + ] + ) + capsys.readouterr() + if not parse_result.success: + pytest.skip(f"dbt Core state:new oracle parse unavailable: {parse_result.exception!r}") + + (project / "models" / "vip_customers.sql").write_text( + "select * from {{ ref('customers') }}\n", + encoding="utf-8", + ) + + dxt_result = subprocess.run( + [ + DXT, + "ls", + "--project-dir", + str(project), + "--state", + str(state_dir), + "--select", + "state:new", + "--output", + "json", + ], + cwd=ROOT, + text=True, + capture_output=True, + ) + assert dxt_result.returncode == 0, dxt_result.stderr + dxt_ids = [item["unique_id"] for item in json.loads(dxt_result.stdout)] + + dbt_result = dbtRunner().invoke( + [ + "ls", + "--project-dir", + str(project), + "--profiles-dir", + str(project), + "--target-path", + str(tmp_path / "dbt-target"), + "--state", + str(state_dir), + "--select", + "state:new", + "--output", + "json", + ] + ) + dbt_stdout = capsys.readouterr().out + dbt_ids = sorted( + json.loads(line)["unique_id"] + for line in dbt_stdout.splitlines() + if line.strip().startswith("{") + ) + if not dbt_result.success and not dbt_ids: + pytest.skip(f"dbt Core state:new oracle unavailable: {dbt_result.exception!r}") + + assert dxt_ids == dbt_ids + + def test_dbt_core_source_status_selector_oracle(tmp_path: Path, capsys: pytest.CaptureFixture[str]): try: has_dbt_core = importlib.util.find_spec("dbt.cli.main") is not None