Skip to content

[Feature] Use fastjsonschema for schema validation to speed up parse times #366

Description

@macks22

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Use fastjsonschema for initial json schema validation and then run the current jsonschema.Draft7Validator.iter_errors only when there are errors. On my local, fastjsonschema was ~5x faster. Since the dominant case is no errors, this results in 5x speedup on average, with very minor slowdown in error cases.

Profiled this on a project with ~19k paths (models, macros, etc.) and saw a 12.4% reduction in parse time. Also tested on smaller projects (~500 and ~1k paths) and did not see any degradation.

Mainly impacts two call sites:

  1. dbt_common.dataclass_schema.dbtClassMixin.validate() (the per-node mashumaro-schema check)
  2. dbt_core.jsonschemas.jsonschemas._validate_with_schema() (the dbt_project.yml / resource-yml deprecation pass)

Describe alternatives you've considered

  • Replace jsonschema with fastjsonschema outright. fastjsonschema raises a single JsonSchemaException on the first error and does not preserve the rich jsonschema.ValidationError shape that jsonschema_validate() / validate_model_config() rely on for dispatching additionalProperties / anyOf / type deprecation warnings. Keeping the slow path for the failure case is essential.
  • Cache Draft7Validator instances per schema: a Draft7Validator is cheap to construct; the cost is in iter_errors itself, not in instantiation. Negligible win.
  • Switch to python-fastjsonschema's compile_to_code + ship the code: would require shipping generated .py files and re-running compile on every schema bump. The in-memory cache keyed by class is functionally equivalent and has no build-step impact.
  • Pre-validate on yml read instead of per-node during parse: would require restructuring the parser. Same total work, same total time, harder to roll back if it regresses.
  • Tighten the schemas themselves so Draft7Validator runs less: addresses cause not symptom, but the schemas are auto-generated from mashumaro types and from dbt-protos. So trimming them is upstream churn for marginal gain.

Who will this benefit?

All dbt users, but mostly those with larger projects where dbt commands run slowly due to slow parse.

Are you interested in contributing this feature?

Yes, just need review of the idea and contributed code.

Anything else?

Example implementation here. Quick summary:

  1. Add fastjsonschema>=2.19.0,<3.0 to dependencies in dbt_common/pyproject.toml.
  2. Wrap dbtClassMixin.validate(cls, data) with a fast path in dbt_common/dataclass_schema.py.
    1. A new _get_fast_validator(cls, schema) helper compiles the per-class schema via fastjsonschema.compile(schema, formats=_NUMERIC_FORMAT_NOOPS, use_default=False) and caches the result in a module-level dict keyed by cls.
    2. On valid data, the fast validator returns immediately and we skip Draft7Validator(json_schema).iter_errors(data) entirely.
    3. On invalid data (or for schemas fastjsonschema cannot compile), the fast validator raises JsonSchemaException and we fall through to the slow path. So the resulting ValidationError.create_from(error) carries the same context, message, and path that callers depend on.
  3. Two fastjsonschema configuration knobs are required:
    1. use_default=False: the default behavior mutates the input dict by injecting schema default values. dbt schemas declare "default": null for several optional fields, which would inject None into typed configs and break the downstream slow-path Draft7Validator (it rejects None for the corresponding typed field).
    2. formats={int32, int64, uint64, float, double} no-op format checkers: fastjsonschema rejects unknown OpenAPI-style numeric format hints at compile time. The actual numeric constraints are already enforced by "type": "integer" / "number". So no validation is lost. The mashumaro-generated schemas don't currently use these hints, but the no-ops are included defensively for future compatibility.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions