Is this your first time submitting a feature request?
Describe the feature
Use fastjsonschema for initial json schema validation and then run the current jsonschema.Draft7Validator.iter_errors only when there are errors. On my local, fastjsonschema was ~5x faster. Since the dominant case is no errors, this results in 5x speedup on average, with very minor slowdown in error cases.
Profiled this on a project with ~19k paths (models, macros, etc.) and saw a 12.4% reduction in parse time. Also tested on smaller projects (~500 and ~1k paths) and did not see any degradation.
Mainly impacts two call sites:
dbt_common.dataclass_schema.dbtClassMixin.validate() (the per-node mashumaro-schema check)
dbt_core.jsonschemas.jsonschemas._validate_with_schema() (the dbt_project.yml / resource-yml deprecation pass)
Describe alternatives you've considered
- Replace
jsonschema with fastjsonschema outright. fastjsonschema raises a single JsonSchemaException on the first error and does not preserve the rich jsonschema.ValidationError shape that jsonschema_validate() / validate_model_config() rely on for dispatching additionalProperties / anyOf / type deprecation warnings. Keeping the slow path for the failure case is essential.
- Cache
Draft7Validator instances per schema: a Draft7Validator is cheap to construct; the cost is in iter_errors itself, not in instantiation. Negligible win.
- Switch to
python-fastjsonschema's compile_to_code + ship the code: would require shipping generated .py files and re-running compile on every schema bump. The in-memory cache keyed by class is functionally equivalent and has no build-step impact.
- Pre-validate on yml read instead of per-node during parse: would require restructuring the parser. Same total work, same total time, harder to roll back if it regresses.
- Tighten the schemas themselves so
Draft7Validator runs less: addresses cause not symptom, but the schemas are auto-generated from mashumaro types and from dbt-protos. So trimming them is upstream churn for marginal gain.
Who will this benefit?
All dbt users, but mostly those with larger projects where dbt commands run slowly due to slow parse.
Are you interested in contributing this feature?
Yes, just need review of the idea and contributed code.
Anything else?
Example implementation here. Quick summary:
- Add
fastjsonschema>=2.19.0,<3.0 to dependencies in dbt_common/pyproject.toml.
- Wrap
dbtClassMixin.validate(cls, data) with a fast path in dbt_common/dataclass_schema.py.
- A new
_get_fast_validator(cls, schema) helper compiles the per-class schema via fastjsonschema.compile(schema, formats=_NUMERIC_FORMAT_NOOPS, use_default=False) and caches the result in a module-level dict keyed by cls.
- On valid data, the fast validator returns immediately and we skip
Draft7Validator(json_schema).iter_errors(data) entirely.
- On invalid data (or for schemas
fastjsonschema cannot compile), the fast validator raises JsonSchemaException and we fall through to the slow path. So the resulting ValidationError.create_from(error) carries the same context, message, and path that callers depend on.
- Two
fastjsonschema configuration knobs are required:
use_default=False: the default behavior mutates the input dict by injecting schema default values. dbt schemas declare "default": null for several optional fields, which would inject None into typed configs and break the downstream slow-path Draft7Validator (it rejects None for the corresponding typed field).
formats={int32, int64, uint64, float, double} no-op format checkers: fastjsonschema rejects unknown OpenAPI-style numeric format hints at compile time. The actual numeric constraints are already enforced by "type": "integer" / "number". So no validation is lost. The mashumaro-generated schemas don't currently use these hints, but the no-ops are included defensively for future compatibility.
Is this your first time submitting a feature request?
Describe the feature
Use
fastjsonschemafor initial json schema validation and then run the currentjsonschema.Draft7Validator.iter_errorsonly when there are errors. On my local,fastjsonschemawas ~5x faster. Since the dominant case is no errors, this results in 5x speedup on average, with very minor slowdown in error cases.Profiled this on a project with ~19k paths (models, macros, etc.) and saw a 12.4% reduction in parse time. Also tested on smaller projects (~500 and ~1k paths) and did not see any degradation.
Mainly impacts two call sites:
dbt_common.dataclass_schema.dbtClassMixin.validate()(the per-node mashumaro-schema check)dbt_core.jsonschemas.jsonschemas._validate_with_schema()(the dbt_project.yml / resource-yml deprecation pass)Describe alternatives you've considered
jsonschemawithfastjsonschemaoutright.fastjsonschemaraises a singleJsonSchemaExceptionon the first error and does not preserve the richjsonschema.ValidationErrorshape thatjsonschema_validate()/validate_model_config()rely on for dispatchingadditionalProperties/anyOf/typedeprecation warnings. Keeping the slow path for the failure case is essential.Draft7Validatorinstances per schema: aDraft7Validatoris cheap to construct; the cost is initer_errorsitself, not in instantiation. Negligible win.python-fastjsonschema'scompile_to_code+ ship the code: would require shipping generated.pyfiles and re-running compile on every schema bump. The in-memory cache keyed by class is functionally equivalent and has no build-step impact.Draft7Validatorruns less: addresses cause not symptom, but the schemas are auto-generated from mashumaro types and fromdbt-protos. So trimming them is upstream churn for marginal gain.Who will this benefit?
All dbt users, but mostly those with larger projects where dbt commands run slowly due to slow parse.
Are you interested in contributing this feature?
Yes, just need review of the idea and contributed code.
Anything else?
Example implementation here. Quick summary:
fastjsonschema>=2.19.0,<3.0todependenciesindbt_common/pyproject.toml.dbtClassMixin.validate(cls, data)with a fast path indbt_common/dataclass_schema.py._get_fast_validator(cls, schema)helper compiles the per-class schema viafastjsonschema.compile(schema, formats=_NUMERIC_FORMAT_NOOPS, use_default=False)and caches the result in a module-level dict keyed bycls.Draft7Validator(json_schema).iter_errors(data)entirely.fastjsonschemacannot compile), the fast validator raisesJsonSchemaExceptionand we fall through to the slow path. So the resultingValidationError.create_from(error)carries the same context, message, and path that callers depend on.fastjsonschemaconfiguration knobs are required:use_default=False: the default behavior mutates the input dict by injecting schemadefaultvalues. dbt schemas declare"default": nullfor several optional fields, which would injectNoneinto typed configs and break the downstream slow-pathDraft7Validator(it rejectsNonefor the corresponding typed field).formats={int32, int64, uint64, float, double}no-op format checkers:fastjsonschemarejects unknown OpenAPI-style numeric format hints at compile time. The actual numeric constraints are already enforced by"type": "integer" / "number". So no validation is lost. The mashumaro-generated schemas don't currently use these hints, but the no-ops are included defensively for future compatibility.