Skip to content

feat(datafusion): support REST Catalog CREATE FUNCTION#499

Merged
JingsongLi merged 3 commits into
apache:mainfrom
JingsongLi:codex/rest-catalog-create-function
Jul 11, 2026
Merged

feat(datafusion): support REST Catalog CREATE FUNCTION#499
JingsongLi merged 3 commits into
apache:mainfrom
JingsongLi:codex/rest-catalog-create-function

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Add persistent CREATE FUNCTION support for SQL scalar functions in Paimon REST Catalogs. The implementation uses the existing REST SQL function expansion path to validate a candidate definition before persisting it, so newly created functions follow the same execution semantics as catalog functions read from REST.

Changes

  • Add the Java-compatible create-function REST request, catalog API, POST endpoint, and create-specific 404/409/501 error mapping.
  • Support bare, two-part, and three-part creation targets, named nullable parameters, a single nullable return value, and scalar RETURN expressions.
  • Align the recommended SQL syntax with Databricks: LANGUAGE SQL is optional with SQL as the default, and determinism is inferred from the planned expression instead of requiring IMMUTABLE; explicit IMMUTABLE remains accepted for compatibility.
  • Add token-level adapters for IF NOT EXISTS and OR ALTER, including comments and quoted identifiers, without rewriting function bodies.
  • Validate candidate functions before POST using typed placeholders, owning-catalog/database resolution, REST dependency expansion, logical and physical DataFusion planning, recursion detection, volatility checks, and scalar-only restrictions.
  • Document supported syntax, resolution rules, validation behavior, and unsupported function forms.

Testing

  • cargo test -p paimon --test rest_object_models_test
  • cargo test -p paimon --test rest_api_test --test rest_catalog_test
  • cargo test -p paimon-datafusion --lib persistent_rest_catalog_function_
  • cargo clippy -p paimon -p paimon-datafusion --all-targets -- -D warnings
  • cargo fmt --all -- --check and git diff --check

Notes

Persistent function creation is implemented by REST Catalog; other catalog implementations return Unsupported by default. Aggregate, table, multi-return, non-SQL, Stable/Volatile, lambda/file, temporary, alter, and drop function forms remain out of scope.

A full local paimon-datafusion --lib run passed 260 tests; seven existing table-scan tests require pre-populated partitioned_log_table fixtures under /tmp/paimon-warehouse, which are not present in this environment.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found one blocking issue around complex argument/return types. The existing targeted suites and formatting checks pass, but focused regression cases with ARRAY<BIGINT> reproduce failures before the function can be created or executed.

Comment thread crates/integrations/datafusion/src/sql_context.rs Outdated

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed b0dbc50. The shared recursive SQL type renderer now handles both CREATE FUNCTION validation placeholders and REST SQL function return casts. The added ARRAY regressions pass, and I also verified MAP and ROW argument signatures locally. The persistent CREATE FUNCTION suite, REST SQL function suite, and formatting check all pass. LGTM.

@JingsongLi JingsongLi merged commit 009039f into apache:main Jul 11, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants