feat: support case-insensitive column matching on reads (read-time parameter)#496
feat: support case-insensitive column matching on reads (read-time parameter)#496JunRuiLee wants to merge 8 commits into
Conversation
JunRuiLee
force-pushed
the
feat/case-insensitive-upstream
branch
from
e671a08 to
9a2131c
Compare
JunRuiLee
changed the title
JunRuiLee
force-pushed
the
feat/case-insensitive-upstream
branch
2 times, most recently
from
df7d35c to
7ae85fc
Compare
JunRuiLee
changed the title
Add ReadBuilder::with_case_sensitive(bool) (default true) and PredicateBuilder::new_with_case_sensitive(fields, bool), mirroring Java's RowType.getFieldIndex(name, caseSensitive). When disabled, projection and predicate column names resolve by ASCII case-folding, error on ambiguity, and the canonical schema name is stored in the predicate leaf. Projection resolution is deferred to scan/read build time so with_projection and with_case_sensitive are order-independent; with_projection does best-effort early validation, rejecting only columns that cannot match under any case sensitivity. Default stays case-sensitive.
Derive case sensitivity from the session (!enable_ident_normalization) for scan and filter translation, and carry it into the physical scan so execute() resolves column names the same way planning did. classify_filter_pushdown (no Session available) classifies case-sensitively and refuses Exact for ASCII-case-colliding schemas so a needed residual filter is never dropped.
…itive Thread the read-time flag into projection and dict_to_predicate; add the type stub and document that it must precede with_filter for predicates.
Add paimon_read_builder_with_case_sensitive and a case_sensitive parameter on the paimon_predicate_* constructors (threaded into integer coercion and PredicateBuilder). Projection names are validated for obvious typos early and resolved lazily. Covers Doris and the Go binding.
JunRuiLee
force-pushed
the
feat/case-insensitive-upstream
branch
from
7ae85fc to
242ce95
Compare
…QL reads The read-time case-insensitive feature broke two consumers: - The C predicate constructors gained a `case_sensitive` parameter in place, but the Go binding's libffi call interfaces still pass the old argument counts, leaving `case_sensitive` undefined at the ABI boundary and corrupting every Go predicate call. Restore the existing `paimon_predicate_*` signatures (case-sensitive) and add `paimon_predicate_*_with_case_sensitive` variants for opt-in callers. Compile-time signature guards pin the existing ABI. - The DataFusion path derived case sensitivity from `enable_ident_normalization`, but DataFusion resolves columns against the provider schema before `scan` runs, so a case mismatch fails at planning and the scan-level folding is never reached. Always match case-sensitively on the SQL path and document why; case-insensitive matching stays a direct-ReadBuilder (core/C/Python) feature, covered by a new end-to-end negative test.
…sitive doc DataFusion classify_filter_pushdown always resolves case-sensitively, so an unrelated ASCII case-folding collision (e.g. Name/name) no longer downgrades an otherwise Exact filter on a different column. Remove the has_ascii_case_collision guard and retarget its test to assert Exact holds for a precise column match in a case-colliding schema. Clarify that paimon_read_builder_with_case_sensitive controls projection matching only; predicate case sensitivity is chosen by the paimon_predicate_* constructor variant, independent of this setting.
| /// Output order follows the caller-specified order. An empty list is a valid | ||
| /// zero-column projection. Column-name resolution is deferred to | ||
| /// `paimon_read_builder_new_read` (order-independent with | ||
| /// `paimon_read_builder_with_case_sensitive`), so unknown, duplicate, or |
There was a problem hiding this comment.
Unknown columns are not always deferred to paimon_read_builder_new_read. The validate_projection_possible call below still rejects names that cannot match under either case mode in this function.
There was a problem hiding this comment.
Fixed the doc to match. An obvious typo — a name that matches no field under any case sensitivity — is rejected by this call (via validate_projection_possible); only case-dependent outcomes (a name matching only case-insensitively, or a case-fold duplicate/ambiguity) are deferred to paimon_read_builder_new_read, which uses the case sensitivity effective then. (2e8c912)
| F: Fn(&Predicate) -> bool, | ||
| { | ||
| let translator = FilterTranslator::new(fields); | ||
| // `FilterTranslator` still supports case-insensitive column resolution for |
There was a problem hiding this comment.
FilterTranslator is private to the DataFusion integration, so direct ReadBuilder callers never reach this case-insensitive branch. Every non-test call site passes true, leaving the false path test-only.
There was a problem hiding this comment.
True that every current call site passes true — but the case_sensitive parameter is intentional, not dead code. It is the explicit carrier of the "SQL path is always case-sensitive" contract: scan and the variant-extraction planner both pass true on purpose, and the end-to-end negative test (SELECT name vs a Name field fails at planning) pins that. The false path keeps FilterTranslator/classify consistent with the rest of the case-insensitivity machinery and is exercised by unit tests. I would rather keep the contract visible in the signature than hardcode true internally and lose that. Happy to revisit if you feel strongly.
| if name == crate::spec::ROW_ID_FIELD_NAME { | ||
| continue; | ||
| } | ||
| let matches_any = fields.iter().any(|f| f.name().eq_ignore_ascii_case(name)); |
There was a problem hiding this comment.
This scans the full schema once per projected name, and the same lookup runs again in new_scan and new_read. The default case-sensitive path regresses from the previous O(fields + projections) lookup to O(fields × projections), which is significant for wide tables.
There was a problem hiding this comment.
Fixed. Both resolve_projected_fields and validate_projection_possible now build the name index once (O(fields)) and then look up each projected name in it, restoring O(fields + projections): an exact-name HashMap for the case-sensitive path, and a folded-name HashMap (folding a collision to None) for the case-insensitive path, which keeps the ambiguity check. Dropped the now-unused find_projection_field. (2e8c912)
resolve_projected_fields and validate_projection_possible scanned the full schema once per projected name (O(fields x projections)), regressing the default case-sensitive path from the previous hash-index lookup. Build the name index once: an exact-name map for the case-sensitive path and a folded-name map (folding collisions -> None) for the case-insensitive path, preserving ambiguity detection. Drop the now-unused find_projection_field. Also correct the C paimon_read_builder_with_projection doc: an obvious typo (no field matches under any case) is rejected by this call; only case-dependent resolution is deferred to paimon_read_builder_new_read.
2 participants
Purpose
Linked issue: close #495
Spark + Java Paimon support case-insensitive column matching on reads. In Java this lives at the engine / catalog layer — Spark's
spark.sql.caseSensitive(a session setting) drives Catalyst to resolve projection/filter columns case-insensitively and normalize them to the schema names before pushdown; Paimon core exposes it as a per-call parameter (RowType.getFieldIndex(name, caseSensitive)). It is not a table property.paimon-rusthad no equivalent — column resolution was always exact-case — so callers reading Paimon through the Rust core / C bindings / Python couldn't offer the behavior users get from Spark.This PR adds case-insensitive column matching as a read-time parameter (mirroring Java's per-call
caseSensitive), not a table option. Default stays case-sensitive, so existing behavior is unchanged; a reader opts in per read. When enabled, column names resolve by ASCII case-folding and an ambiguous reference (schema fields colliding under folding) is reported, mirroring Spark'sAMBIGUOUSbehavior.Scope of the opt-in
Case-insensitive matching is offered on the direct ReadBuilder API paths: core, the C binding, and Python. It is not offered through DataFusion / SQL: DataFusion resolves projection/filter columns against the provider schema during logical planning, before
TableProvider::scanruns, andenable_ident_normalizationonly lowercases unquoted identifiers — it does not make schema resolution case-insensitive. A genuine case mismatch (SELECT nameagainst aNamefield) therefore fails at planning and never reaches Paimon's resolution. So the DataFusion path always matches case-sensitively; a new end-to-end negative test pins this.Brief change log
ReadBuilder::with_case_sensitive(bool)(defaulttrue) andPredicateBuilder::new_with_case_sensitive(fields, bool)(the existingnewdelegates withtrue). When disabled, column resolution folds case, errors on ambiguity, and stores the canonical schema name in the predicate leaf so downstream name-based lookups (e.g. index pruning) keep matching. Projection duplicate detection folds case so a case-colliding projection is reported as a duplicate.with_projectionandwith_case_sensitivemay be called in any order.with_projectiondoes best-effort early validation, rejecting only columns that cannot match under any case sensitivity (an obvious typo); case-dependent outcomes surface fromnew_read.supports_filters_pushdownclassifies case-sensitively and still refuses to reportExactfor a schema with ASCII-case-folding collisions, as a conservative guard so a needed residual filter is never dropped.PyReadBuilder.with_case_sensitive(bool)threads the flag into projection and predicate conversion (type stub added).paimon_read_builder_with_case_sensitive(rb, bool)(new symbol), plus additivepaimon_predicate_*_with_case_sensitivevariants of each predicate constructor. The existingpaimon_predicate_*symbols keep their original signatures and case-sensitive behavior, so the current C ABI and its consumers (including the in-repo Go binding, which prepares fixed libffi call interfaces per symbol) are unaffected. Compile-time signature guards pin the existing ABI. (Go's own opt-in case-insensitive API is left to a follow-up.)ASCII-only folding throughout (
eq_ignore_ascii_case); no Unicode case folding.Scope: read path, top-level columns only. Nested/dotted paths, the
_ROW_IDsystem column (kept an exact reserved name), the write path, and the DataFusion/SQL path are out of scope.Tests
cargo test -p paimon --lib— projection/predicate case-insensitive match, default-sensitive rejection, ambiguity error, canonical-name storage, duplicate detection, order-independence (projection ↔ case-sensitive either order), and deferral of case-dependent errors tonew_read.cargo test -p paimon-datafusion—filter_pushdowntranslation/classify guard, and an end-to-end SQL negative test asserting thatSELECT nameagainst aNamefield fails at planning (documenting why the SQL path stays case-sensitive).PYO3_PYTHON=python3.12).API and Format
ReadBuilder::with_case_sensitive,PredicateBuilder::new_with_case_sensitive; Cpaimon_read_builder_with_case_sensitiveand additivepaimon_predicate_*_with_case_sensitivevariants. ExistingPredicateBuilder::newand the existingpaimon_predicate_*C symbols are unchanged (case-sensitive), preserving ABI compatibility.Documentation
Behavior described above and in the linked issue; the switch is a read-time API parameter, controlled by the reader/engine (not stored on the table), and is order-independent with projection. Case-insensitive matching is available via the direct ReadBuilder API (core / C / Python); the DataFusion/SQL path stays case-sensitive.