Disable PROJECT_SET_OP_TRANSPOSE rule in BeamRuleSets#39459
Draft
damccorm wants to merge 3 commits into
Draft
Conversation
Remove CoreRules.PROJECT_SET_OP_TRANSPOSE from the active rule set. This rule pushes projections through set operations (UNION/INTERSECT/ EXCEPT), but doing so collapses IEEE-754 FP surrogates wrapped around FLOAT/DOUBLE set-operation columns, reintroducing non-deterministic key coders that Beam's CoGroup-based set operators reject. Adds BeamRuleSetsTest verifying PROJECT_SET_OP_TRANSPOSE is disabled and FILTER_SET_OP_TRANSPOSE remains enabled.
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Removes
CoreRules.PROJECT_SET_OP_TRANSPOSEfrom the active planner rule set inBeamRuleSets.This rule pushes projections through set operations (UNION/INTERSECT/EXCEPT), transposing the projection into each branch where
PROJECT_MERGEthen fuses it with adjacent branch projections. This fusion cancels the deterministic IEEE-754 FP surrogate wrapped around FLOAT/DOUBLE set-operation columns (collapsingfrom_bits(bits(x))back to the rawDOUBLE), reintroducing the non-deterministicDouble/Floatkey coder that Beam's CoGroup-based set operators reject with "the keyCoder of a GroupByKey must be deterministic." The rule is a pure optimization, so omitting it is correctness-preserving.Changes
BeamRuleSets.java: ReplacedCoreRules.PROJECT_SET_OP_TRANSPOSEwith a comment explaining why it is intentionally disabled.BeamRuleSetsTest.java(new): Tests verifyingPROJECT_SET_OP_TRANSPOSEis not in the active rule set andFILTER_SET_OP_TRANSPOSEremains enabled.Files Changed
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRuleSets.javasdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRuleSetsTest.java(new)