[python] Reject changing immutable options once a table has snapshots#8565
Open
plusplusjiajia wants to merge 1 commit into
Open
[python] Reject changing immutable options once a table has snapshots#8565plusplusjiajia wants to merge 1 commit into
plusplusjiajia wants to merge 1 commit into
Conversation
plusplusjiajia
force-pushed
the
python-immutable-options
branch
2 times, most recently
from
78499e2 to
06382a9
Compare
The Java SchemaManager rejects altering structural options (IMMUTABLE_OPTIONS: primary-key, partition, merge-engine, bucket-key, type, ...) once a table has snapshots, but pypaimon's SchemaManager._generate_table_schema applied SetOption / RemoveOption with no guard at all. Through the filesystem/jdbc catalogs this allowed altering e.g. merge-engine or type on a table with data, silently changing read semantics or making the table unreadable (the REST catalog path was safe because the server enforces the guard). Port the guard: CoreOptions.IMMUTABLE_OPTIONS mirrors the @immutable annotated options in the Java CoreOptions (canonical key strings), and commit_changes rejects SET (when the value actually changes) and RESET of those keys once the table has a snapshot, with the same error message as Java. The Java empty-table carve-out is kept, and semantic no-ops are recognized as unchanged, mirroring Java's isUnchangedNormalizedKey: replaying the actual primary/partition keys (stored in the schema fields, not the options map) or setting the default type explicitly passes without inserting a duplicate option. No-op detection compares against the accumulated options so repeated changes to the same key in one batch still apply in order. The 'type' no-op is compared against CoreOptions.TYPE.default_value() rather than a hard-coded string, and that default is aligned with Java's TableType.TABLE ('table', not the previous 'primary-key' which is not even a valid table type and is rejected at create time by the filesystem/jdbc catalogs). So an implicit-type table now resolves its effective type to 'table', and 'type'='table' is a true no-op instead of silently reporting success while the reloaded type differed. Like the Java side, 'type' is rejected even without snapshots: table kinds like format tables never create Paimon snapshots but can still hold data. The snapshot-presence check must not fail open, so _get_latest_snapshot_from_filesystem treats the LATEST file as the hint it is and reconciles with a scan of the snapshot files, like the Java SnapshotManager.findLatest: it trusts the hint only when it points to an existing snapshot and no newer snapshot exists (a lagging _commit_latest_hint leaves it stale), and when the hint indicates the table has snapshots but none can be resolved -- an unreadable LATEST plus a listing an IO error truncated to empty -- it raises rather than reporting the populated table as empty. test_time_travel_before_row_tracking_raises relied on enabling row-tracking.enabled on a table that already had a snapshot -- a state the Java guard forbids constructing -- so it now asserts the alter itself is rejected.
plusplusjiajia
force-pushed
the
python-immutable-options
branch
from
06382a9 to
f60d646
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Java's
SchemaManagerrejects altering structural options (IMMUTABLE_OPTIONS:primary-key,partition,merge-engine,bucket-key,type, ...) once a table has snapshots, but pypaimon appliedSetOption/RemoveOptionwith no guard at all — through the filesystem/jdbc catalogs one could alter e.g.merge-engineortypeon a table with data, silently changing read semantics or making the tableunreadable (the REST path is safe: the server enforces it).
This ports the guard:
CoreOptions.IMMUTABLE_OPTIONSmirrors the Java@Immutableoptions;commit_changesrejects SET (when the value actually changes) and RESET of those keys once a snapshot exists, withthe Java-identical error message. The empty-table carve-out is kept.
test_time_travel_before_row_tracking_raisesrelied on enablingrow-tracking.enabledon a table that already had a snapshot — a state the Java guard forbids constructing — so it now asserts the alter itselfis rejected.
Tests
New
test_alter_immutable_options(empty-table allowed; with snapshots SET/RESET rejected; mutable options unaffected). 229 tests across the alter-related files pass.