add opt-in Iceberg table soft-delete (#5054)

[iprithv]

Polaris DROP is hard from the catalog's point of view: the active name disappears immediately, and there is no hold period. Operators who need a recoverable drop (compliance / legal hold, Nessie-to-Polaris migration) have no first-class path.

part of #5054.

This is Phase 1 only. Default remains today's hard DROP. The GitHub RFC (R4) freed the identifier on soft-delete; this PR reserves the name during the hold, matching the review comment that name reuse should wait until unique table locations are the catalog invariant. Drop-and-recreate ETL is unchanged on catalogs that leave the feature off. Catalogs that enable it can DROP PURGE when they want the name back immediately.

current behavior:

DROP TABLE without purge permanently removes the catalog entity. Recreate of the same identifier succeeds. There is no hold, and Iceberg REST list/load/HEAD cannot see a dropped table because it is gone.

expected behavior (opt-in):

When polaris.config.table-soft-delete.enabled=true on an INTERNAL catalog:

Action	Result
DROP TABLE without purge	Soft-delete: entity retained, metadata location retained, files not deleted
Iceberg REST list / load / HEAD	Hidden (isDropped())
CREATE / register same identifier during hold	Conflict (name reserved)
DROP TABLE PURGE	Immediate permanent delete (unchanged)
After hold (P7D default)	Next list/create/drop-namespace in that namespace permanently removes catalog state

File cleanup on expire is off unless polaris.config.table-soft-delete.purge-data-on-permanent-delete=true. Enabling the flag on an external catalog returns 400.

[dimas-b]

Glad to see progress on this feature 🚀

This is a partial review 😅

[dimas-b]

nit: today is different every day 😉 Maybe just hard DROP?

[iprithv]

updated to hard DROP :)

[dimas-b]

How do you envision actually purging soft-deleted tables?

Cf. https://lists.apache.org/thread/kqm0w38p7bnojq455yz7d2vdfp6ky1h7

[iprithv]

phase 1 is lazy on namespace access, not a scheduled sweep (this is also @jbonofre's question on https://lists.apache.org/thread/1djyz0x3k7m71d71bqhw11c3syyshtdr).

when TABLE_SOFT_DELETE_ENABLED is false (the default), expiration is a no-op: a config check only, no extra scan.

when it is enabled, the next list, create, or drop-namespace in that namespace loads Iceberg tables under it (including dropped ones) and permanently deletes those whose toPurgeTimestamp has passed. catalog state is always removed then. file cleanup is a separate flag and uses the existing ENTITY_CLEANUP_SCHEDULER. There is no dedicated expiration task, and no purge-now API, in phase 1.

I spelled that out on TABLE_SOFT_DELETE_HOLD_PERIOD. Cost is confined to catalogs that opt in; a namespace with many held tables pays a full Iceberg-table listing on those operations. scheduled sweep can wait until we know whether lazy-on-access is enough.

[dimas-b]

If false reusing the table name will effectively enable the new table owner to access old table's data. This is still a serious risk, I think... until Polaris enforces unique table locations.

I'd prefer to avoid this risk by always deleting data files on purge.

[iprithv]

hmm yes, agreed that is a real risk until unique table locations are the catalog invariant. after the hold we free the name, so a recreate at the same location can see leftover files.

I left the flag default-off for phase 1 so expire matches today's DROP without purge (catalog state gone, files remain) and so opting into hold/recover does not also become delayed file deletion. DROP PURGE on a live table is still immediate destroy, including files.

if you would rather Phase 1 always enqueue entity-cleanup when a soft-deleted table is permanently deleted (drop the flag, or default it true), I can do that. That would make expire match DROP PURGE for files and close the recreate-at-same-location leak while unique locations are still opt-in.