Skip to content

port to Rust to leverage deterministic simulating testing - #67

Merged
hasyimibhar merged 84 commits into
mainfrom
rustify
May 2, 2026
Merged

hasyimibhar merged 84 commits into
mainfrom
rustify

Conversation

@hasyimibhar

@hasyimibhar hasyimibhar commented Apr 29, 2026 •

Copy link
Copy Markdown
Collaborator

After some consideration, I've decided to port pg2iceberg to Rust specifically to leverage deterministic simulating testing (DST). This is crucial to ensure that pg2iceberg data mirroring is correct. The architecture remains the same, but there are breaking changes due to interface differences (it's not 1.0.0 yet, so it's fine).

E2E tests are copied almost verbatim, with few additional changes:

  • Test 5 includes time data type, it was missing
  • Test 5 is switched to DuckDB instead of ClickHouse, because ClickHouse has a bug where uuid type is queried in the wrong byte ordering (should be fixed in 26.4). Pre-Rust port, pg2iceberg did not use uuid type, it uses string instead because I didn't know Iceberg has native uuid type.
  • Added test 40 for testing support for new Postgres partitions while streaming

pg2iceberg also depends on forked iceberg-rust due to missing features upstream (only 4 commits): https://github.com/polynya-dev/iceberg-rust/tree/polynya-patches

@hasyimibhar
hasyimibhar merged commit 8894d99 into main May 2, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant