We recently implemented Alembic for schema migration management on Databricks and came across a major concern-
There is no native CLUSTER BY support
Alembic and SQLAlchemy have no concept of Databricks-specific DDL like CLUSTER BY
Every migration file generated by autogenerate needs to be manually edited to add op.execute("ALTER TABLE ... CLUSTER BY (...)")
This is a recurring manual step that is easy to forget and can't be automated
Is there a roadmap for native CLUSTER BY and liquid clustering support and if not, can this be considered to be implemented?
We recently implemented Alembic for schema migration management on Databricks and came across a major concern-
There is no native CLUSTER BY support
Alembic and SQLAlchemy have no concept of Databricks-specific DDL like CLUSTER BY
Every migration file generated by autogenerate needs to be manually edited to add
op.execute("ALTER TABLE ... CLUSTER BY (...)")This is a recurring manual step that is easy to forget and can't be automated
Is there a roadmap for native CLUSTER BY and liquid clustering support and if not, can this be considered to be implemented?