Why not use Delta instead of Iceberg - Delta 3.0 released recently with Uniform support meaning you can write in Delta and read as either Delta or Iceberg with no data transformation needed for then use in either format and high performance on either and so even broader support across the ecosystem?
For folks who then use DBT for their sql code management there is an auto-migration for either snowflake or redshift to Delta as well open sourced to just use the best engine for the best jobs.
It saves the parquet once and then creates the metadata in each needed open source lakehouse type to be able to interacted with natively. The extra metadata storage ends up being pretty minimal.
I really enjoy your articles on exploring the multi-engine topic and I'm especially curious to see if we can use duckdb and a cloud data warehouse such as Snowflake together on the same data. This was for me also one of the reasons behind creating dbt-ibis. Thanks for linking to it in your newsletter :)
The article is missing https://substrait.io/. An intermediate representation language that could finally bring cross-engine compatible plans.
It is like the LLVM for query engines .
Substrait is also proposed in this paper from Voltron Data: https://bit.ly/codex-early
I will have a deeper look
Why not use Delta instead of Iceberg - Delta 3.0 released recently with Uniform support meaning you can write in Delta and read as either Delta or Iceberg with no data transformation needed for then use in either format and high performance on either and so even broader support across the ecosystem?
For folks who then use DBT for their sql code management there is an auto-migration for either snowflake or redshift to Delta as well open sourced to just use the best engine for the best jobs.
Thanks Tom.
I don't understand how Delta works exactly ? They save data in Iceberg format and offer an API on top of it ?
Release notes on Delta 3.0 and Uniform (and liquid clustering is now available) - https://delta.io/blog/delta-lake-3-0/
It saves the parquet once and then creates the metadata in each needed open source lakehouse type to be able to interacted with natively. The extra metadata storage ends up being pretty minimal.
Toby from SQLmesh just shared your article on LinkedIn. I am hoping that SQLmesh or SQLglot will bring our unification dreams come true!
Dream of all data eng. :)
I really enjoy your articles on exploring the multi-engine topic and I'm especially curious to see if we can use duckdb and a cloud data warehouse such as Snowflake together on the same data. This was for me also one of the reasons behind creating dbt-ibis. Thanks for linking to it in your newsletter :)
Thanks Stefan.
I am really entousiatic as well on the possiblity to mix duckdb and Snowflake together.
This will be doable next year (hopefully) when Snowflake will release Icebeg tables in public preview.