For the query layer, I wish the query languages such as Malloy and PRQL would gain more adoption but lack of tooling support is tricky so I believe that's why they don't have enough community yet so I guess we're most likely stuck with SQL. :)
Ibis is a great example of it and it uses SQLGlot under the hood, it also powers SQLMesh. There are also tools like SDF that helps with query translation and I think just like S3 API becoming standard for data lake, Snowflake is in the best position to become the API layer for data warehouses.
I have been looking into the vendor APIs and settled on Snowflake already for a similar project, I'm also using SQLGlot under the hood: https://github.com/buremba/universql
Another tool to watch in the query translation layer is probably Substrait, it's trying to standardize the query plan that can be executed in multiple engines & languages other than SQL, developed by Apache Arrow people.
This is amazing 👍🙌
This is an excellent article. Thanks, Ju & Ben, for this contribution to the data community.
Great content as always !
For the query layer, I wish the query languages such as Malloy and PRQL would gain more adoption but lack of tooling support is tricky so I believe that's why they don't have enough community yet so I guess we're most likely stuck with SQL. :)
Ibis is a great example of it and it uses SQLGlot under the hood, it also powers SQLMesh. There are also tools like SDF that helps with query translation and I think just like S3 API becoming standard for data lake, Snowflake is in the best position to become the API layer for data warehouses.
I have been looking into the vendor APIs and settled on Snowflake already for a similar project, I'm also using SQLGlot under the hood: https://github.com/buremba/universql
Another tool to watch in the query translation layer is probably Substrait, it's trying to standardize the query plan that can be executed in multiple engines & languages other than SQL, developed by Apache Arrow people.
Great read and love the metaphors!
When AWS becomes the new mainframe, you just know a lot of data engineers are in a world of pain having to deal with IAM!