DBT Labs rate increase
Ju Data Engineering Weekly - Ep 29
As you may have heard, DBT announced a change of its business models last August 8th.
Originally DBT was priced based on the number of users of your team. They have announced that the team plan will be limited by the number of models built per month.
Their main argument for this pricing shift is based on the belief that DBT labs offer too much value in comparison to the price they used to charge.
Since its launch in January 2020, dbt Cloud’s pricing has been almost completely based on the number of developer seats. But we’ve always known that seats are a poor way to measure the value that organizations get out of the dbt Cloud platform.
It's likely that this change in pricing strategy was influenced by their recent funding round with a valuation of $4.2 billion $ (!!) and the pressure to increase revenue in the coming months.
I wanted to take this opportunity to have a look at DBT Labs and other alternatives to deploy DBT models in this newsletter release.
DBT Labs?
DBT (core) is a tool that helps you organize objects in a data warehouse.
Tables and views are defined as models linked to each other. This allows (among others) visualizing the relationships between SQL objects and getting data lineage across the transformations you perform on the data.
One model (table or view) is defined as 1 SQL file in your project.
When the developer runs the command dbt run, the tables, and views are created in the data warehouse, following the execution path defined by how you link the models to each other.
In order to recompute a model at regular time intervals, you need a server instance capable of running these DBT commands.
This is exactly the added value of DBT Labs: they host your SQL files and run the models according to the schedule you define.
The core of DBT is open-source and free, but DBT Labs (the hosted version of DBT) is not.
The new pricing model imposes a restriction on the number of models (tables and views) you can create per month. Specifically, you can build a maximum of 15,000 models per month in addition to your team license, which already costs $100.
Let's consider an example: if you want to update a table every hour, you would already have 720 models built per month.
Now, let's say you have 100 models to update every hour. This would result in 72,000 model runs per month, leading to a cost of $720 per month for model runs alone, in addition to the $100 team license fee. This totals to $820 per month + the actually compute cost of your WH !
Alternatives?
This move has left a sour taste for many community members. As some members of the community have pointed out, this change appears to be a recurring pattern in the software industry:
Create a product that is open source... get the community to be the beta-testers and contribute to the code base, and build third-party add-ons... then once you have enough momentum, setup a closed or locked version, let the open source version atrophy and start charging.
In my perspective, paying $820 solely to execute regular DBT commands seems excessive compared to the value it adds: running commands at set intervals is a problem already addressed by schedulers…
Platforms such as Airflow, Prefect, and Dagster offer the same feature at a significantly lower cost.
Airflow has a DBT operator
Dagster has a DBT asset
→ demo video made by Dagster team called “You might not need dbt Cloud“ :)
Prefect has a DBT block
This situation highlights the challenge of sustainability for open-core software.
Cloud hosting and usage-based pricing might generate revenue but also raises the risk of alienating the very community that constitutes the true asset of such software.
DBT's popularity has been significantly boosted by its extensive plugin ecosystem. Plugins like dbt-utils provide a wide array of pre-defined tests and simplify the process of creating data tests within your data warehouse.
DBT is taken between 2 fires: its productive asset (dev community) and its capital assets (VC investors).
Satisfying both of them is probably not possible as they have opposite interests.
On one hand, the strong and engaged developer community is a valuable asset that contributes to the tool's popularity and functionality.
On the other hand, venture capital investors often seek ways to maximize their return on investment, which can lead to changes in pricing models or strategies that may not align with the community's expectations.
This is quite unique in a business: the asset of a company is rarely limiting its revenues….
New open-source projects similar to DBT are now emerging:
https://dataform.co/
https://sqlmesh.com/
What would happen if the DBT community moved (at least partially) to one of these projects?
Community shifts and price updates of the hosted version are, in my view, the primary risk drivers when considering using open-source software. While there isn't much you can do to control the community's direction, evaluating its dynamism is crucial. When it comes to the risk of price updates, self-hosting, even if it requires DevOps resources and costs more to set up, can become an investment with good return...
Thanks for reading,
-Ju
I would be grateful if you could help me to improve this newsletter. Don’t hesitate to share with me what you liked/disliked and the topic you would like to be tackled.
P.S. you can reply to this email; it will get to me.