I can never understand why people are up in arms when they are required to pay for something valuable.
In the world of open source, I think it's a bit different though. Imagine the open source project to be a community thing, where different people donate their blood, sweat and tears into. It's a project that not only did the company who owns the project invest into, but others did too.
Take SQLGlot, which we use (at Solid). The top contributors are from Tobiko, as you'd expect.
But there are many contributors who are not from Tobiko. Each such contributor made one or two commits, and probably most were immaterial to the project.
By contributing code/work, do they get a "share" of the project? A vote? Maybe each person gets a vote per commit they make, and those contributors who work for Tobiko pool their shares/votes for the interest of the company (which means Tobiko has 99% of the votes anyway)?
If SQLGlot was a company, and 99% of the shares were owned/controlled by one entity (Tobiko), and that entity made a decision to sell the company for X, is that ok?
In the world of companies and shares, there are concepts by which the 1% shouldn't be screwed in this process. Is that maybe what people are seeking here?
(I'm not justifying it btw - I think Tobiko made the right decision and have a ton of respect to the people there... just trying to rationalize through this)
People are up in arms because they create their own social contracts and expectations. Often these are naive due to lack of consideration about how software is made.
nevertheless, we try to respect people's social contracts to the extent that we can. As an opinionated software creator we regularly have to say no and let people down, to enable us to say yes and carry out our mission.
Yeah, that's the other thing about it, I guess. I doesn't exactly matter where the feeling comes from; it's there, so these companies have to deal with it in some way. You could probably write a post that was the same thing, just inverted, where you said: "What did the companies expect to happen?" and it wouldn't be any more wrong.
I think it's not just having to pay, but also how much. People were quite happy to pay for DBT Cloud in the beginning, and even considered the pricing to be a bit low.
But then when you get a pricing analyst on board in order to determine "the right price" (i.e. what the market can bear / how to charge as much as possible), no one is happy. A price bump is one thing, but a multiplication is something else. They've invested themselves into a product/company, and it feels like they've been lied to. If we take that out of a business context and apply it to let's say a partnership or a marriage, that's the kind of stuff that results in divorce (because no one wants to be married to someone who tries to determine what the relationship can bear / how to take as much as possible for themselves, which is quite the opposite of how we should treat each other).
With the number of models we're running and their latest pricing, DBT Cloud would charge us more than what we're paying for Snowflake, which is nonsense.
And no one likes sudden changes either, whatever they are. Even small UI/UX changes generate tons of complaints, so it's not just about pricing.
It seems like in the end, the solution is just to ignore the haters, and things will eventually quiet down. 🤷♂️
That's another layer of the problem, but it's also the beauty of a competitive market. You can choose other options. Fivetran's pricing increases didn't go well for some people, so cheaper competitors came in.
This may be an opportunity to build a cheaper dbt Cloud - someone surely is thinking of that startup.
At one point in time, we were using Zapier (which is quite expensive). Then we found a competitor which was much cheaper, and migrated everything. Then the competitor increased their price to match Zapier.
On the shareholder point, that's exactly how it effectively works. Small shareholders might technically get a vote, but in cases where they can't outvote the majority, they don't really. There is still typically a vote for these things, it's a formality, and those investors (and employees!) don't have a choice but to go along for the ride.
I get why someone who contributed to an open source thing might feel a little slighted by this, but they can see that this is the dynamic just as well as a customer. (And for the very very rare contributor that adds a lot, I'd guess that a lot of those people could get hired by the sponsoring company if they wanted to. Which isn't to say bad things here might not happen in this way, but I'd guess it's extremely rare, and not what drives the overall sentiment around all of this.)
And I'm kind of mixed on the pricing thing. Sure, it's unsavory, but, that's the deal. If someone's worried about it, don't use it when it's free. And I don't think that most companies want to price gouge people, and generally try to make it bearable for people - but, as I talked about here, every company has one real value. "Face eating leopard eats customer's face" isn't a good look for the leopard, but the customer probably should've seen it coming.
It's surreal to have both been on the outside looking in and the inside looking out across the companies you name here. You hit the nail on the head my guy. Incentives and time constraints pretty much enforce this narrative.
Part of the story is that to materially increase TAM under high pressure timelines, the easiest path is to make it easier and easier to use your platform. That biases for an audience of data analysts and leaders that aren't open source die hards. Heck, most dbt/sqlmesh users use bigquery and snowflake by far which aren't open source in the slightest. Hint: they're the easiest to use ;)
Then there's the counterargument of, "What about databricks and confluent?" The big difference is they sell compute. And another big keyword difference in front of their open source tools: APACHE kafka, APACHE spark. This make it very high friction to haphazardly paywall major features.
I've always believed that one of the main reasons Databricks and Confluent (and to some degree, the hosted Airflow products) have been able to make this work is because they're hard to use. Spark is a huge pain, so it's not that hard to sell "Spark, without the pain." Tools like dbt and SQLMesh are victims of their own ease of use. You can run them yourself too easily. Had they been harder - to run dbt, you need to run all these docker containers, and they're finicky, and oh the whole thing is written in scala - I think the story might've been different.
To your point the hard part of all those is kubernetes, networking, and knowing how much compute is enough to make it all work well. Spark SQL and to a degree pyspark is easy enough to learn and use on your own. It's the managing infrastructure, which is tightly coupled to making the framework work well, that people pay for to outsource entirely. Funnily enough spotify uses scala with dbt and it's wacky.
Ah, right, yeah, that's what I meant. It's not running queries that's hard, but running the platform. In a sense, dbt in probably harder to use, day to day, than Spark. But running it is like, a six character script. Whereas running spark is....not that.
Whenever you write in the style of option two, Iggy Pop starts playing in my head, and the voice reading the words to me is Ewan McGregor doing the "Choose life" monologue at the start of Trainspotting.
Man, who knew information and intelligence (which is derived from information) were so valuable? Wait... don't we all work in or with Information Technology? 🙃
I guess it depends how we define BI. If we define it as "graphs and tables in a BI tool", it's probably much less valuable. Does Excel count as a BI tool? 🤔
But without information/intelligence about our business, we're flying blind. How do you even do A/B tests?
But I guess some businesses need it less than others. Our fraud team is constantly asking for more data/information from everywhere. Science is 100% based on data, and wouldn't exist without it. And intelligence agencies sure collect a lot of it! But that's a whole other type of business. 😅
But yeah, you probably don't need a lot of BI if you're selling homemade candles on Etsy. Or those famous jaffles.
I imagine that a graph charting the *actual* value of data/BI per business would very much be a hockey stick, with companies like Google & Meta being at the top, and then dropping fast from there.
- Basic reporting and stuff - very valuable. You gotta know how many people buy your stuff, and use this or that thing.
- "Analytics" and answering deeper questions - eeeeeh pretty iffy. Theoretically it's very useful, but it's really hard to extract useful stuff from messy data. If you're Google and have huge amounts of high potential data, sure. If you're a random SaaS company, not so much. The "insight" might be in there, but the amount of energy it takes to extract it is really high, to the point that I'm not sure it's worth it for most people.
- Specialized stuff like fraud reporting - very useful in the right pockets. But I think that's sort of the temptation. We see data being useful in pockets and assume it means "data is useful" rather than thinking "data was a useful way to solve this very specific problem."
Agreed. I think for many businesses, the most useful insight is merely "this product sells better than that one". And then perhaps "this specific demographic likes this specific product". But haven't we had this for ages? I mean, I'm pretty sure that even in the stone age, they could figure out something like "pistachios sell better than walnuts, but old people like walnuts better".
Is there some hidden insight beyond that? I'm not so sure. Unless you have the scale where small details become more significant opportunities, like let's say Amazon or Walmart.
It's kinda like oil (and "data is the new oil"). If you're BP or some other huge company, you can extract all sorts of byproducts from your oil. But if you own a small backyard oil well... just sell the crude.
Same thing for operational BI. If you sell homemade candles, just make your candles. But if you operate a complex assembly line, warehouse, or a fleet of things, you probably want as much data as possible in order to track how things are going, bottlenecks where things are slowing down, and what is due for repair or replacement.
Yeah, exactly. And I think that's where people got excited, was they saw BP doing all this fancy stuff and thought they could to. But in reality, the just had some small backyard.
It's the same thing in traditional software/web development. Some big company posts an article about the reactive event-driven microservice architecture they developed to solve a very specific problem of their own, and then even the smallest engineering team dreams of implementing it at their tiny company.
Getting mad about paying for software makes even less sense in the vibe coding era. If you don't think the software is valuable enough to pay for - go ask Claude to make you one.
Maybe the pretense that it won't happen is what irritates people. And "you should have known better" won't make them any less irritated (probably the opposite).
I think what that tells you is that it's not possible to successfully operate that technology at lower pricing. Generally these companies aren't being greedy. They are trying to build sustainable businesses.
Of course, you do have the odd price gauger.
But generally, it's more a matter of can they make numbers work for the business to succeed.
I can never understand why people are up in arms when they are required to pay for something valuable.
In the world of open source, I think it's a bit different though. Imagine the open source project to be a community thing, where different people donate their blood, sweat and tears into. It's a project that not only did the company who owns the project invest into, but others did too.
Take SQLGlot, which we use (at Solid). The top contributors are from Tobiko, as you'd expect.
But there are many contributors who are not from Tobiko. Each such contributor made one or two commits, and probably most were immaterial to the project.
By contributing code/work, do they get a "share" of the project? A vote? Maybe each person gets a vote per commit they make, and those contributors who work for Tobiko pool their shares/votes for the interest of the company (which means Tobiko has 99% of the votes anyway)?
If SQLGlot was a company, and 99% of the shares were owned/controlled by one entity (Tobiko), and that entity made a decision to sell the company for X, is that ok?
In the world of companies and shares, there are concepts by which the 1% shouldn't be screwed in this process. Is that maybe what people are seeking here?
(I'm not justifying it btw - I think Tobiko made the right decision and have a ton of respect to the people there... just trying to rationalize through this)
People are up in arms because they create their own social contracts and expectations. Often these are naive due to lack of consideration about how software is made.
nevertheless, we try to respect people's social contracts to the extent that we can. As an opinionated software creator we regularly have to say no and let people down, to enable us to say yes and carry out our mission.
- adrian
Yeah, that's the other thing about it, I guess. I doesn't exactly matter where the feeling comes from; it's there, so these companies have to deal with it in some way. You could probably write a post that was the same thing, just inverted, where you said: "What did the companies expect to happen?" and it wouldn't be any more wrong.
I think it's not just having to pay, but also how much. People were quite happy to pay for DBT Cloud in the beginning, and even considered the pricing to be a bit low.
But then when you get a pricing analyst on board in order to determine "the right price" (i.e. what the market can bear / how to charge as much as possible), no one is happy. A price bump is one thing, but a multiplication is something else. They've invested themselves into a product/company, and it feels like they've been lied to. If we take that out of a business context and apply it to let's say a partnership or a marriage, that's the kind of stuff that results in divorce (because no one wants to be married to someone who tries to determine what the relationship can bear / how to take as much as possible for themselves, which is quite the opposite of how we should treat each other).
With the number of models we're running and their latest pricing, DBT Cloud would charge us more than what we're paying for Snowflake, which is nonsense.
And no one likes sudden changes either, whatever they are. Even small UI/UX changes generate tons of complaints, so it's not just about pricing.
It seems like in the end, the solution is just to ignore the haters, and things will eventually quiet down. 🤷♂️
That's another layer of the problem, but it's also the beauty of a competitive market. You can choose other options. Fivetran's pricing increases didn't go well for some people, so cheaper competitors came in.
This may be an opportunity to build a cheaper dbt Cloud - someone surely is thinking of that startup.
Here's a fun story in that vein:
At one point in time, we were using Zapier (which is quite expensive). Then we found a competitor which was much cheaper, and migrated everything. Then the competitor increased their price to match Zapier.
😖
Commented via my phone and posted it in the wrong spot by accident: https://benn.substack.com/p/stuff-costs-money/comment/153794281?utm_source=share&utm_medium=android&r=5b9smj
On the shareholder point, that's exactly how it effectively works. Small shareholders might technically get a vote, but in cases where they can't outvote the majority, they don't really. There is still typically a vote for these things, it's a formality, and those investors (and employees!) don't have a choice but to go along for the ride.
I get why someone who contributed to an open source thing might feel a little slighted by this, but they can see that this is the dynamic just as well as a customer. (And for the very very rare contributor that adds a lot, I'd guess that a lot of those people could get hired by the sponsoring company if they wanted to. Which isn't to say bad things here might not happen in this way, but I'd guess it's extremely rare, and not what drives the overall sentiment around all of this.)
And I'm kind of mixed on the pricing thing. Sure, it's unsavory, but, that's the deal. If someone's worried about it, don't use it when it's free. And I don't think that most companies want to price gouge people, and generally try to make it bearable for people - but, as I talked about here, every company has one real value. "Face eating leopard eats customer's face" isn't a good look for the leopard, but the customer probably should've seen it coming.
https://benn.substack.com/p/your-companys-values-will-be-used
It's surreal to have both been on the outside looking in and the inside looking out across the companies you name here. You hit the nail on the head my guy. Incentives and time constraints pretty much enforce this narrative.
Part of the story is that to materially increase TAM under high pressure timelines, the easiest path is to make it easier and easier to use your platform. That biases for an audience of data analysts and leaders that aren't open source die hards. Heck, most dbt/sqlmesh users use bigquery and snowflake by far which aren't open source in the slightest. Hint: they're the easiest to use ;)
Then there's the counterargument of, "What about databricks and confluent?" The big difference is they sell compute. And another big keyword difference in front of their open source tools: APACHE kafka, APACHE spark. This make it very high friction to haphazardly paywall major features.
I've always believed that one of the main reasons Databricks and Confluent (and to some degree, the hosted Airflow products) have been able to make this work is because they're hard to use. Spark is a huge pain, so it's not that hard to sell "Spark, without the pain." Tools like dbt and SQLMesh are victims of their own ease of use. You can run them yourself too easily. Had they been harder - to run dbt, you need to run all these docker containers, and they're finicky, and oh the whole thing is written in scala - I think the story might've been different.
To your point the hard part of all those is kubernetes, networking, and knowing how much compute is enough to make it all work well. Spark SQL and to a degree pyspark is easy enough to learn and use on your own. It's the managing infrastructure, which is tightly coupled to making the framework work well, that people pay for to outsource entirely. Funnily enough spotify uses scala with dbt and it's wacky.
Ah, right, yeah, that's what I meant. It's not running queries that's hard, but running the platform. In a sense, dbt in probably harder to use, day to day, than Spark. But running it is like, a six character script. Whereas running spark is....not that.
Lovely read!
If you give a mouse a cookie... https://a.co/d/6yD1Og0
Whenever you write in the style of option two, Iggy Pop starts playing in my head, and the voice reading the words to me is Ewan McGregor doing the "Choose life" monologue at the start of Trainspotting.
I'm not sure if I should feel this way or not, but this is probably the greatest thing anyone has ever said about this blog.
> "Man, everything really does becomes BI"
Man, who knew information and intelligence (which is derived from information) were so valuable? Wait... don't we all work in or with Information Technology? 🙃
The ironic thing is I'm not sure BI *is* all that valuable?
I guess it depends how we define BI. If we define it as "graphs and tables in a BI tool", it's probably much less valuable. Does Excel count as a BI tool? 🤔
But without information/intelligence about our business, we're flying blind. How do you even do A/B tests?
But I guess some businesses need it less than others. Our fraud team is constantly asking for more data/information from everywhere. Science is 100% based on data, and wouldn't exist without it. And intelligence agencies sure collect a lot of it! But that's a whole other type of business. 😅
But yeah, you probably don't need a lot of BI if you're selling homemade candles on Etsy. Or those famous jaffles.
I imagine that a graph charting the *actual* value of data/BI per business would very much be a hockey stick, with companies like Google & Meta being at the top, and then dropping fast from there.
Yeah, I think I'd group it like this:
- Basic reporting and stuff - very valuable. You gotta know how many people buy your stuff, and use this or that thing.
- "Analytics" and answering deeper questions - eeeeeh pretty iffy. Theoretically it's very useful, but it's really hard to extract useful stuff from messy data. If you're Google and have huge amounts of high potential data, sure. If you're a random SaaS company, not so much. The "insight" might be in there, but the amount of energy it takes to extract it is really high, to the point that I'm not sure it's worth it for most people.
- Specialized stuff like fraud reporting - very useful in the right pockets. But I think that's sort of the temptation. We see data being useful in pockets and assume it means "data is useful" rather than thinking "data was a useful way to solve this very specific problem."
Agreed. I think for many businesses, the most useful insight is merely "this product sells better than that one". And then perhaps "this specific demographic likes this specific product". But haven't we had this for ages? I mean, I'm pretty sure that even in the stone age, they could figure out something like "pistachios sell better than walnuts, but old people like walnuts better".
Is there some hidden insight beyond that? I'm not so sure. Unless you have the scale where small details become more significant opportunities, like let's say Amazon or Walmart.
It's kinda like oil (and "data is the new oil"). If you're BP or some other huge company, you can extract all sorts of byproducts from your oil. But if you own a small backyard oil well... just sell the crude.
Same thing for operational BI. If you sell homemade candles, just make your candles. But if you operate a complex assembly line, warehouse, or a fleet of things, you probably want as much data as possible in order to track how things are going, bottlenecks where things are slowing down, and what is due for repair or replacement.
Yeah, exactly. And I think that's where people got excited, was they saw BP doing all this fancy stuff and thought they could to. But in reality, the just had some small backyard.
It's the same thing in traditional software/web development. Some big company posts an article about the reactive event-driven microservice architecture they developed to solve a very specific problem of their own, and then even the smallest engineering team dreams of implementing it at their tiny company.
"We must use Kafka for our blog!"
Getting mad about paying for software makes even less sense in the vibe coding era. If you don't think the software is valuable enough to pay for - go ask Claude to make you one.
Oh you can't? Its difficult? Interesting that...
Yeah, I can sorta understand the frustration with the perceived bait and switch, but like, is it a bait and switch if everyone knows it'll happen??
Maybe the pretense that it won't happen is what irritates people. And "you should have known better" won't make them any less irritated (probably the opposite).
Another well written and timely post, thank you!
I think what that tells you is that it's not possible to successfully operate that technology at lower pricing. Generally these companies aren't being greedy. They are trying to build sustainable businesses.
Of course, you do have the odd price gauger.
But generally, it's more a matter of can they make numbers work for the business to succeed.