I think you'd get a kick out of and draw parallels from DHH's piece here, which is a rant on the 'circus empire' around Kubernetes and dev tools world.
The weird thing, in the data world, it feels similar, but almost inverted. It's the rebels (ie, startups) who keep finding more and more complicated ways to construct what probably should be a four or five piece Lego. Of course, huge companies aren't really doing anything to simplify it either, so I'm not sure what stops the ratchet. The economy, I suppose.
At the end of the day, the MDS reduces friction in data development, but nothing really helps address the fundamental issues. For one, analytics value is extraordinarily hard to measure, and most places aren't doing any kind of attribution modeling to capture revenue / cost savings. And two, the business frequently asks for assets that don't align with their actual goals. Because analytics is "accessible" (anyone can pick up a spreadsheet), we frequently see requirements sessions where otherwise very smart business people make very silly / bad requests because they attempt to translate a domain they understand well into one they don't (data and analytics).
Analytics for me was a more democratic way to make decisions based on data instead of politics, including the measure of value through metrics and A/B testing.
Many data engineers unfortunately have not applied these practices to their own choices.
I've been going over decisions in our org for the last year, and many just seem to be the highest paid person's opinion (HiPPO), or constant agitation without supporting metrics, even denying tests that have already been done. It's gotten worse as we've added headcount.
We make a lot of money based on data (advertising) but don't drive our internal decision making with the same rigor. It's a weird blend of a data-driven app but not data-driven app development.
Ahh, gotcha. That definitely seems fairly common, and it's interesting that data is so readily applied to operational problems, but more reluctantly to analytical ones. There's kind of an interesting question in there to me about what causes the difference.
Agreed the tooling isn't bad, though I think it certainly makes it easy for us to make a lot of messes too. While some conglomerate selling semi-bundled tools won't solve that, they might enforce some order, which seems like a thing we could use. Given how long we've been trying to get these processes right, I don't have that much faith that we're gonna figure that out on our own.
There aren't many of these co's out there but constellation has built an amazing business doing just this, the berkshire of software - https://www.csisoftware.com/
I think you'd get a kick out of and draw parallels from DHH's piece here, which is a rant on the 'circus empire' around Kubernetes and dev tools world.
https://world.hey.com/dhh/they-re-rebuilding-the-death-star-of-complexity-4fb5d08d
The weird thing, in the data world, it feels similar, but almost inverted. It's the rebels (ie, startups) who keep finding more and more complicated ways to construct what probably should be a four or five piece Lego. Of course, huge companies aren't really doing anything to simplify it either, so I'm not sure what stops the ratchet. The economy, I suppose.
always an entertaining take on the data community with johnoliveresque delivery.
one day, benn dot substack will have the budget for this. one day.
https://www.youtube.com/watch?v=mDk_R__Relw
At the end of the day, the MDS reduces friction in data development, but nothing really helps address the fundamental issues. For one, analytics value is extraordinarily hard to measure, and most places aren't doing any kind of attribution modeling to capture revenue / cost savings. And two, the business frequently asks for assets that don't align with their actual goals. Because analytics is "accessible" (anyone can pick up a spreadsheet), we frequently see requirements sessions where otherwise very smart business people make very silly / bad requests because they attempt to translate a domain they understand well into one they don't (data and analytics).
Our tooling ain't bad. Our processes suck.
Analytics for me was a more democratic way to make decisions based on data instead of politics, including the measure of value through metrics and A/B testing.
Many data engineers unfortunately have not applied these practices to their own choices.
Curious what you mean?
I've been going over decisions in our org for the last year, and many just seem to be the highest paid person's opinion (HiPPO), or constant agitation without supporting metrics, even denying tests that have already been done. It's gotten worse as we've added headcount.
We make a lot of money based on data (advertising) but don't drive our internal decision making with the same rigor. It's a weird blend of a data-driven app but not data-driven app development.
Ahh, gotcha. That definitely seems fairly common, and it's interesting that data is so readily applied to operational problems, but more reluctantly to analytical ones. There's kind of an interesting question in there to me about what causes the difference.
Agreed the tooling isn't bad, though I think it certainly makes it easy for us to make a lot of messes too. While some conglomerate selling semi-bundled tools won't solve that, they might enforce some order, which seems like a thing we could use. Given how long we've been trying to get these processes right, I don't have that much faith that we're gonna figure that out on our own.
There aren't many of these co's out there but constellation has built an amazing business doing just this, the berkshire of software - https://www.csisoftware.com/
I'm sure there are lots of reasons for it, but it's honestly kind of weird to me that this isn't a much more common thing.
The wordplay and delivery of this piece was masterful. Refreshing lens on the upcoming consolidation of services.
Thanks! Gotta distract from the fact that the piece is about economics, niche software businesses, and minor financial transactions.