MDS tools have been solving the same technical problems on their own. Notably, integrations. Data movement, orchestration, transformation, quality, bi tools... all need integrations for each data storage system... And each tool solved it from scratch, with no common ground. Let me explain, when I worked for a BI tool, we wrote integrations to read data from data storages. When I worked for a data movement tool, we wrote connectors to read and write data from data storages. When I worked for a data quality tool, we wrote connectors to execute queries (tests) on data storages. Didn't work for a data orchestration tool, but operators basically make API calls to external systems, notably data storages. The value is in the data, or metadata if the system is not a data storage per se. And the common thing, is making api calls to these systems: read data, write data, query data (execute tests), trigger actions (orchestration). If there was a common framework to interact with external systems, then I think data tool companies could be more ambitious with a clear vision focused around data. We are missing standards, and a vision like you said. A standard to connect to any external system and be able to read, write, query and trigger actions. Then we need user experiences on top of that. And probably also a common standard on what to build user experiences.
Also, any outsider can look at Matt Turck's Machine learning AI, and data landscape and think - how in the, what?, am I to make sense of this? There's no connections illustrated between all of these products, and even if there was, it would look like every conspiracy theorist's pictures with string mood board. Hard to make sense of any gathering point in the chaos indeed.
I audibly laughed when I got to " Anyway, speaking of the modern data stack." I think a fair amount of us that have been around the block a while can see exactly what you're highlighting so succinctly in this essay. Every conference I wind up going to, I keep thinking, aren't these the same sorts of problems we've been trying to "solve" for the last 20 some odd years I've been doing this, just re-packaged against the latest in the Gartner hype cycle? Having a Schelling point has to be something that is at the front of whatever evolution the modern data stack turns into. In a way, it's that "so what?" we're constantly grappling for in the majority of the solutions we're trying to deliver.
Two thoughts - I don’t know him personally - but I think Tristan is one of the type 1 founders - and I agree there are not many in the MDS ecosystem. I really respect what dbt has done around mission and ecosystem.
I think a phase II of this would be getting engineers / developers AND data scientists full buy in for the MDS or analytics stack - or whatever we are calling it now. If those groups actually started fully adopting snowflake and dbt and other tools in the space I think there is a huge second wave of growth. (I know some did - but I think a lot did not). Then the promise of a unified data platform with data you trust to run your business on could actually be more true - and that was always my north star goal. Plus the added benefit of all your data in one place to hook up to AI stuff is a bonus - I think…
Excellent critique. I've seen half a dozen different ways to construct a data architecture, each using a different set of vendors, a different approach to common problems, and a myriad of permutations of the MDS.
I found the use of the Schelling point as a "test" for the viability of a product / idea very compelling. It's telling that the MDS seems to be failing in that area. We must take a step back and reconsider.
I'm of the optimistic bent that data is a thing. That data at the enterprise scale, especially the global / multi-national enterprise scale, is a both a hard thing but a thing of immense value. Knowing the number of pumps in operation in an energy company is a good and necessary thing, and connecting that both to the maintenance records as well as to the revenue projections should be possible, maybe even easy. But for some reason our MDS in its current incarnation has struggled to solve the core challenges of the modern enterprise.
The journey must continue and this is a marker of a critical waypoint.
The costs are the part that never got better. I've been "On the Cloud" since 2009, and I've been six months away from the true cost savings for the entire time. Until the last few years people were fairly rational about it, but lately the dynamic has become accelerationist, with execs spending millions on "cloud native" snake oil and blundering into layoffs.
DHH's recent Cloud Exit of 37signals' app hosting has been impressive, and I think the cost differentials are even bigger for the data space.
Can the Vision be as simple as creating the "organizational nervous system"? To create the next generation of AI-integrated companies where data is the electricity flowing through the body and computation pathways. There are Type I's out there. But where?
Appreciate the perspective Benn...to finding more of those Type 1's, who fall in love with the problem being solved, and less so with their current solution on the table. 🍻
palantir's got a real well integrated vision for how all the pieces fit together in both an analytical and operational capacity
abhi sivasailam's described the most compelling one i've heard in an analytical capacity
clearly not designed to be a point solution but rather a series of pieces that come together to form a whole larger than the sum of its parts (as opposed to msft fabric)
MDS tools have been solving the same technical problems on their own. Notably, integrations. Data movement, orchestration, transformation, quality, bi tools... all need integrations for each data storage system... And each tool solved it from scratch, with no common ground. Let me explain, when I worked for a BI tool, we wrote integrations to read data from data storages. When I worked for a data movement tool, we wrote connectors to read and write data from data storages. When I worked for a data quality tool, we wrote connectors to execute queries (tests) on data storages. Didn't work for a data orchestration tool, but operators basically make API calls to external systems, notably data storages. The value is in the data, or metadata if the system is not a data storage per se. And the common thing, is making api calls to these systems: read data, write data, query data (execute tests), trigger actions (orchestration). If there was a common framework to interact with external systems, then I think data tool companies could be more ambitious with a clear vision focused around data. We are missing standards, and a vision like you said. A standard to connect to any external system and be able to read, write, query and trigger actions. Then we need user experiences on top of that. And probably also a common standard on what to build user experiences.
Also, any outsider can look at Matt Turck's Machine learning AI, and data landscape and think - how in the, what?, am I to make sense of this? There's no connections illustrated between all of these products, and even if there was, it would look like every conspiracy theorist's pictures with string mood board. Hard to make sense of any gathering point in the chaos indeed.
My search turned up 2021's - how many products can one person know, have skills in, and even further, know now to fit it all together? https://mattturck.com/wp-content/uploads/2021/12/2021-MAD-Landscape-v3.pdf)
I audibly laughed when I got to " Anyway, speaking of the modern data stack." I think a fair amount of us that have been around the block a while can see exactly what you're highlighting so succinctly in this essay. Every conference I wind up going to, I keep thinking, aren't these the same sorts of problems we've been trying to "solve" for the last 20 some odd years I've been doing this, just re-packaged against the latest in the Gartner hype cycle? Having a Schelling point has to be something that is at the front of whatever evolution the modern data stack turns into. In a way, it's that "so what?" we're constantly grappling for in the majority of the solutions we're trying to deliver.
Two thoughts - I don’t know him personally - but I think Tristan is one of the type 1 founders - and I agree there are not many in the MDS ecosystem. I really respect what dbt has done around mission and ecosystem.
I think a phase II of this would be getting engineers / developers AND data scientists full buy in for the MDS or analytics stack - or whatever we are calling it now. If those groups actually started fully adopting snowflake and dbt and other tools in the space I think there is a huge second wave of growth. (I know some did - but I think a lot did not). Then the promise of a unified data platform with data you trust to run your business on could actually be more true - and that was always my north star goal. Plus the added benefit of all your data in one place to hook up to AI stuff is a bonus - I think…
Stands up.
Starts slow clap.
Excellent critique. I've seen half a dozen different ways to construct a data architecture, each using a different set of vendors, a different approach to common problems, and a myriad of permutations of the MDS.
I found the use of the Schelling point as a "test" for the viability of a product / idea very compelling. It's telling that the MDS seems to be failing in that area. We must take a step back and reconsider.
I'm of the optimistic bent that data is a thing. That data at the enterprise scale, especially the global / multi-national enterprise scale, is a both a hard thing but a thing of immense value. Knowing the number of pumps in operation in an energy company is a good and necessary thing, and connecting that both to the maintenance records as well as to the revenue projections should be possible, maybe even easy. But for some reason our MDS in its current incarnation has struggled to solve the core challenges of the modern enterprise.
The journey must continue and this is a marker of a critical waypoint.
The costs are the part that never got better. I've been "On the Cloud" since 2009, and I've been six months away from the true cost savings for the entire time. Until the last few years people were fairly rational about it, but lately the dynamic has become accelerationist, with execs spending millions on "cloud native" snake oil and blundering into layoffs.
DHH's recent Cloud Exit of 37signals' app hosting has been impressive, and I think the cost differentials are even bigger for the data space.
Can the Vision be as simple as creating the "organizational nervous system"? To create the next generation of AI-integrated companies where data is the electricity flowing through the body and computation pathways. There are Type I's out there. But where?
Appreciate the perspective Benn...to finding more of those Type 1's, who fall in love with the problem being solved, and less so with their current solution on the table. 🍻
palantir's got a real well integrated vision for how all the pieces fit together in both an analytical and operational capacity
abhi sivasailam's described the most compelling one i've heard in an analytical capacity
clearly not designed to be a point solution but rather a series of pieces that come together to form a whole larger than the sum of its parts (as opposed to msft fabric)