This is a great post. So many thoughtful problems to solve to make data "usable". I've experienced many of these working on data platforms at companies large and small.
I really like the planetary level reference. You mention the various layers and tools. BUT, I feel there is a human component that is missing. This is something I am actively working on. From my experience, many issues stem from the consumers of the data (via the consumption planet level) don't know what data is available (storage/transformation), nor where it comes from (ingestion) should they need to request a change.
The technology challenges are difficult problems to solve. It is a given we'll work on them as an industry. However, I feel there is a people problem involving collaboration and communication that is not being recognized. As I mentioned, I'm working on something that I felt a previous organization would have really benefited from.
I really enjoyed this post and look forward to learning from your future posts on DataOS, etc.
Thanks! And yeah, I think that's all fair. I remember someone making a comment a few months ago to the same effect, that a lot of our new tools solve technology problems but create people problems by fragmenting how things get done. I think it's a very smart point, and unfortunately, more often than not, a bad trade.
One other important reason for avoiding "thin" tooling from the user perspective - if I have to pay for ten tools (or get ten OS tools), I need to speak with ten companies, which means I need to get legal and procurement and CyberSec involved ten times, and there's a risk that I'll only get eight or five approved, which then leaves me short of the seamless experience promised by that "data OS" idea. So I'll always rather go with the one tool that gives me 80% of everything that I need than with the ten tools giving me 100% of what I need, but I might not get all ten for reasons that are outside of my control, and honestly, outside of the companies selling these tools...
Agreed. One possible solution to this is a data "app store," where you can install apps easily like you can apps on a phone. That's a cool idea, but I'm not sure it works because 1) some companies would still want to have everything vetted by IT/security/compliance, and 2) the economics of it get tricky. Mobile apps can be cheap because the market is so huge. I don't know if the same applies for data apps.
This is a great post. So many thoughtful problems to solve to make data "usable". I've experienced many of these working on data platforms at companies large and small.
I really like the planetary level reference. You mention the various layers and tools. BUT, I feel there is a human component that is missing. This is something I am actively working on. From my experience, many issues stem from the consumers of the data (via the consumption planet level) don't know what data is available (storage/transformation), nor where it comes from (ingestion) should they need to request a change.
The technology challenges are difficult problems to solve. It is a given we'll work on them as an industry. However, I feel there is a people problem involving collaboration and communication that is not being recognized. As I mentioned, I'm working on something that I felt a previous organization would have really benefited from.
I really enjoyed this post and look forward to learning from your future posts on DataOS, etc.
Thanks! And yeah, I think that's all fair. I remember someone making a comment a few months ago to the same effect, that a lot of our new tools solve technology problems but create people problems by fragmenting how things get done. I think it's a very smart point, and unfortunately, more often than not, a bad trade.
Thank you so much Benn for mentioning Zingg
One other important reason for avoiding "thin" tooling from the user perspective - if I have to pay for ten tools (or get ten OS tools), I need to speak with ten companies, which means I need to get legal and procurement and CyberSec involved ten times, and there's a risk that I'll only get eight or five approved, which then leaves me short of the seamless experience promised by that "data OS" idea. So I'll always rather go with the one tool that gives me 80% of everything that I need than with the ten tools giving me 100% of what I need, but I might not get all ten for reasons that are outside of my control, and honestly, outside of the companies selling these tools...
Agreed. One possible solution to this is a data "app store," where you can install apps easily like you can apps on a phone. That's a cool idea, but I'm not sure it works because 1) some companies would still want to have everything vetted by IT/security/compliance, and 2) the economics of it get tricky. Mobile apps can be cheap because the market is so huge. I don't know if the same applies for data apps.