This is a great post. So many thoughtful problems to solve to make data "usable". I've experienced many of these working on data platforms at companies large and small.
I really like the planetary level reference. You mention the various layers and tools. BUT, I feel there is a human component that is missing. This is something I am actively working on. From my experience, many issues stem from the consumers of the data (via the consumption planet level) don't know what data is available (storage/transformation), nor where it comes from (ingestion) should they need to request a change.
The technology challenges are difficult problems to solve. It is a given we'll work on them as an industry. However, I feel there is a people problem involving collaboration and communication that is not being recognized. As I mentioned, I'm working on something that I felt a previous organization would have really benefited from.
I really enjoyed this post and look forward to learning from your future posts on DataOS, etc.
One other important reason for avoiding "thin" tooling from the user perspective - if I have to pay for ten tools (or get ten OS tools), I need to speak with ten companies, which means I need to get legal and procurement and CyberSec involved ten times, and there's a risk that I'll only get eight or five approved, which then leaves me short of the seamless experience promised by that "data OS" idea. So I'll always rather go with the one tool that gives me 80% of everything that I need than with the ten tools giving me 100% of what I need, but I might not get all ten for reasons that are outside of my control, and honestly, outside of the companies selling these tools...
The intergalactic data stack
This is a great post. So many thoughtful problems to solve to make data "usable". I've experienced many of these working on data platforms at companies large and small.
I really like the planetary level reference. You mention the various layers and tools. BUT, I feel there is a human component that is missing. This is something I am actively working on. From my experience, many issues stem from the consumers of the data (via the consumption planet level) don't know what data is available (storage/transformation), nor where it comes from (ingestion) should they need to request a change.
The technology challenges are difficult problems to solve. It is a given we'll work on them as an industry. However, I feel there is a people problem involving collaboration and communication that is not being recognized. As I mentioned, I'm working on something that I felt a previous organization would have really benefited from.
I really enjoyed this post and look forward to learning from your future posts on DataOS, etc.
Thank you so much Benn for mentioning Zingg
One other important reason for avoiding "thin" tooling from the user perspective - if I have to pay for ten tools (or get ten OS tools), I need to speak with ten companies, which means I need to get legal and procurement and CyberSec involved ten times, and there's a risk that I'll only get eight or five approved, which then leaves me short of the seamless experience promised by that "data OS" idea. So I'll always rather go with the one tool that gives me 80% of everything that I need than with the ten tools giving me 100% of what I need, but I might not get all ten for reasons that are outside of my control, and honestly, outside of the companies selling these tools...