Well, if makes you feel any better, I have no idea what I'm talking about, so this is probably something like the anti-galloway index for data predictions.
Yes, but, I'd also be kind of fun to answer questions with stuff like "Metrics will surge this week, for those who deserve it. There will be opportunities to be found. Patience is required, but know when to act with urgency."
I'm finally caught up on my reading :-) - and perfectly timed for me. I got a 1000+ SAMPLE XML file this week - the real thing will be tens or hundreds of thousands of lines that will be arriving daily via sftp... and yes this is the only option... so ya. Thats the "bad place" where many of us live.
More seriously, the Avocado tool at Yammer/Microsoft is closest to what I think a data mesh is. When it scaled out to MS as a whole, individual groups created data store accounts in the tool and published what were effectively individually managed data products and connected them via ETL jobs.
It had a huge dependency graph that I once plotted in Graphviz as an example of managing chaos, but since each piece was managed by an individual team it didn't matter. It scaled to thousands of jobs per hour without much dev involvement.
That's sort of how I feel about the whole thing, to be honest. The solution is pretty simple - give different groups of people their own folders, basically - but that doesn't make people excited, so we created The Data Mesh to make it sound more complicated and cool.
Reading too much of you does not make my career look very promising as a new product manager for a data team lol
Well, if makes you feel any better, I have no idea what I'm talking about, so this is probably something like the anti-galloway index for data predictions.
https://galloway-index.webflow.io/
Data horoscopes is a great analogy… I hope it doesn’t make us (the industry) data fortune tellers.
Yes, but, I'd also be kind of fun to answer questions with stuff like "Metrics will surge this week, for those who deserve it. There will be opportunities to be found. Patience is required, but know when to act with urgency."
I'm finally caught up on my reading :-) - and perfectly timed for me. I got a 1000+ SAMPLE XML file this week - the real thing will be tens or hundreds of thousands of lines that will be arriving daily via sftp... and yes this is the only option... so ya. Thats the "bad place" where many of us live.
Yeah, but I'm sure it's all perfectly formatted and orderly and tells lots of nice, satisfying stories.
Hahaha. Your funny.
Now do this but for data mesh, please!
I would, but I would first have to understand what data mesh is.
It's a Data Warehouse that has exceeded Dunbar's Number.
Which is bad, because personal Dunbar's number is like, 4.
More seriously, the Avocado tool at Yammer/Microsoft is closest to what I think a data mesh is. When it scaled out to MS as a whole, individual groups created data store accounts in the tool and published what were effectively individually managed data products and connected them via ETL jobs.
It had a huge dependency graph that I once plotted in Graphviz as an example of managing chaos, but since each piece was managed by an individual team it didn't matter. It scaled to thousands of jobs per hour without much dev involvement.
That's sort of how I feel about the whole thing, to be honest. The solution is pretty simple - give different groups of people their own folders, basically - but that doesn't make people excited, so we created The Data Mesh to make it sound more complicated and cool.
This was very funny
Hear me out: design for the trough analysis.
It's almost a real thing! https://archive.nytimes.com/krugman.blogs.nytimes.com/2011/06/23/a-fit-of-peaks/