I agree that the cache concept is part of what makes things challenging, but also that we effectively have to cache what the state of the world was at most any point of time in the history of the company and maintain that robustly as the world changes around us.
While it is good enough for the billing system to know "who should I bill for a subscription today" the data side needs to know "who should have been billed on this day 2 years ago" and at any point in time in between as well as "who did I actually bill 2 years ago given the billing rules in place 2 years ago and what happened to that bill" in a way that most production applications do not need outside of perhaps an audit or transaction log that someone can inspect when needed.
I once had a project manager on a data warehousing project that said to me one day, "You know what, 90% of I.T. is just Copy and Paste". And he was right.
Data Hubs, Data Lakes, Data Warehouses, and lots of application integration is just copying data from one place to another. Data Warehousing may step it up with a Copy Paste Special Values variety.
The root of all these problems is actually the app-centricity of IT. Why is it that the analyst is left to translate some stressed-out, over-caffeinated software developer's data model to something interpretable by the business? Then we multiply up the difficulty by having to merge multiple application data models into one unified view for the business to understand their own data.
That is why we have the cache's and copy data around. As we try to solve that problem in a multi-step and manageable way.
Surely, an organisation should first develop the data model that supported the data needed to make the decisions they need to make. Then applications update the central model as and if required. It's the responsibility of the app to put their worthwhile data in the right place for the business.
In that alternate reality of no cache's are required, or copy and pasting data around for analytics. The data in the middle is always up to date and in the right shape for the business to interpret. Call it data-centric, data-first, or maybe even business-centric! They are paying the bill after all.
It'll never happen because of, mainly, misaligned incentives, but, one can dream.
Not a fan of analogies. Apps and Dashboards solve different problems here. As a user most apps I use solve things where I just need current state (my insta feed, bank balance, navigation etc), vs dashboard is almost always a comparison of states (plus some explainability).
Faster horses here wouldnt solve the problem, even with quantum computers ownership of an ETL would be unclear, may be the data engineer tested the change with the stakeholder who didn't own the change.
The hunt for the Holy Grail
I agree that the cache concept is part of what makes things challenging, but also that we effectively have to cache what the state of the world was at most any point of time in the history of the company and maintain that robustly as the world changes around us.
While it is good enough for the billing system to know "who should I bill for a subscription today" the data side needs to know "who should have been billed on this day 2 years ago" and at any point in time in between as well as "who did I actually bill 2 years ago given the billing rules in place 2 years ago and what happened to that bill" in a way that most production applications do not need outside of perhaps an audit or transaction log that someone can inspect when needed.
I once had a project manager on a data warehousing project that said to me one day, "You know what, 90% of I.T. is just Copy and Paste". And he was right.
Data Hubs, Data Lakes, Data Warehouses, and lots of application integration is just copying data from one place to another. Data Warehousing may step it up with a Copy Paste Special Values variety.
The root of all these problems is actually the app-centricity of IT. Why is it that the analyst is left to translate some stressed-out, over-caffeinated software developer's data model to something interpretable by the business? Then we multiply up the difficulty by having to merge multiple application data models into one unified view for the business to understand their own data.
That is why we have the cache's and copy data around. As we try to solve that problem in a multi-step and manageable way.
Surely, an organisation should first develop the data model that supported the data needed to make the decisions they need to make. Then applications update the central model as and if required. It's the responsibility of the app to put their worthwhile data in the right place for the business.
In that alternate reality of no cache's are required, or copy and pasting data around for analytics. The data in the middle is always up to date and in the right shape for the business to interpret. Call it data-centric, data-first, or maybe even business-centric! They are paying the bill after all.
It'll never happen because of, mainly, misaligned incentives, but, one can dream.
This is terrific and I've shared it with many folks over the last few days.
Not a fan of analogies. Apps and Dashboards solve different problems here. As a user most apps I use solve things where I just need current state (my insta feed, bank balance, navigation etc), vs dashboard is almost always a comparison of states (plus some explainability).
Faster horses here wouldnt solve the problem, even with quantum computers ownership of an ETL would be unclear, may be the data engineer tested the change with the stakeholder who didn't own the change.
Enjoyed reading man