Discussion about this post

User's avatar
Laurie @ Role Call's avatar

I spent some time doing contract work with a team of analysts that almost exclusively used AI for analysis. I don't mean that they asked AI to write complex SQL queries or occasionally asked it to label unstructured data. I mean that they uploaded the dataset and told it to "act as a senior analyst" and write the report, and then they copy and pasted the report into a doc and sent it.

They weren't being lazy or irresponsible, that's quite literally what we were instructed to do, and the workload didn't allow for anything much more 'bespoke' than that. I used AI to write Python scripts or Google Apps scripts or to help me finesse the wording of a tricky paragraph, but I refused to use it for wholesale analysis because every time I tried, the result was absolute nonsense. Or at least it was like 30% nonsense, which I personally believe is too much nonsense.

But what I realized over time is that it actually didn't matter. The people charged with reading the reports were mostly not reading them anyway, and the ones that did mostly didn't care if the data was even accurate, let alone if it was statistically significant or used a "rigorous methodology." They just wanted a stat they could bring to their boss to say "something I did worked" or to say "here's why we should do this idea I have."

Juha Korpela's avatar

Hmm, I don't know about this one. There is the implementation-focused techbro viewpoint on analytics: "I just build dashboards when asked, I don't know who uses them, and I don't care - probably they don't either", and then there's actual analytics. Say, in a manufacturing setting, for example, an analytical solution that optimizes the various inputs of a chemical process and provides the operator of a honking big machine with instructions and warnings is *hardly* the same as the stereotypical concept of a monthly board report that no-one reads.

It's fun to attack the latter as pointless busywork, and such busywork could (and maybe should) be easily replaced by an army of bots fishing for "insight", but that's not nearly the full picture. In real life the numbers and trends and metrics do actually often matter, some analytics teams do actually work together with their business counterparts, and if you (or your bot army) screw these up you lose their confidence in you and probably your job as well (or you end up assigned to dashboard busywork duties).

47 more comments...

No posts

Ready for more?