I spent some time doing contract work with a team of analysts that almost exclusively used AI for analysis. I don't mean that they asked AI to write complex SQL queries or occasionally asked it to label unstructured data. I mean that they uploaded the dataset and told it to "act as a senior analyst" and write the report, and then they copy and pasted the report into a doc and sent it.
They weren't being lazy or irresponsible, that's quite literally what we were instructed to do, and the workload didn't allow for anything much more 'bespoke' than that. I used AI to write Python scripts or Google Apps scripts or to help me finesse the wording of a tricky paragraph, but I refused to use it for wholesale analysis because every time I tried, the result was absolute nonsense. Or at least it was like 30% nonsense, which I personally believe is too much nonsense.
But what I realized over time is that it actually didn't matter. The people charged with reading the reports were mostly not reading them anyway, and the ones that did mostly didn't care if the data was even accurate, let alone if it was statistically significant or used a "rigorous methodology." They just wanted a stat they could bring to their boss to say "something I did worked" or to say "here's why we should do this idea I have."
What are you talking about? Data analysts are cautious because if you’re wrong once, stakeholders stop trusting you. When I’ve tried to let copilot act as a senior analyst, it spits out nonsense so I use it to write subqueries and individual cells, which it’s quite good at.
Also, like there’s real world consequences to being wrong? Just on a daily basis, things stop adding up. And, since I work in product growth, the product stops growing lol
It sounds like your work is much more directly connected to outcomes than the work on this team was, which is enviable! I suspect most analysts on that team would have loved for their work to have more real world impact.
Apologies — I meant to reply to Ben, not you. But, yeah, I mean, sadly, lots of consultants bullshit their way through life. A long term embedded data team can’t plausibly get things wrong all the time and not see consequences
What you said though really hits the nail on the head for me. The reason I care about data accuracy is that I think data should be (one of the things) used to make actual decisions that have actual real world consequences. When it's just window dressing, it genuinely kind of doesn't matter if it's accurate or not, which I find depressing.
Isn’t that everything in life and business? A loop of gathering information to guide the next loop (that gathers more info)?
Analysis is literally this. And as these things become more widely adopted the tentacles of those loops will reach into every single department, regardless of our concern.
The loops don’t scare me. The people who don’t want this do. They concern me greatly. How did they learn anything? A feedback loop. An imperfect one that was refined over time that eventually built habits, beliefs, and intuition (we call this “taste” now).
It’s how any child learns anything.
AI won’t eat your lunch; it’ll just make it far less interesting than the kid who brought a full course meal to their grade school cafeteria and then shared it with others.
And we all know what it’s like to show up with a soggy ham sandwich.
I wasn't really a believer in this for a while, but more and more, I struggle coming up good counter-arguments. To your point, it's less about how AI is doing something divine; it's more that I don't think I do anything all that special. I ask myself questions, I look stuff up, I repeat.
People periodically say stuff like, ah, but, the AI doesn't know. Which is true, but that feels very stuck in the "AI is ChatGPT" mindset, where it's just prompt and response. But we seem pretty far beyond that at this point. Once a model gets good enough to deal with fairly ambiguous stuff, and once a harness gets good enough at looping itself without spiraling out of control, it's hard to see where that stops.
I think about chaos and how a lot of people see that as scary; they avoid anything that feels, smells, and tastes like it.
But then the few say: Creativity and innovation require the context of chaos to even begin. Out of nothing we create something. Out of the storm delivers a real gem.
The question is one of control. Anyone trying to "control" AI has already taken a backseat. This is so counter-intuitive for most that they won't even consider their built-in bias to control because it's based on fear.
But the best entrepreneurs and inventors across time and space and history never really saw it that way. They saw it as part of their canvas. Their plaything. I think AI is the same. Let the mf just go.
I'm not sure I'm quite so optimistic about it, but I'm also pretty sure there's nothing to do to stop it at this point. And at least as far as creativity goes (to the extent that a blog about dumb tech nonsense is "creative"), smashing a bunch of weird chaotic ideas together in your head is the only reliable way I've found to come up with anything. So, we at least have a lot of that these days.
"Once a model gets good enough to deal with fairly ambiguous stuff..."
I'm not sure there is any reason to believe this will happen. An LLM is not good at ambiguity. It is not good at nuance. This isn't surprising because their very architecture is not built in a way to handle this. It cannot reason.
Genuine question - why do you say that? At least in some domains, I find them very good at that.
Like, one of the things I've realized as I use them is I talk to them at the wrong level. It's very hard not to give them specific instructions about what you want and what they're supposed to do, but increasingly, I've realized you're better off just telling them what you're thinking. If you're stuck, say that. If you have a problem and don't know how to solve it, say that. If someone's complaining about this thing and you don't know what to do about it, say that. Trying to translate your ideas into instructions - ie, trying to resolve the ambiguity, rather than giving it to it - seems like what you're supposed to do. But, man, they seem to do just fine when I don't do that.
I think saying OpenAI and Anthropic are two of the world's most valuable companies needs a huge caveat; this is only because a relatively tiny number of a certain class of investors. A class that is wrong far more often than it is right. We'll see.
Fair, though, if someone gave you chance to take a big short on either of them, would you do it? The smart answer may well be yes, but, I'm not sure I'm taking that bet.
Probably not, because I don't like risks like that. Not to pick on them specifically either. Other pre-IPO companies are / have been similar. Arguably Databricks is overvalued. Snowflake came out of the gate way too hot when it IPOd (I was there).
Analysis is testable in the same way that programs are testable. You can test that you are using the statistics correctly. But which statistics are you using and why?
AI isn't good at figuring out what program needs to be written in the same way that it isn't good at encapsulating all the knowledge that goes into data analysis.
Sure, insofar as analysis is the math. But I think analysis is a lot messier than that in two ways? First, it's really hard to judge what is "right." Is playing the numbers right if the result is bad? In cases where you don't really know the numbers (which is most situations), how do you even know if you're playing the numbers? And second, even if you can say if the math is right, a lot of analysis is just judgement. It's closer to a legal argument than a mathematical proof. And what makes a good legal argument? There's some amount of symbolic logic, but a lot of it is fuzzy grey areas.
Hmm, I don't know about this one. There is the implementation-focused techbro viewpoint on analytics: "I just build dashboards when asked, I don't know who uses them, and I don't care - probably they don't either", and then there's actual analytics. Say, in a manufacturing setting, for example, an analytical solution that optimizes the various inputs of a chemical process and provides the operator of a honking big machine with instructions and warnings is *hardly* the same as the stereotypical concept of a monthly board report that no-one reads.
It's fun to attack the latter as pointless busywork, and such busywork could (and maybe should) be easily replaced by an army of bots fishing for "insight", but that's not nearly the full picture. In real life the numbers and trends and metrics do actually often matter, some analytics teams do actually work together with their business counterparts, and if you (or your bot army) screw these up you lose their confidence in you and probably your job as well (or you end up assigned to dashboard busywork duties).
I think that our current framing for junior analyst positions will go away once the right scaffolding is in place. I'm less confident that Sr+ goes away, except for roles that didn't really need effective analysts.
And so, if you work in credit analytics, finance, internal consulting, or something where accuracy is a critical to quality value, then I expect the need to not change very quickly.
Is the valuation, and the hype, coming from the fact that no one has a sane counterpoint that is able to scale the way these cowboy companies are? In the absence of which, FOMO wins.
Essentially open the Pandora's box of "give it full control" and let the end users handle the rest.
It's true to a point. How many flawed analyses and massaged metrics and nonsense graphs have business users generated and accepted over the years before the AI slop came for us all? Why wouldn't they want to pay the low low cost of having AI spit out some stuff so you can call yourself "data-driven"?
If the bar is in hell, why the hell not? No need to clutch pearls in the attention economy.
I spent some time doing contract work with a team of analysts that almost exclusively used AI for analysis. I don't mean that they asked AI to write complex SQL queries or occasionally asked it to label unstructured data. I mean that they uploaded the dataset and told it to "act as a senior analyst" and write the report, and then they copy and pasted the report into a doc and sent it.
They weren't being lazy or irresponsible, that's quite literally what we were instructed to do, and the workload didn't allow for anything much more 'bespoke' than that. I used AI to write Python scripts or Google Apps scripts or to help me finesse the wording of a tricky paragraph, but I refused to use it for wholesale analysis because every time I tried, the result was absolute nonsense. Or at least it was like 30% nonsense, which I personally believe is too much nonsense.
But what I realized over time is that it actually didn't matter. The people charged with reading the reports were mostly not reading them anyway, and the ones that did mostly didn't care if the data was even accurate, let alone if it was statistically significant or used a "rigorous methodology." They just wanted a stat they could bring to their boss to say "something I did worked" or to say "here's why we should do this idea I have."
What are you talking about? Data analysts are cautious because if you’re wrong once, stakeholders stop trusting you. When I’ve tried to let copilot act as a senior analyst, it spits out nonsense so I use it to write subqueries and individual cells, which it’s quite good at.
Also, like there’s real world consequences to being wrong? Just on a daily basis, things stop adding up. And, since I work in product growth, the product stops growing lol
It sounds like your work is much more directly connected to outcomes than the work on this team was, which is enviable! I suspect most analysts on that team would have loved for their work to have more real world impact.
Apologies — I meant to reply to Ben, not you. But, yeah, I mean, sadly, lots of consultants bullshit their way through life. A long term embedded data team can’t plausibly get things wrong all the time and not see consequences
What you said though really hits the nail on the head for me. The reason I care about data accuracy is that I think data should be (one of the things) used to make actual decisions that have actual real world consequences. When it's just window dressing, it genuinely kind of doesn't matter if it's accurate or not, which I find depressing.
As you said: “Claude is a bunch of loops.”
Isn’t that everything in life and business? A loop of gathering information to guide the next loop (that gathers more info)?
Analysis is literally this. And as these things become more widely adopted the tentacles of those loops will reach into every single department, regardless of our concern.
The loops don’t scare me. The people who don’t want this do. They concern me greatly. How did they learn anything? A feedback loop. An imperfect one that was refined over time that eventually built habits, beliefs, and intuition (we call this “taste” now).
It’s how any child learns anything.
AI won’t eat your lunch; it’ll just make it far less interesting than the kid who brought a full course meal to their grade school cafeteria and then shared it with others.
And we all know what it’s like to show up with a soggy ham sandwich.
I wasn't really a believer in this for a while, but more and more, I struggle coming up good counter-arguments. To your point, it's less about how AI is doing something divine; it's more that I don't think I do anything all that special. I ask myself questions, I look stuff up, I repeat.
People periodically say stuff like, ah, but, the AI doesn't know. Which is true, but that feels very stuck in the "AI is ChatGPT" mindset, where it's just prompt and response. But we seem pretty far beyond that at this point. Once a model gets good enough to deal with fairly ambiguous stuff, and once a harness gets good enough at looping itself without spiraling out of control, it's hard to see where that stops.
I think about chaos and how a lot of people see that as scary; they avoid anything that feels, smells, and tastes like it.
But then the few say: Creativity and innovation require the context of chaos to even begin. Out of nothing we create something. Out of the storm delivers a real gem.
The question is one of control. Anyone trying to "control" AI has already taken a backseat. This is so counter-intuitive for most that they won't even consider their built-in bias to control because it's based on fear.
But the best entrepreneurs and inventors across time and space and history never really saw it that way. They saw it as part of their canvas. Their plaything. I think AI is the same. Let the mf just go.
I'm not sure I'm quite so optimistic about it, but I'm also pretty sure there's nothing to do to stop it at this point. And at least as far as creativity goes (to the extent that a blog about dumb tech nonsense is "creative"), smashing a bunch of weird chaotic ideas together in your head is the only reliable way I've found to come up with anything. So, we at least have a lot of that these days.
"Once a model gets good enough to deal with fairly ambiguous stuff..."
I'm not sure there is any reason to believe this will happen. An LLM is not good at ambiguity. It is not good at nuance. This isn't surprising because their very architecture is not built in a way to handle this. It cannot reason.
Genuine question - why do you say that? At least in some domains, I find them very good at that.
Like, one of the things I've realized as I use them is I talk to them at the wrong level. It's very hard not to give them specific instructions about what you want and what they're supposed to do, but increasingly, I've realized you're better off just telling them what you're thinking. If you're stuck, say that. If you have a problem and don't know how to solve it, say that. If someone's complaining about this thing and you don't know what to do about it, say that. Trying to translate your ideas into instructions - ie, trying to resolve the ambiguity, rather than giving it to it - seems like what you're supposed to do. But, man, they seem to do just fine when I don't do that.
I think saying OpenAI and Anthropic are two of the world's most valuable companies needs a huge caveat; this is only because a relatively tiny number of a certain class of investors. A class that is wrong far more often than it is right. We'll see.
Fair, though, if someone gave you chance to take a big short on either of them, would you do it? The smart answer may well be yes, but, I'm not sure I'm taking that bet.
Probably not, because I don't like risks like that. Not to pick on them specifically either. Other pre-IPO companies are / have been similar. Arguably Databricks is overvalued. Snowflake came out of the gate way too hot when it IPOd (I was there).
Raise your hand if you knew one of the 3 "soundtracks to my life" links would open Griff on Spotify.
a man must have a code
Analysis is testable in the same way that programs are testable. You can test that you are using the statistics correctly. But which statistics are you using and why?
AI isn't good at figuring out what program needs to be written in the same way that it isn't good at encapsulating all the knowledge that goes into data analysis.
Sure, insofar as analysis is the math. But I think analysis is a lot messier than that in two ways? First, it's really hard to judge what is "right." Is playing the numbers right if the result is bad? In cases where you don't really know the numbers (which is most situations), how do you even know if you're playing the numbers? And second, even if you can say if the math is right, a lot of analysis is just judgement. It's closer to a legal argument than a mathematical proof. And what makes a good legal argument? There's some amount of symbolic logic, but a lot of it is fuzzy grey areas.
https://open.spotify.com/track/6FMIVQPZg9cmMY8hPxAacD?si=FB1fKzPNRjGmvqRnshCteQ
https://www.youtube.com/watch?v=-1F7vaNP9w0
it's not looking good out there: https://theshamblog.com/an-ai-agent-published-a-hit-piece-on-me/
OMG... the time to enshittification - breathtaking.
This car is going very fast
Hmm, I don't know about this one. There is the implementation-focused techbro viewpoint on analytics: "I just build dashboards when asked, I don't know who uses them, and I don't care - probably they don't either", and then there's actual analytics. Say, in a manufacturing setting, for example, an analytical solution that optimizes the various inputs of a chemical process and provides the operator of a honking big machine with instructions and warnings is *hardly* the same as the stereotypical concept of a monthly board report that no-one reads.
It's fun to attack the latter as pointless busywork, and such busywork could (and maybe should) be easily replaced by an army of bots fishing for "insight", but that's not nearly the full picture. In real life the numbers and trends and metrics do actually often matter, some analytics teams do actually work together with their business counterparts, and if you (or your bot army) screw these up you lose their confidence in you and probably your job as well (or you end up assigned to dashboard busywork duties).
I think that our current framing for junior analyst positions will go away once the right scaffolding is in place. I'm less confident that Sr+ goes away, except for roles that didn't really need effective analysts.
And so, if you work in credit analytics, finance, internal consulting, or something where accuracy is a critical to quality value, then I expect the need to not change very quickly.
Is the valuation, and the hype, coming from the fact that no one has a sane counterpoint that is able to scale the way these cowboy companies are? In the absence of which, FOMO wins.
Essentially open the Pandora's box of "give it full control" and let the end users handle the rest.
It's true to a point. How many flawed analyses and massaged metrics and nonsense graphs have business users generated and accepted over the years before the AI slop came for us all? Why wouldn't they want to pay the low low cost of having AI spit out some stuff so you can call yourself "data-driven"?
If the bar is in hell, why the hell not? No need to clutch pearls in the attention economy.
I dunno man... I feel like there is a balance to be found between good structure and YOLOing with AI: https://www.loom.com/share/a6e7051bb22642e5b9de4642b60fe67a (shameless plug, sorry)
And speaking with analysts myself, more and more are picking up on this direction.