I am, like you, an "old" and took part in the Data Science Revolution of 2010 - now.
Basically, for about a decade, you could pull together a team of Phd’s (or those unambiguously smart enough to get a STEM Phd), and you could point them at some data and give them a business outcome or goal, and they could lift things by absurdly massive amounts, generally driving tens of millions of value per year with a team that cost only $1-2M. Bump conversion by 20-30%, drop costs by 20-50% by targeting things or using resources more intelligently, really dial in what factors were actually most important for driving various outcomes via modeling, segment customers in much more predictive ways, and so on. It was an arms race, of sorts - business is a competitive landscape, and those deltas are too big to ignore.
Sure, the amount of lift has leveled off after a decade plus of mining that arbitrage - but that level of optimization is now table stakes!
I really don't see how businesses operating on vibes can compete with such a high degree of data ingestion, processing, and modeling being the optimization table stakes today. Won't their competitors eat them alive, by pulling this very apparent and available lever?
Still, definitely some interesting food for thought, and I appreciated the read and the challenge to that viewpoint. Increasingly, maybe you have to get both of these right to succeed - good data handling and modeling may be table stakes, but so too might be cultural fluency and having the right vibe in your customer touch points, marketing, and sentiment.
150% of 1M is still less than 50% of 10M. But that 5M can probably be optimized to over 15M with data.
It basically depends who captures the market first, and then data becomes most useful once the market is saturated. It's almost like data is a fight over the scraps (it often feels like that IMO). We're like scavengers or junkyard miners: we find value after the fact. 😅 Which is still immensely useful, don't get me wrong. Operational optimization alone is a goldmine (which can hardly be done without data), which Amazon has chiefly demonstrated.
Another way to put it would be: do markets get disrupted because someone found some data that no one else has or knows about? No. Usually it's because someone tried a new vibe, or because of a significant innovation. Neither of these have much to do with data (although in some rare cases, data IS that vibe/innovation, like in Moneyball).
It's like in a gold rush: what matters most at first is digging. But once the craze has passed, then data becomes the most useful tool (and hopefully helps us to find a few new veins). Maybe the craze is just the initial data collection phase (but data people are usually not crazed diggers, because we'd rather sit back, takes notes, and measure things in order to find a guaranteed spot to dig down, like modern dowsers). But that initial phase will still crown winners & losers whether or not data people were involved.
Another analogy would be politics: data may not be the most helpful tool in the primaries (where vibe/appeal matters most), but it is absolutely necessary in order to win the election (short of having an undefeatable vibe, like Obama).
Yeah, I think I would more or less agree with Marco here, that data has a ton of value in optimization, and that optimization can be worth a ton of money at a certain scale. But that's not true for a lot of companies (ie, their data just isn't that useful), and I'd dispute that that many companies actually can throw mathematicians at their data and get anything particularly valuable out of it.
This is now pretty old, but it's basically the question here: https://benn.substack.com/p/do-data-driven-companies-win To me, the company you bet on in this one isn't the data one. Which suggests to me that data isn't table stakes at all, until you get to something of truly staggering scale.
But, either way, I don't necessarily think that being vibes-driven is capital-R Right. It's more that it's the fad of the moment. It could certainly implode too.
Someone already said it, but a have a "vibe" to say it again. Vibe is for revolution, data is for evolution. For if we wait for the data points to confirm everything before we act, we likely kill agency and adventure. But once agency and adventure create paths worth threading, it helps to become data-driven, specifically, for know which specific path to thread.
Yeah, I think that's ideally true, though practically, sort of difficult? When everyone was trying to be data-driven, everyone always talked about how important culture was in that. You need to have a data-driven *culture,* etc etc. So yeah, the best way to run things is probably with some amount of balance - start with vibes, use data later, roughly - but companies seem to take on the character of one or the other.
I felt like this question, "Will us abandon data-driven decision making and join the vibe-side" is too early to tell.
First: AI company as a industry is still in untested water a.k.a the capital cycle hasn't turned yet. It's good and dandy to say "No no, we build that feature by just using our taste". It's a different story where that particular feature, which even adopted by million of users, cost a lot of money and you forget to put a tracker on it (which, uh, could you vibe all the way to bankruptcy?)
Second (kinda related to 1st one tho): the most competitive industry always resort to be an "operational excellence" company, which basically combining data-driven + clear agency. If old industry that already been here (banking, logistic, etc) are run through data, what make young industry like AI be so different they can skip the basic?
That's fair, and I don't know the vibes people will necessarily win. (there will probably be a bunch of vibes-driven politicians running US elections in 2026; I have no idea if they'll do better than the data-driven ones or not). There certainly could be a big backlash to it, or it could end up being seen as a fad that led us all astray.
The counter-argument to that, though, is that it could take a long time? And fads, being self-sustaining on the way up, could become pretty popular before they ever prove themselves out one way or the other. (I'd argue that data itself has followed that arc. We got enamored with quantifying stuff, with fairly mixed results. Though there are surely some very big success stories, there are also lots of companies that invested a lot in it for what'd I'd say are pretty disappointing results.)
The way I think of the rule of quantification in the context of Simon Caulkin's "What gets measured gets managed" is that measurement serves bureaucracy, which serves scaling and automation. Not a bad thing, but not always the desired end goal. Measurements capture a static sense of priorities and value, whereas culture requires updating these with fresh notions regularly. Technology exacerbates the problem both by allowing us to effortlessly double down on potentially ossified metrics, and doing so at a pace we have trouble adapting to often leading to the enforcement of the rules becoming the goal in itself.
Yeah, and that seems to be what a lot of these newer companies are trying to avoid. I don't think that they're necessarily explicit in that, nor do they have some rigorous theory behind why they choose vibes over metrics. But there's definitely a tilt towards "founder mode / move fast / follow your own tastes / etc," and that all seems culturally aligned with vibe-driven decision making over data-driven decision making. And it probably doesn't help that data - and especially giant binders of "metrics" - feels bureacracy-coded.
I understand that's a joke, but also, yeah, kinda? Like, it is *cooler*, in the sense that maverick tom cruise was definitely cooler than by-the-book ice man. Did he kill people? maybe. But he was the one everyone wanted to be too. Like, everyone wants to be Rick Rubin because it seems cooler to just intuit stuff than it does to meticulously math it.
Yes, AND... its the mavericks who open up new spaces that are then quantized and made safe and efficient.
Lincoln Beachey was an early aerobatic pilot who figured out how to do loop the loops by understanding stalling. He retired when a few other pilots were killed trying to replicate his feats. A short while after, others were doing multiple loops and he went back to performing. He was killed in 1915 off of Crissy Field in San Francisco when the wings tore off his experimental home-made plane. Today, airplanes systems have built in systems and metrics encapsulating his learnings.
Oh for sure; all of this is probably somewhat cyclical (though the vibe-y renegades will always be cooler than the accountants, I suspect). As it relates to data stuff, my specific point is a little more of the moment, in that it seems like 1) there are new, vibey way to do things, and 2) that has some natural, inherent appeal, so 3) seems like that may well become a new fad. But, fads fade, including that one, I suspect.
Agreed. Dragging this out a bit, but the "vibey" trend could be seen to have been taking hold for awhile. Consider the move round 2011 to Agile, MVP, and other iterative methods of development and planning. As the pace of change and requirements becomes faster, the ability to make small moves and reassess the landscape becomes more valuable.
In the olden days one could build an ERP system which would take 5 years and end up being nothing anyone exactly wanted but good enough nevertheless. This was rational when the investment and momentum required was huge. In much of today's world that's no longer useful or practical.
Another place to look for metaphors for this might be Stewart Brand's pace layers.
do you hear anything from econ or finance types about the government statistical releases we’ve skipped or stopped collecting/publishing this year? I’m curious how they think about running on vibes, how their info sourcing strategies are shifting. like what are Bloomberg’s product managers working on right now?
edit to add: remembering the prediction market data is probably a big part of this already
No, not really? Most of what I heard was just people leaning on private data sources (ADP over BLS data, etc), but those big private sources were already pretty tightly watched, so I'm not sure it was a big change. Plus, I suspect people act differently when they assume the shutdown is temporary. If people thought it was a longer thing, that might change.
On prediction markets, my sense is that they aren't big enough to be meaningful on that sort of stuff yet? Like, the volume isn't so much smaller than traditional financial markets, so they seem to be more day trading wall street bets types (or, you know, insiders) and less giant fund managers. So it's not clear how much they represent some true market view.
As a data engineer myself, previously a data analyst, I can feel it at some level, but I haven't seen anybody describing it this way. Really excellent text (and awesome storytelling in the beginning.)
Oh this is interesting. The end I think was good too, about how we we come to love this moment too.
Another friend sent me this, which I thought was also a good. Less about what is dead or live, or popular or out of fashion, but more about how it is us that's aging, and not the world.
Gymnastics scoring has always been a mess. It's just a different sort of mess than it was in the past. They idea that they could craft a definitive set of rules about what is ultimately a subjective system is amusing. They are great - nearly miraculous - athletes, but having your success determined by judges has to suck.
The idea that either data or design/taste - but not both - should guide product and business decisions is two sides of the same myopic coin.
Don't some accountants become CFOs? I'd argue CFOs play a large, but often unnoticed, role in the success of a firm.
All the rage about data science in the recent past was mostly coming from people with a financial interest in data science become the next big thing. Much like AI right now.
On gymnastics scoring - yeah, and I certainly don't know what's better. Though one thought I had (that I cut from the piece) was you could imagine a world where the rules are all a little fuzzy. Like, the precise out of bounds lines and time limits exist because we need heuristics for scoring it, but it's not crazy to say "judge the routine in its entirety, and if you get further away from the middle, I'll judge you progressively more negatively, rather than make this one millimeter over the line result in some big penalty."
On either/or - I agree that you'd probably want a balance, but I'm not sure that's culturally possible. Companies (and people) seem to take on a particular character, and it seems practically very hard to sometimes be craft and vibe focused, and other times very data oriented.
On accountants - oh, for sure, and it's not that accountants aren't useful or valuable. But it's that they're a bit more mechanical. Though there are a few companies that might say "our competitive advantage is our CFO," that only seems to happen for a particular type of company (or one of ridiculous scale).
On data science - I don't think I'm that cynical about it? Or like, sure, they had a vested interest in data science working, because that was a career path or strategic bet that they made. But I think they were genuine in that belief.
I think the balance would be something along the lines of:
- Try a vibe
- Measure the results with data
It's basically a feedback loop like so many others. But without the feedback/data, you're flying blind. Sports and video games are full of short feedback loops, and that's how people can get better so quickly.
How do you become a better shot if you can't see where your bullets are landing? And A/B tests are basically just a way to try something on a portion of your customer base, rather than on the entire thing. Because you're not sure until you have data. But the vibe people seem to be confident enough to try without data, and believe that they're hitting bullseyes.
And then of course they will try to say that it works because it worked for one of them. But that's like saying playing the lottery works, because there's always a winner. But if you look at the data, you put your money elsewhere.
tl;dr: Yeah, sure, go ahead and try stuff. *But always measure success*.
Maybe it's because culturally, in many cases, the data *is* the try.
Like if you make music, you can't really A/B test a song. So you go with your gut.
Maybe it's the result of a new generation raised on views & likes. In the past, we did not have this data/signal/measure to use as a feedback loop (save for things like billboard charts), and the feedback wasn't so fast. And maybe they're right. Maybe that's good enough (for certain things).
It could also be that someone who is at least trying is in a better position to become successful than someone who isn't. So it's worth investing in them and hoping for the best. Who knows what they might come up with tomorrow? Maybe they'll hit a large gold vein.
I think you're right: at this speed, the math does start to change.
And it kinda makes sense. If you can dig faster than you can measure the amount of gold in the soil... just keep digging until you hit something big rather than worrying about the scraps you may have left behind.
That gold analogy makes a lot of sense to me. Where, yeah, a lot of what data is supposed to do is let you test - do a small thing, see if it works, scale it. But if you can do the full scale version pretty quickly...maybe just do that? (Though I realize that's a messy analogy. Like, could you do that with music? Maybe? If you wanted to test a song, you could try 10 versions of it to small groups, and then push the good one out to everyone. But if you can make songs really quickly, is that right? Should you just constantly put out records and not worry that some are busts and one might be a hit? I don't think we know that yet. When the "cost of content goes to 0," does that mean the winners are people who put out the most hits? Or the people who have the highest hit rate?)
On the A/B testing point, though, I think that's a place where data has actually performed the worst. A lot of companies got infatuated with it, and started shipping features because they had statistically significant improvements. But outside of huge companies where tiny optimizations really matter, I think most of those improvements were fiddling with edges, and took a ton of time and effort - to both build the things and to run the tests - to find functionally meaningless gains. But we did it because data was the cool thing to do. For most companies, though, I think they would've been better off making decisions by asking "is this obviously better?" (Which, may still be somewhat informed by data. People might complain about something and use it way, way more, and you could conclude that''s obviously better. But I don't think it was smart to ignore people complaining about it because there was 0.8 percent lift in engagement among this and that user cohort.)
When I was at dbt, the closest thing I did to data driven analysis for contributing new features to open source was, "I remember this guy brought it up 3 times with me. It sounds acute and chronic. I guess I'll fix it." Then lots of people 👍 the PR and all was merry. No fancy ceremony with PMs and engineering leaders and designers. Just a human moment.
There's a charm to this that feels analogous to those viral Tik Toks in 360p of candid moments. People can smell at a gutteral level when a person just wants to connect with another person.
Side note: data driven analysis now has a reputation for "confirmation bias + public speaking skills" rather than the truth.
Yeah, a big part of this whole idea to me is that every CEO basically makes decisions because they heard some story from a customer. That's obviously an exaggeration, but not that much - where, at the root of every high conviction "we have to do this" bet, it's almost always something that someone told them when they were really happy or mad or whatever. Data is great and all, and maybe it's "right," but the personal, emotional stuff is what we really listen to. And if we can get something that's that at scale (which, to me, really what "vibes" are), then that's what people are probably going to pay attention to.
If we can somehow measure or classify/segment the vibes... wouldn't that be data? Maybe we can't optimize it with A/B tests, but we can still leverage/wield/direct it, or use it as part of our overall strategy.
Or basically, how do we answer the question "what's hot right now?". By asking Mugatu or our niece who is very active on social media, or by measuring it? Isn't the latter exactly what Davidowitz did in order to write Everybody Lies? But yes, maybe sometimes the answer is so obvious that we don't even really need data. Like in a gold rush: just dig, man! But the even more important question is "what will be hot tomorrow?", and I'm not sure how much data can really help with that. It's not a crystal ball.
Maybe data science & engineering just needs to catch on (in both senses). Maybe our senses just got dulled due to looking at numbers for too long.
Sentiment analysis has been like a holy grail for a long time. Maybe we just need to move beyond that "simple" goal into data 2.0 now (i.e. vibe analysis). And one day there might be a data 3.0 (just like the Web 1.0, 2.0, and 3.0).
If we could collect & classify this data, then we could integrate it into the rest of our data architecture. Maybe it's something that looks more like a graph database, rather than a table (whether structured or unstructured). That would be a novel way of working for most data scientists & engineers. How do you perform advanced aggregations on a graph? Or how do you query or visualize the intersection/union/difference between segments? Or how do you even segment in the first place?
And then maybe we could figure out a way to turn those vibes into metrics & dimensions. Or just visualize them completely differently. It could be like a bubble network chart where the axes are the dimensions we care about, and the bubbles are the metrics (or vice versa). So we would be limited to the three dimensions that we can display physically (and possibly a 4th dimension to show how the graph changes over time, or according to some other dimension).
And then if we can visualize how the vibes shift over time, we may even be able to predict where they are going (perhaps as simply as watching water flow).
And it would probably be best visualized in VR. With gloves. Because data is cool.
Or maybe some people just intuitively know (or think they know) where things are headed, and they do not need any of this fancy science & engineering. Or maybe the extroverts just want to reclaim the field (I know, I know, this is a poor generalization).
So, yeah, "vibes" is just data too, at least in a literal sense. But it's not necessarily "math" (that we understand). Like, to use the gymnastics example, you could score it in two ways:
1. Quantify the routine into a bunch of elements and scores and deductions, and add it all up on a really complicated spreadsheet. This is basically turning an unstructured thing into structured metrics, and all of that.
2. Watching a ton of routines, learning what's good and what's bad, and then watching all the gymnasts perform and decide which one you think is best. There is no quantification here; it's just like, an expert review.
The problem with 2 is that it's very fallible. Humans make mistakes; they can't watch every routine; they're biased; they forget stuff; etc. But it's not that much of a stretch to imagine what LLMs do as creating a kind of vibey average. They remember their training and weight all of it. Each new thing they "watch" nudges their perceptions in some subtle way. It's kind of like a person with a very good memory watching all of it, and then being a judge.
That's all very rough, but it strikes me as what we really want, in principle. The quantification of stuff is kind of a necessary and loss-y evil, because it's the only way we have to do it. But I'm not sure it's what we want to do, if there was an alternative.
The most fun contrarian take I've read this week!
But isn't "performance" the ultima ratio regum?
I am, like you, an "old" and took part in the Data Science Revolution of 2010 - now.
Basically, for about a decade, you could pull together a team of Phd’s (or those unambiguously smart enough to get a STEM Phd), and you could point them at some data and give them a business outcome or goal, and they could lift things by absurdly massive amounts, generally driving tens of millions of value per year with a team that cost only $1-2M. Bump conversion by 20-30%, drop costs by 20-50% by targeting things or using resources more intelligently, really dial in what factors were actually most important for driving various outcomes via modeling, segment customers in much more predictive ways, and so on. It was an arms race, of sorts - business is a competitive landscape, and those deltas are too big to ignore.
Sure, the amount of lift has leveled off after a decade plus of mining that arbitrage - but that level of optimization is now table stakes!
I really don't see how businesses operating on vibes can compete with such a high degree of data ingestion, processing, and modeling being the optimization table stakes today. Won't their competitors eat them alive, by pulling this very apparent and available lever?
Still, definitely some interesting food for thought, and I appreciated the read and the challenge to that viewpoint. Increasingly, maybe you have to get both of these right to succeed - good data handling and modeling may be table stakes, but so too might be cultural fluency and having the right vibe in your customer touch points, marketing, and sentiment.
150% of 1M is still less than 50% of 10M. But that 5M can probably be optimized to over 15M with data.
It basically depends who captures the market first, and then data becomes most useful once the market is saturated. It's almost like data is a fight over the scraps (it often feels like that IMO). We're like scavengers or junkyard miners: we find value after the fact. 😅 Which is still immensely useful, don't get me wrong. Operational optimization alone is a goldmine (which can hardly be done without data), which Amazon has chiefly demonstrated.
Another way to put it would be: do markets get disrupted because someone found some data that no one else has or knows about? No. Usually it's because someone tried a new vibe, or because of a significant innovation. Neither of these have much to do with data (although in some rare cases, data IS that vibe/innovation, like in Moneyball).
It's like in a gold rush: what matters most at first is digging. But once the craze has passed, then data becomes the most useful tool (and hopefully helps us to find a few new veins). Maybe the craze is just the initial data collection phase (but data people are usually not crazed diggers, because we'd rather sit back, takes notes, and measure things in order to find a guaranteed spot to dig down, like modern dowsers). But that initial phase will still crown winners & losers whether or not data people were involved.
Another analogy would be politics: data may not be the most helpful tool in the primaries (where vibe/appeal matters most), but it is absolutely necessary in order to win the election (short of having an undefeatable vibe, like Obama).
Data is for evolution, vibes are for revolution?
Yeah, I think I would more or less agree with Marco here, that data has a ton of value in optimization, and that optimization can be worth a ton of money at a certain scale. But that's not true for a lot of companies (ie, their data just isn't that useful), and I'd dispute that that many companies actually can throw mathematicians at their data and get anything particularly valuable out of it.
This is now pretty old, but it's basically the question here: https://benn.substack.com/p/do-data-driven-companies-win To me, the company you bet on in this one isn't the data one. Which suggests to me that data isn't table stakes at all, until you get to something of truly staggering scale.
But, either way, I don't necessarily think that being vibes-driven is capital-R Right. It's more that it's the fad of the moment. It could certainly implode too.
Someone already said it, but a have a "vibe" to say it again. Vibe is for revolution, data is for evolution. For if we wait for the data points to confirm everything before we act, we likely kill agency and adventure. But once agency and adventure create paths worth threading, it helps to become data-driven, specifically, for know which specific path to thread.
Great read, as always.
Yeah, I think that's ideally true, though practically, sort of difficult? When everyone was trying to be data-driven, everyone always talked about how important culture was in that. You need to have a data-driven *culture,* etc etc. So yeah, the best way to run things is probably with some amount of balance - start with vibes, use data later, roughly - but companies seem to take on the character of one or the other.
I felt like this question, "Will us abandon data-driven decision making and join the vibe-side" is too early to tell.
First: AI company as a industry is still in untested water a.k.a the capital cycle hasn't turned yet. It's good and dandy to say "No no, we build that feature by just using our taste". It's a different story where that particular feature, which even adopted by million of users, cost a lot of money and you forget to put a tracker on it (which, uh, could you vibe all the way to bankruptcy?)
Second (kinda related to 1st one tho): the most competitive industry always resort to be an "operational excellence" company, which basically combining data-driven + clear agency. If old industry that already been here (banking, logistic, etc) are run through data, what make young industry like AI be so different they can skip the basic?
That's fair, and I don't know the vibes people will necessarily win. (there will probably be a bunch of vibes-driven politicians running US elections in 2026; I have no idea if they'll do better than the data-driven ones or not). There certainly could be a big backlash to it, or it could end up being seen as a fad that led us all astray.
The counter-argument to that, though, is that it could take a long time? And fads, being self-sustaining on the way up, could become pretty popular before they ever prove themselves out one way or the other. (I'd argue that data itself has followed that arc. We got enamored with quantifying stuff, with fairly mixed results. Though there are surely some very big success stories, there are also lots of companies that invested a lot in it for what'd I'd say are pretty disappointing results.)
Well put.
The way I think of the rule of quantification in the context of Simon Caulkin's "What gets measured gets managed" is that measurement serves bureaucracy, which serves scaling and automation. Not a bad thing, but not always the desired end goal. Measurements capture a static sense of priorities and value, whereas culture requires updating these with fresh notions regularly. Technology exacerbates the problem both by allowing us to effortlessly double down on potentially ossified metrics, and doing so at a pace we have trouble adapting to often leading to the enforcement of the rules becoming the goal in itself.
Yeah, and that seems to be what a lot of these newer companies are trying to avoid. I don't think that they're necessarily explicit in that, nor do they have some rigorous theory behind why they choose vibes over metrics. But there's definitely a tilt towards "founder mode / move fast / follow your own tastes / etc," and that all seems culturally aligned with vibe-driven decision making over data-driven decision making. And it probably doesn't help that data - and especially giant binders of "metrics" - feels bureacracy-coded.
It's so much cooler to fly a plane without all of those instruments.
I understand that's a joke, but also, yeah, kinda? Like, it is *cooler*, in the sense that maverick tom cruise was definitely cooler than by-the-book ice man. Did he kill people? maybe. But he was the one everyone wanted to be too. Like, everyone wants to be Rick Rubin because it seems cooler to just intuit stuff than it does to meticulously math it.
Yes, AND... its the mavericks who open up new spaces that are then quantized and made safe and efficient.
Lincoln Beachey was an early aerobatic pilot who figured out how to do loop the loops by understanding stalling. He retired when a few other pilots were killed trying to replicate his feats. A short while after, others were doing multiple loops and he went back to performing. He was killed in 1915 off of Crissy Field in San Francisco when the wings tore off his experimental home-made plane. Today, airplanes systems have built in systems and metrics encapsulating his learnings.
Oh for sure; all of this is probably somewhat cyclical (though the vibe-y renegades will always be cooler than the accountants, I suspect). As it relates to data stuff, my specific point is a little more of the moment, in that it seems like 1) there are new, vibey way to do things, and 2) that has some natural, inherent appeal, so 3) seems like that may well become a new fad. But, fads fade, including that one, I suspect.
Agreed. Dragging this out a bit, but the "vibey" trend could be seen to have been taking hold for awhile. Consider the move round 2011 to Agile, MVP, and other iterative methods of development and planning. As the pace of change and requirements becomes faster, the ability to make small moves and reassess the landscape becomes more valuable.
In the olden days one could build an ERP system which would take 5 years and end up being nothing anyone exactly wanted but good enough nevertheless. This was rational when the investment and momentum required was huge. In much of today's world that's no longer useful or practical.
Another place to look for metaphors for this might be Stewart Brand's pace layers.
do you hear anything from econ or finance types about the government statistical releases we’ve skipped or stopped collecting/publishing this year? I’m curious how they think about running on vibes, how their info sourcing strategies are shifting. like what are Bloomberg’s product managers working on right now?
edit to add: remembering the prediction market data is probably a big part of this already
No, not really? Most of what I heard was just people leaning on private data sources (ADP over BLS data, etc), but those big private sources were already pretty tightly watched, so I'm not sure it was a big change. Plus, I suspect people act differently when they assume the shutdown is temporary. If people thought it was a longer thing, that might change.
On prediction markets, my sense is that they aren't big enough to be meaningful on that sort of stuff yet? Like, the volume isn't so much smaller than traditional financial markets, so they seem to be more day trading wall street bets types (or, you know, insiders) and less giant fund managers. So it's not clear how much they represent some true market view.
As a data engineer myself, previously a data analyst, I can feel it at some level, but I haven't seen anybody describing it this way. Really excellent text (and awesome storytelling in the beginning.)
On your points about "thinking things were better back then" it reminded me of this incredible YT video about nostalgia i saw recently.
check it out here
https://www.youtube.com/watch?v=_RZ-w4hU8Dw
first 12 minutes are awesome, will prob write ab this soon
Oh this is interesting. The end I think was good too, about how we we come to love this moment too.
Another friend sent me this, which I thought was also a good. Less about what is dead or live, or popular or out of fashion, but more about how it is us that's aging, and not the world.
https://www.joanwestenberg.com/everything-is-dead-and-we-killed-it/
Random responses:
Interesting read, as always.
Gymnastics scoring has always been a mess. It's just a different sort of mess than it was in the past. They idea that they could craft a definitive set of rules about what is ultimately a subjective system is amusing. They are great - nearly miraculous - athletes, but having your success determined by judges has to suck.
The idea that either data or design/taste - but not both - should guide product and business decisions is two sides of the same myopic coin.
Don't some accountants become CFOs? I'd argue CFOs play a large, but often unnoticed, role in the success of a firm.
All the rage about data science in the recent past was mostly coming from people with a financial interest in data science become the next big thing. Much like AI right now.
On gymnastics scoring - yeah, and I certainly don't know what's better. Though one thought I had (that I cut from the piece) was you could imagine a world where the rules are all a little fuzzy. Like, the precise out of bounds lines and time limits exist because we need heuristics for scoring it, but it's not crazy to say "judge the routine in its entirety, and if you get further away from the middle, I'll judge you progressively more negatively, rather than make this one millimeter over the line result in some big penalty."
On either/or - I agree that you'd probably want a balance, but I'm not sure that's culturally possible. Companies (and people) seem to take on a particular character, and it seems practically very hard to sometimes be craft and vibe focused, and other times very data oriented.
On accountants - oh, for sure, and it's not that accountants aren't useful or valuable. But it's that they're a bit more mechanical. Though there are a few companies that might say "our competitive advantage is our CFO," that only seems to happen for a particular type of company (or one of ridiculous scale).
On data science - I don't think I'm that cynical about it? Or like, sure, they had a vested interest in data science working, because that was a career path or strategic bet that they made. But I think they were genuine in that belief.
I think the balance would be something along the lines of:
- Try a vibe
- Measure the results with data
It's basically a feedback loop like so many others. But without the feedback/data, you're flying blind. Sports and video games are full of short feedback loops, and that's how people can get better so quickly.
How do you become a better shot if you can't see where your bullets are landing? And A/B tests are basically just a way to try something on a portion of your customer base, rather than on the entire thing. Because you're not sure until you have data. But the vibe people seem to be confident enough to try without data, and believe that they're hitting bullseyes.
And then of course they will try to say that it works because it worked for one of them. But that's like saying playing the lottery works, because there's always a winner. But if you look at the data, you put your money elsewhere.
tl;dr: Yeah, sure, go ahead and try stuff. *But always measure success*.
Maybe it's because culturally, in many cases, the data *is* the try.
Like if you make music, you can't really A/B test a song. So you go with your gut.
Maybe it's the result of a new generation raised on views & likes. In the past, we did not have this data/signal/measure to use as a feedback loop (save for things like billboard charts), and the feedback wasn't so fast. And maybe they're right. Maybe that's good enough (for certain things).
It could also be that someone who is at least trying is in a better position to become successful than someone who isn't. So it's worth investing in them and hoping for the best. Who knows what they might come up with tomorrow? Maybe they'll hit a large gold vein.
I think you're right: at this speed, the math does start to change.
And it kinda makes sense. If you can dig faster than you can measure the amount of gold in the soil... just keep digging until you hit something big rather than worrying about the scraps you may have left behind.
That gold analogy makes a lot of sense to me. Where, yeah, a lot of what data is supposed to do is let you test - do a small thing, see if it works, scale it. But if you can do the full scale version pretty quickly...maybe just do that? (Though I realize that's a messy analogy. Like, could you do that with music? Maybe? If you wanted to test a song, you could try 10 versions of it to small groups, and then push the good one out to everyone. But if you can make songs really quickly, is that right? Should you just constantly put out records and not worry that some are busts and one might be a hit? I don't think we know that yet. When the "cost of content goes to 0," does that mean the winners are people who put out the most hits? Or the people who have the highest hit rate?)
On the A/B testing point, though, I think that's a place where data has actually performed the worst. A lot of companies got infatuated with it, and started shipping features because they had statistically significant improvements. But outside of huge companies where tiny optimizations really matter, I think most of those improvements were fiddling with edges, and took a ton of time and effort - to both build the things and to run the tests - to find functionally meaningless gains. But we did it because data was the cool thing to do. For most companies, though, I think they would've been better off making decisions by asking "is this obviously better?" (Which, may still be somewhat informed by data. People might complain about something and use it way, way more, and you could conclude that''s obviously better. But I don't think it was smart to ignore people complaining about it because there was 0.8 percent lift in engagement among this and that user cohort.)
When I was at dbt, the closest thing I did to data driven analysis for contributing new features to open source was, "I remember this guy brought it up 3 times with me. It sounds acute and chronic. I guess I'll fix it." Then lots of people 👍 the PR and all was merry. No fancy ceremony with PMs and engineering leaders and designers. Just a human moment.
There's a charm to this that feels analogous to those viral Tik Toks in 360p of candid moments. People can smell at a gutteral level when a person just wants to connect with another person.
Side note: data driven analysis now has a reputation for "confirmation bias + public speaking skills" rather than the truth.
Yeah, a big part of this whole idea to me is that every CEO basically makes decisions because they heard some story from a customer. That's obviously an exaggeration, but not that much - where, at the root of every high conviction "we have to do this" bet, it's almost always something that someone told them when they were really happy or mad or whatever. Data is great and all, and maybe it's "right," but the personal, emotional stuff is what we really listen to. And if we can get something that's that at scale (which, to me, really what "vibes" are), then that's what people are probably going to pay attention to.
If we can somehow measure or classify/segment the vibes... wouldn't that be data? Maybe we can't optimize it with A/B tests, but we can still leverage/wield/direct it, or use it as part of our overall strategy.
Or basically, how do we answer the question "what's hot right now?". By asking Mugatu or our niece who is very active on social media, or by measuring it? Isn't the latter exactly what Davidowitz did in order to write Everybody Lies? But yes, maybe sometimes the answer is so obvious that we don't even really need data. Like in a gold rush: just dig, man! But the even more important question is "what will be hot tomorrow?", and I'm not sure how much data can really help with that. It's not a crystal ball.
Maybe data science & engineering just needs to catch on (in both senses). Maybe our senses just got dulled due to looking at numbers for too long.
Sentiment analysis has been like a holy grail for a long time. Maybe we just need to move beyond that "simple" goal into data 2.0 now (i.e. vibe analysis). And one day there might be a data 3.0 (just like the Web 1.0, 2.0, and 3.0).
If we could collect & classify this data, then we could integrate it into the rest of our data architecture. Maybe it's something that looks more like a graph database, rather than a table (whether structured or unstructured). That would be a novel way of working for most data scientists & engineers. How do you perform advanced aggregations on a graph? Or how do you query or visualize the intersection/union/difference between segments? Or how do you even segment in the first place?
And then maybe we could figure out a way to turn those vibes into metrics & dimensions. Or just visualize them completely differently. It could be like a bubble network chart where the axes are the dimensions we care about, and the bubbles are the metrics (or vice versa). So we would be limited to the three dimensions that we can display physically (and possibly a 4th dimension to show how the graph changes over time, or according to some other dimension).
And then if we can visualize how the vibes shift over time, we may even be able to predict where they are going (perhaps as simply as watching water flow).
And it would probably be best visualized in VR. With gloves. Because data is cool.
Or maybe some people just intuitively know (or think they know) where things are headed, and they do not need any of this fancy science & engineering. Or maybe the extroverts just want to reclaim the field (I know, I know, this is a poor generalization).
So, yeah, "vibes" is just data too, at least in a literal sense. But it's not necessarily "math" (that we understand). Like, to use the gymnastics example, you could score it in two ways:
1. Quantify the routine into a bunch of elements and scores and deductions, and add it all up on a really complicated spreadsheet. This is basically turning an unstructured thing into structured metrics, and all of that.
2. Watching a ton of routines, learning what's good and what's bad, and then watching all the gymnasts perform and decide which one you think is best. There is no quantification here; it's just like, an expert review.
The problem with 2 is that it's very fallible. Humans make mistakes; they can't watch every routine; they're biased; they forget stuff; etc. But it's not that much of a stretch to imagine what LLMs do as creating a kind of vibey average. They remember their training and weight all of it. Each new thing they "watch" nudges their perceptions in some subtle way. It's kind of like a person with a very good memory watching all of it, and then being a judge.
That's all very rough, but it strikes me as what we really want, in principle. The quantification of stuff is kind of a necessary and loss-y evil, because it's the only way we have to do it. But I'm not sure it's what we want to do, if there was an alternative.