No, of course it's not a fad. Business has always and will always whatever data is useful and obtainable in decision making. Even in fashion. But this certainly doesn't mean every decision is going to be based on a dashboard in Tableau derived from set of tables in Snowflake.
As for stocks - no amount of data will *ever* allow you to reliably and consistently predict the stock price 24 hours into the future for anything. Not unless humans are completely out of the loop. Completely. But then who cares?
Like, sure, but isn't that basically saying "nothing changes and nothing has ever changed?" I agree that there will always be degrees of data; there will always be some desire to be data-driven; stock prices will never be perfectly predictable; etc. But it seems pretty clear that, over the last 20-30 years, we've tried to introduce data into a lot more decisions than we did before. My question is mostly about that. The rough assumption seems to be that it's a one-way ratchet. But could it not go in reverse?
"But it seems pretty clear that, over the last 20-30 years, we've tried to introduce data into a lot more decisions than we did before."
Yes, definitely, because we've been able to more easily capture more data about people and processes. Whether or not it has all been useful is debatable. Yes, I think some of this will go in reverse because it's not all useful. Witness the debate/growing dismissal of marketing attribution data.
I really doubt all the data collected in an effort to predict stock prices is useful, but the payoffs are so high when someone momentarily hits that people will keep trying.
Exactly, I think that's my question. There was a lot of hype around the big explosion of stuff in the last 20-30 years, and part of that hype was selling that as this inevitable march upwards. And to your point about the dismissal of marketing attribution, that's a good example of "the vibes have shifted" thing I was trying to say: It seems like there's now a lot more skepticism of that stuff - and to some degree, interest in other stuff - than there was a few years ago.
(On stock prediction, yeah, the point of whole stock thing wasn't really to say that we can *predict* stock prices. It was just that there are philosophical regime changes. For a long time, there has been - and still is, really - a regime that says stocks should be roughly valued based on these various financial metrics. Matt Levine's point was perhaps that regime will recede, even though going back to vibes and animal spirits seems like a regression. And my point is that maybe the "data driven" regime recedes to something else, even if that also feels like a regression.)
On stocks, I think in the long run they are valued primarily on the earnings the company produces. In the short run, *anything* can drive the price. And I think this has become a lot more volatile with the rise of easy retail investing. But I think vibe investing has been around since stocks have been traded, too. There's just more of it and it's more obvious.
It never went away. Ask probably any executive. I can't think of many management roles where some intuition isn't part of decision making because we can't measure everything.
I think I agree with both of these points? "Founder Mode" or whatever never went away, but people had less permission to operate that way (partly because the push was for execs to be data driven). The shift to me is that the founder mode style is becoming acceptable, and the data driven style is feeling more dated. (Which isn't necessarily a statement about what's good, or what should be, or any of that. It's just a statement about what's in style.)
"The shift to me is that the founder mode style is becoming acceptable, and the data driven style is feeling more dated."
I interpreted Founder Mode as ensuring that you're in on the key decisions, rather than handing them off to other people. Brian Chesky gave the talk, and he runs a very "data-driven" company (they gave us Airflow!)
Where are you seeing this applied and successful in reality (versus Venture Capitalist marketing)?
“I am sorry, but if you are telling people that you can rank the effectiveness of a political ad “down to a tenth of a percentage point of precision in multiple categories,” you aren’t learning something; you are selling something.” That quote is from the journalists, not the people doing the ranking. The folks doing the ranking are serious social scientists who actually know how to calculate standard errors…
It depends what the standard error is. My point is that I guarantee they calculate that and put it in decks and memos but that the journalists were ignoring that. And sure—you can always question methods—external validity of RCT’s etc. But the original snark was at falsely claimed precision, which I guarantee the practitioners are not doing.
Sure, you can calculate standard errors, but is the data *really* telling you that an ad was 34.6% effective? Not 33.0%? At the bottom of all this precision is the assumption that 'effectiveness' can even be measured and that the methods for measuring and collecting the data are remotely as precise as the math.
My take is that we can over rely on data and think the precision we've calculated is meaningful when it's not. Just because I can give you a result of 34.6%, doesn't mean we're being data driven, in the sense that we're working with reliable and accurate measurements. If data driven simply means we're using numbers from some sort of measurement, that's not necessarily anything to crow about.
The roles of taste and data can co-exist. I'm reminded of this fantastic quote from this NYT article about SSENSE, a fashion aggregator that has become known for its own unique taste:
“Our growth is a result of our having two strong approaches — the art of it and the science of it,” said Krishna Nikhil, the chief merchandising and marketing officer. “We don’t blend art and science. If you blend, you get mush. We toggle.”
Mr. Atallah put it more directly: “I look at data day in and day out. That’s what feeds the intuition. Intuition is not just ‘I feel like doing it.’”
This mode of computer-science thinking, as Eric Hu, Ssense’s design director from 2016 to 2018, said, guides every aspect of the company’s decision making. “Rami has a very clear-cut vision for how things should be,” he said. “And he’s able to make very bold decisions because he requires data and actual intel to prove his hunches.”
I think your outliers can be very easily explained by psychology, and more specifically, psychopathy. The allure of psychopathic traits to the often "volatile" or "exciting" business and political world drive people to take risks they normally do not. I don't think data is a fad at all, but rather, it matters what sort of data you collect for stable vs unstable situations.
I was tickled to see you ruminating on the history of analytics! I have for a while been noodling on trying to explore that topic more deeply. Moneyball is certainly one branch. I once ran across a discussion from intelligence analysts that felt awfully resonant with my professional experience. Data quality concerns, the futility of writing the perfect memo that gets ignored by leaders, getting held accountable for bad outcomes when all you said was your estimate was that success was more likely than not, and so on. Maybe there's something about "staff officers" in here too; the cadre of people who support a senior leader in their thinking and operations. Feels similar to some elements of data leadership.
My not-yet-researched sense is that "analyst" as a title was an intelligence or defense title before it was a private sector financial title. Perhaps this is part of the etymology of "business intelligence?" Again, zero evidence of this but I have a hunch there's something going on in the history that would be illuminating.
What a strange take. There's no dichotomy between "gut" and "data". There's no magic business instinct. The idea the Elon Musk and the engineers and SpaceX and Tesla are just winging it is silly. One of the highly publicized reasons for making cuts at Twitter was the low revenue per employee figure.
Oh well. Can't wait for a bunch of people to start saying "who needs data?" so those of us who understand why data is useful in decision making can get a huge edge...again.
I think I would frame the question pretty differently. I'm certainly not saying engineers at SpaceX are winging it, nor am I saying Elon Musk did what he did at Twitter without looking at some basic charts. But, I do think it's pretty clear he approached it much more aggressively, more quickly, and bluntly than a bunch of very "data-driven" McKinsey consultants would've. To your point, he probably saw a handful of numbers, said "wow that employee number is high and that revenue number is low" and then hacked off most of the employees. And that's sort of the point - that's both 1) *not* something people would've called data driven two year ago, and 2) seems to be an increasingly more popular way to do things today.
My point is that for the last 15 years, people have written thousands of articles like these, about how companies need to be more data driven, how it's hard, how it's critical, how it's a huge competitive advantage, and so on. These articles - and the industry around them - aren't arguing for looking at basic charts; they're saying you need to be much more rigorous, detailed, and analytical. So while we could call looking at basic charts and dashboards and revenue numbers data driven, I think that's a pretty big deviation from how the term has been used (and sold) for a while.
Put differently, were sports teams in the 1990s data driven? They looked at stats. But they did so in a way that was far more rudimentary than teams do today. My question is, is that latter thing a fad?
Ah. Yes, definitely on how it's been sold; in some ways 'data driven' has been pushed like magic beans by the data industry. My definition is not nearly as demanding.
Yes, I think it's possible to over rely on numbers and data . We can trick ourselves into thinking we can precisely measure something when we cannot.
I think they are arguing to look at basic charts, because that's where you start. I'm not sure about your experience, but mine is that 90% of companies aren't even doing that. You start simple and go from there. You can't jump to the end and magically be Google.
Yes, 1990s sports teams were data-driven. It was rudimentary because those were the tools, data and skills available. Now it is more advanced because the tools, data and skills have advanced. I think this is the point.
Maybe our difference here is being data driven versus "data driven" (the brand)?
No, of course it's not a fad. Business has always and will always whatever data is useful and obtainable in decision making. Even in fashion. But this certainly doesn't mean every decision is going to be based on a dashboard in Tableau derived from set of tables in Snowflake.
As for stocks - no amount of data will *ever* allow you to reliably and consistently predict the stock price 24 hours into the future for anything. Not unless humans are completely out of the loop. Completely. But then who cares?
Like, sure, but isn't that basically saying "nothing changes and nothing has ever changed?" I agree that there will always be degrees of data; there will always be some desire to be data-driven; stock prices will never be perfectly predictable; etc. But it seems pretty clear that, over the last 20-30 years, we've tried to introduce data into a lot more decisions than we did before. My question is mostly about that. The rough assumption seems to be that it's a one-way ratchet. But could it not go in reverse?
"But it seems pretty clear that, over the last 20-30 years, we've tried to introduce data into a lot more decisions than we did before."
Yes, definitely, because we've been able to more easily capture more data about people and processes. Whether or not it has all been useful is debatable. Yes, I think some of this will go in reverse because it's not all useful. Witness the debate/growing dismissal of marketing attribution data.
I really doubt all the data collected in an effort to predict stock prices is useful, but the payoffs are so high when someone momentarily hits that people will keep trying.
Exactly, I think that's my question. There was a lot of hype around the big explosion of stuff in the last 20-30 years, and part of that hype was selling that as this inevitable march upwards. And to your point about the dismissal of marketing attribution, that's a good example of "the vibes have shifted" thing I was trying to say: It seems like there's now a lot more skepticism of that stuff - and to some degree, interest in other stuff - than there was a few years ago.
(On stock prediction, yeah, the point of whole stock thing wasn't really to say that we can *predict* stock prices. It was just that there are philosophical regime changes. For a long time, there has been - and still is, really - a regime that says stocks should be roughly valued based on these various financial metrics. Matt Levine's point was perhaps that regime will recede, even though going back to vibes and animal spirits seems like a regression. And my point is that maybe the "data driven" regime recedes to something else, even if that also feels like a regression.)
"It seems like there's now a lot more skepticism of that stuff"
I think the stuff you're talking about here is marketing, not data.
On stocks, I think in the long run they are valued primarily on the earnings the company produces. In the short run, *anything* can drive the price. And I think this has become a lot more volatile with the rise of easy retail investing. But I think vibe investing has been around since stocks have been traded, too. There's just more of it and it's more obvious.
The main insight from posts like yours and Founder Mode is that intuition, once deemed unprofessional, has re-emerged as a valid decision-making tool.
It never went away. Ask probably any executive. I can't think of many management roles where some intuition isn't part of decision making because we can't measure everything.
I think I agree with both of these points? "Founder Mode" or whatever never went away, but people had less permission to operate that way (partly because the push was for execs to be data driven). The shift to me is that the founder mode style is becoming acceptable, and the data driven style is feeling more dated. (Which isn't necessarily a statement about what's good, or what should be, or any of that. It's just a statement about what's in style.)
"The shift to me is that the founder mode style is becoming acceptable, and the data driven style is feeling more dated."
I interpreted Founder Mode as ensuring that you're in on the key decisions, rather than handing them off to other people. Brian Chesky gave the talk, and he runs a very "data-driven" company (they gave us Airflow!)
Where are you seeing this applied and successful in reality (versus Venture Capitalist marketing)?
Of course. The point is to limit where you have to use intuition, and where you can use "hard data", by measuring things.
I mean, we're not taking estimates on how much money we made last year, are we?
I don't think we disagree. If you can use valid and reliable data to make a decision, that's better.
“I am sorry, but if you are telling people that you can rank the effectiveness of a political ad “down to a tenth of a percentage point of precision in multiple categories,” you aren’t learning something; you are selling something.” That quote is from the journalists, not the people doing the ranking. The folks doing the ranking are serious social scientists who actually know how to calculate standard errors…
It depends what the standard error is. My point is that I guarantee they calculate that and put it in decks and memos but that the journalists were ignoring that. And sure—you can always question methods—external validity of RCT’s etc. But the original snark was at falsely claimed precision, which I guarantee the practitioners are not doing.
Sure, you can calculate standard errors, but is the data *really* telling you that an ad was 34.6% effective? Not 33.0%? At the bottom of all this precision is the assumption that 'effectiveness' can even be measured and that the methods for measuring and collecting the data are remotely as precise as the math.
What does any of this have to do with being data-driven, as opposed to a lack of statistical knowledge?
My take is that we can over rely on data and think the precision we've calculated is meaningful when it's not. Just because I can give you a result of 34.6%, doesn't mean we're being data driven, in the sense that we're working with reliable and accurate measurements. If data driven simply means we're using numbers from some sort of measurement, that's not necessarily anything to crow about.
The roles of taste and data can co-exist. I'm reminded of this fantastic quote from this NYT article about SSENSE, a fashion aggregator that has become known for its own unique taste:
https://www.nytimes.com/2021/11/23/style/the-sensibility-of-ssense.html
“Our growth is a result of our having two strong approaches — the art of it and the science of it,” said Krishna Nikhil, the chief merchandising and marketing officer. “We don’t blend art and science. If you blend, you get mush. We toggle.”
Mr. Atallah put it more directly: “I look at data day in and day out. That’s what feeds the intuition. Intuition is not just ‘I feel like doing it.’”
This mode of computer-science thinking, as Eric Hu, Ssense’s design director from 2016 to 2018, said, guides every aspect of the company’s decision making. “Rami has a very clear-cut vision for how things should be,” he said. “And he’s able to make very bold decisions because he requires data and actual intel to prove his hunches.”
I think your outliers can be very easily explained by psychology, and more specifically, psychopathy. The allure of psychopathic traits to the often "volatile" or "exciting" business and political world drive people to take risks they normally do not. I don't think data is a fad at all, but rather, it matters what sort of data you collect for stable vs unstable situations.
I was tickled to see you ruminating on the history of analytics! I have for a while been noodling on trying to explore that topic more deeply. Moneyball is certainly one branch. I once ran across a discussion from intelligence analysts that felt awfully resonant with my professional experience. Data quality concerns, the futility of writing the perfect memo that gets ignored by leaders, getting held accountable for bad outcomes when all you said was your estimate was that success was more likely than not, and so on. Maybe there's something about "staff officers" in here too; the cadre of people who support a senior leader in their thinking and operations. Feels similar to some elements of data leadership.
My not-yet-researched sense is that "analyst" as a title was an intelligence or defense title before it was a private sector financial title. Perhaps this is part of the etymology of "business intelligence?" Again, zero evidence of this but I have a hunch there's something going on in the history that would be illuminating.
What a strange take. There's no dichotomy between "gut" and "data". There's no magic business instinct. The idea the Elon Musk and the engineers and SpaceX and Tesla are just winging it is silly. One of the highly publicized reasons for making cuts at Twitter was the low revenue per employee figure.
Oh well. Can't wait for a bunch of people to start saying "who needs data?" so those of us who understand why data is useful in decision making can get a huge edge...again.
I think I would frame the question pretty differently. I'm certainly not saying engineers at SpaceX are winging it, nor am I saying Elon Musk did what he did at Twitter without looking at some basic charts. But, I do think it's pretty clear he approached it much more aggressively, more quickly, and bluntly than a bunch of very "data-driven" McKinsey consultants would've. To your point, he probably saw a handful of numbers, said "wow that employee number is high and that revenue number is low" and then hacked off most of the employees. And that's sort of the point - that's both 1) *not* something people would've called data driven two year ago, and 2) seems to be an increasingly more popular way to do things today.
Why wouldn't making decisions based on comparatively poor revenue/employee numbers be data driven?
My point is that for the last 15 years, people have written thousands of articles like these, about how companies need to be more data driven, how it's hard, how it's critical, how it's a huge competitive advantage, and so on. These articles - and the industry around them - aren't arguing for looking at basic charts; they're saying you need to be much more rigorous, detailed, and analytical. So while we could call looking at basic charts and dashboards and revenue numbers data driven, I think that's a pretty big deviation from how the term has been used (and sold) for a while.
- https://hbr.org/2021/02/why-is-it-so-hard-to-become-a-data-driven-company
- https://hbr.org/2021/06/legacy-companies-need-to-become-more-data-driven-fast
- https://hbr.org/2022/02/why-becoming-a-data-driven-organization-is-so-hard
Put differently, were sports teams in the 1990s data driven? They looked at stats. But they did so in a way that was far more rudimentary than teams do today. My question is, is that latter thing a fad?
Ah. Yes, definitely on how it's been sold; in some ways 'data driven' has been pushed like magic beans by the data industry. My definition is not nearly as demanding.
Yes, I think it's possible to over rely on numbers and data . We can trick ourselves into thinking we can precisely measure something when we cannot.
I think they are arguing to look at basic charts, because that's where you start. I'm not sure about your experience, but mine is that 90% of companies aren't even doing that. You start simple and go from there. You can't jump to the end and magically be Google.
Yes, 1990s sports teams were data-driven. It was rudimentary because those were the tools, data and skills available. Now it is more advanced because the tools, data and skills have advanced. I think this is the point.
Maybe our difference here is being data driven versus "data driven" (the brand)?