Great post, Benn. Your first para about the VC made me chuckle. I'd add a couple of observations:
- IMHO, a (very) important component of being data-driven is using data to define goals/outcomes, and then using those outcomes to frame decisions. There's so much pointless retrospective analysis done (especially in areas like marketing) that doesn't lead to any different decision being made because the team doesn't even know what they're aiming for. So if MTRX came to the pitch with a really crisp/interesting set of metrics that they were going to manage the business to, and a good story about why those metrics were important and spoke to how they saw themselves creating business differentiation, I'd be interested.
- Another benefit of the "counting cards" approach to using data is that over time, teams & leaders develop an intuitive grasp of the core dynamics of the business they're managing through the data (which, as an aside, is why stable, well-engineered analytical models are so important). Over time this intuitive understanding helps people to make better decisions, whether or not they actually use data to make those decisions, because they understand how the different measures of business performance are connected - effectively learning the business through the data it generates.
- On goals, I agree, though mostly as a political tool. I think things like OKRs are generally kind of useless as an actual scoreboard. However, I do think that OKRs (and quantitative targets generally) are very useful to make clear where the company wants to go, and what it values (eg, revenue over user adoption, new customer growth over retention, product love over revenue, etc).
- I very much agree on the core dynamics bit. This is a longer post for another day, but I think this is actually the most valuable thing data teams can do: Give people a sense for how the business works. It's not metrics and KPIs; it's a model for what inputs affect what outputs, and things like that.
also - I've spent all weekend pondering the implications of your modern data stack demise commentary - from prior years to latest. Shit, I'll have to quit my day job if you start hammering out content full time.
Super weird to put Long View last, when they're described as having experience and intuition, and your criteria to pick is "talent and intuition". You seem to have an implicit bias that people who know an industry, regardless of what experience they have, don't have talent or intuition, and, furthermore, will be slow and indecisive?
I think of it slightly differently. It's not that they don't have talent or intuition, or that they're slow and indecisive; it's that they're average at all of those things.
If I decided to be a salesperson, for example, I think I could be a decent one if I put thirty years of effort into learning it. I'd probably be better than someone who was two years into their career, because I'd know the tricks of the trade, figured out what I was good and bad at, and so on.
But there might be some reps who are two years into their careers because they're just really talented. They're charismatic; they have good selling instincts. Or maybe they're braver than I am, and aren't afraid to make the big ask. That doesn't say anything bad about me; it just means that there are different ways to be good.
You didn't actually *say* that in your description, though. Long View's description doesn't say anything about being average at everything.
How are we to interpret their description ("The founders of the first company, Long View, have been working in fashion for decades. We know the market, they say; our experience, and the intuition we’ve developed on top of it, will make us successful.") to mean "they are average at everything", unless you yourself inherently believe that people who have been working in an industry for decades must be average at everything?
I think you must believe that, because even your reply here is telling, essentially saying that the difference between 2 and 30 years of experience in a profession pales in comparison to "talent". You also seem to be implying that "talent" lacks a contextual element, as if there are people who have "talent" who will kick ass at any job, and others who just don't.
Do you think your ranking here changes if the industry is not fashion, but one that remains essentially undisrupted by new entrants despite billions of VC investment to date? (e.g., consumer banking, consumer insurance)
Certainly. I don't know that VC investment matters directly, but it's probably correlated, in that VCs are less inclined to invest in industries where it's hard for inexperienced upstarts to win.
To add something on playing different games at different tables - if MTRX does find a competitive AI driven advantage, unassailable by the others, then they would win by 10x, but if not then they certainly fail.
Square Corner will win in most scenarios, but only with a marginal % lead.
To the casino analogy (I know very little about casinos), MTRX is playing a few high stakes poker hands, while Square Corner is grinding away counting cards.
I think that's the danger, actually. When we talk about data helping us make better decisions, there's this kind of implicit assumption that we should use it to play high stakes poker—put in the analytical work to make the best big decision we can.
But that doesn't actually work. Data gives you marginal edge, which doesn't help you if you're just playing a few hands. You've got to apply it grinding away at the blackjack table, where that edge compounds. For the big hands, I'd rather have someone with experience and talent where the variance is higher rather than someone who just runs the numbers and has slightly better odds than the regular palyer.
I really enjoyed this piece — you nailed it. One data driven decision doesn't directly lead to an advantage or huge return, but rather a bunch of small ones added up over time.
Agree with your points (clarified after reading the footnotes): [paraphrasing] companies hit home runs by swinging more often - not by few sings of a better bat. But the better bat (data) allows them to swing more often. Companies with strong a priori statements on strategy or positioning (e.g. “our clothes will be this or that”) are single swings. Some will, by random luck alone, hit home runs. imho, the rate of such home runs will be *less* than a random draw (ie. Less than 1/n where n is the number of such companies). That is, intuition not only doesn’t help , but actually hurts. Yet, the statement does provide a nice hindsight narrative.
Data (and algorithms and insights) allows you to reflect on your swing, adjust and take another swing. That subsequent swings are more informed. You are informed by high-frequency-events (transactions, customer interactions, etc) which gives you many, many, many opportunities to adjust the swing.
I half agree. I think that data-oriented companies probably take more swings, but I don't think that's necessarily because data enables them to do it. An intuition based organization could also try lots of things.
But, I think companies that want to try things tend to use more data, because they want to measure those things. In other words, I'd argue that the philosophy that causes companies to try lots of things also causes them to use data, rather than data causing them to try stuff.
Probably true that companies that want to try more things also want to use data. But without the data (and proper AB Testing), there is no way to know which ideas are working and which are failing. And most business ideas do fail! Some evidence suggest that as many as 70-90% of our business ideas fail (See our article here: https://www.oreilly.com/radar/the-sobering-truth-about-the-impact-of-your-business-ideas/). With such high failure rates, the only way to find enough wins is to try a lot of things. So, merely trying more things, without proper measurement, wont always lead to good outcomes.
I agree that most things are crapshoots, but I'm a bit skeptical that A/B testing is actually all that necessary or useful for most companies in figuring out what works and what doesn't. For most companies, finding statistically significant but otherwise hard to see wins doesn't really matter that much. To quote a leader from Lyft from Sean Taylor's last blog post, "when a project we work on succeeds, we don’t need statistics to know it." (https://notes.causal.engineering/archive/locally-optimal/) If you take some swings, you'll still be able to tell what's a big win without doing all that much data work.
Having provided analysis of fashion (mainly footwear ) businesses for over 40 years here's my 2c.
For a business to be successful the owners need to understand the business/industry really well and that's where they spend their time.
Companies and people like me can then use KPIs and processes developed over the decades to make the business work well operationally, and then apply analysis from the formulas derived over years of research to show how and where the business can grow.
To me, these are critical. Industry knowledge, well run operations, proper analysis.
eg one of the leading causes of retail failure is accumulation of unsaleable stock. It's incredibly misleading because assets accumulate when they are not assets and retailers struggle to convince themselves to write them off.
If you run B&M as well then my measure - stock vitality - is critical to using online to help clear stock from where it won't sell and therefore allow stocking a location with stock that might sell.
If you run an online only enterprise and I ask how are you going to add another $1m to turnover it's tricky. More advertising, Google adwords, etc all of which can cost more than the increase in notional profit.
Same question and you have B&M? Open another store.
Until ML and AI insisted they could do a better job and sort of won the marketing war we were adding significant profits to our clients. We still do, but those who believed the data approach have never been able to quite get there. In retail you see so many fads over the years.
So the company I would back is the one where the owners know their product space and customers really well, look to business operations as critical support, and are prepared to bring in outside help to achieve their goals.
Several companies have dragged me out of semi retirement for exactly this reason.
That gets at one very obvious element of this that I completely ignored, which is the choice of investment would also be somewhat industry specific. Certainly there are some industries where experience matters much more than others. As might be the case with fashion or retail (neither of which I know much about), there could be 1,000 gotchas that first time execs would trip over. For other types of companies, there could be a lot fewer (or more).
I followed you the whole way through on this one, but got stuck at the positioning of Long View vs. Square Corner. I had my rankings flipped from yours (I had Long View at #3 and Square Corner at #5). From my understanding, your opposing ranking is suggesting that military precision in communication will outperform experience in the industry.
I saw the article you linked on "speed" and the importance of speed in analytical decision making. However, I'd be curious to hear any more thoughts you have on ranking Speed vs. Experience? I've always thought Speed is important, but have weighted Experience more heavily.
Side note - laughed at "a budding oak sprig that symbolizes your commitment to helping small saplings grow into enduring landmarks, and your “commitment” to sustainable investing"
Thanks! For Square Corner, I meant that the team is ruthless in execution (hits delivery dates, on top of customer communications, etc). I think that can cover a lot of warts and "bad" decisions.
My view on experience is half of the reason it's useful is because it enables you to be fast. Experienced leaders don't have to figure out how to do things from first principles; they can skip straight to the end. But that only applies to decisions that the team has experience with, and I think I'd rather invest in a team that does that by default, in all areas.
So when data pundits talk about the need to “bring value” how do you interpret this when the value doesn’t exist at any point in time ? how do you measure the measurer?
Great post, Benn. Your first para about the VC made me chuckle. I'd add a couple of observations:
- IMHO, a (very) important component of being data-driven is using data to define goals/outcomes, and then using those outcomes to frame decisions. There's so much pointless retrospective analysis done (especially in areas like marketing) that doesn't lead to any different decision being made because the team doesn't even know what they're aiming for. So if MTRX came to the pitch with a really crisp/interesting set of metrics that they were going to manage the business to, and a good story about why those metrics were important and spoke to how they saw themselves creating business differentiation, I'd be interested.
- Another benefit of the "counting cards" approach to using data is that over time, teams & leaders develop an intuitive grasp of the core dynamics of the business they're managing through the data (which, as an aside, is why stable, well-engineered analytical models are so important). Over time this intuitive understanding helps people to make better decisions, whether or not they actually use data to make those decisions, because they understand how the different measures of business performance are connected - effectively learning the business through the data it generates.
- On goals, I agree, though mostly as a political tool. I think things like OKRs are generally kind of useless as an actual scoreboard. However, I do think that OKRs (and quantitative targets generally) are very useful to make clear where the company wants to go, and what it values (eg, revenue over user adoption, new customer growth over retention, product love over revenue, etc).
- I very much agree on the core dynamics bit. This is a longer post for another day, but I think this is actually the most valuable thing data teams can do: Give people a sense for how the business works. It's not metrics and KPIs; it's a model for what inputs affect what outputs, and things like that.
Yes. Its experience. One can think of the best 'data driven' efforts as the senses of the organizational entity. Experience takes time.
re. cabin-fevered - backyard, but same, and o.k. Mark Twain https://marktwainstudies.com/about/mark-twains-study/
also - I've spent all weekend pondering the implications of your modern data stack demise commentary - from prior years to latest. Shit, I'll have to quit my day job if you start hammering out content full time.
Excellent, and then you can join me in my backyard cabin where we grow crazy mustaches and write tales of adventurous young men.
(But also, would be curious what you think about the implications of our potential demise, before we get to all that other stuff.)
"Data is a means to an end, not an end in itself"- said no amatuer data professional ever!
Super weird to put Long View last, when they're described as having experience and intuition, and your criteria to pick is "talent and intuition". You seem to have an implicit bias that people who know an industry, regardless of what experience they have, don't have talent or intuition, and, furthermore, will be slow and indecisive?
I think of it slightly differently. It's not that they don't have talent or intuition, or that they're slow and indecisive; it's that they're average at all of those things.
If I decided to be a salesperson, for example, I think I could be a decent one if I put thirty years of effort into learning it. I'd probably be better than someone who was two years into their career, because I'd know the tricks of the trade, figured out what I was good and bad at, and so on.
But there might be some reps who are two years into their careers because they're just really talented. They're charismatic; they have good selling instincts. Or maybe they're braver than I am, and aren't afraid to make the big ask. That doesn't say anything bad about me; it just means that there are different ways to be good.
You didn't actually *say* that in your description, though. Long View's description doesn't say anything about being average at everything.
How are we to interpret their description ("The founders of the first company, Long View, have been working in fashion for decades. We know the market, they say; our experience, and the intuition we’ve developed on top of it, will make us successful.") to mean "they are average at everything", unless you yourself inherently believe that people who have been working in an industry for decades must be average at everything?
I think you must believe that, because even your reply here is telling, essentially saying that the difference between 2 and 30 years of experience in a profession pales in comparison to "talent". You also seem to be implying that "talent" lacks a contextual element, as if there are people who have "talent" who will kick ass at any job, and others who just don't.
The "all else is equal" was meant to be implied by saying they're the same aside from the highlighted differences.
Do you think your ranking here changes if the industry is not fashion, but one that remains essentially undisrupted by new entrants despite billions of VC investment to date? (e.g., consumer banking, consumer insurance)
Certainly. I don't know that VC investment matters directly, but it's probably correlated, in that VCs are less inclined to invest in industries where it's hard for inexperienced upstarts to win.
Very crisp writing, I didn't even skim.
To add something on playing different games at different tables - if MTRX does find a competitive AI driven advantage, unassailable by the others, then they would win by 10x, but if not then they certainly fail.
Square Corner will win in most scenarios, but only with a marginal % lead.
To the casino analogy (I know very little about casinos), MTRX is playing a few high stakes poker hands, while Square Corner is grinding away counting cards.
I think that's the danger, actually. When we talk about data helping us make better decisions, there's this kind of implicit assumption that we should use it to play high stakes poker—put in the analytical work to make the best big decision we can.
But that doesn't actually work. Data gives you marginal edge, which doesn't help you if you're just playing a few hands. You've got to apply it grinding away at the blackjack table, where that edge compounds. For the big hands, I'd rather have someone with experience and talent where the variance is higher rather than someone who just runs the numbers and has slightly better odds than the regular palyer.
I really enjoyed this piece — you nailed it. One data driven decision doesn't directly lead to an advantage or huge return, but rather a bunch of small ones added up over time.
Agree with your points (clarified after reading the footnotes): [paraphrasing] companies hit home runs by swinging more often - not by few sings of a better bat. But the better bat (data) allows them to swing more often. Companies with strong a priori statements on strategy or positioning (e.g. “our clothes will be this or that”) are single swings. Some will, by random luck alone, hit home runs. imho, the rate of such home runs will be *less* than a random draw (ie. Less than 1/n where n is the number of such companies). That is, intuition not only doesn’t help , but actually hurts. Yet, the statement does provide a nice hindsight narrative.
Data (and algorithms and insights) allows you to reflect on your swing, adjust and take another swing. That subsequent swings are more informed. You are informed by high-frequency-events (transactions, customer interactions, etc) which gives you many, many, many opportunities to adjust the swing.
I half agree. I think that data-oriented companies probably take more swings, but I don't think that's necessarily because data enables them to do it. An intuition based organization could also try lots of things.
But, I think companies that want to try things tend to use more data, because they want to measure those things. In other words, I'd argue that the philosophy that causes companies to try lots of things also causes them to use data, rather than data causing them to try stuff.
Probably true that companies that want to try more things also want to use data. But without the data (and proper AB Testing), there is no way to know which ideas are working and which are failing. And most business ideas do fail! Some evidence suggest that as many as 70-90% of our business ideas fail (See our article here: https://www.oreilly.com/radar/the-sobering-truth-about-the-impact-of-your-business-ideas/). With such high failure rates, the only way to find enough wins is to try a lot of things. So, merely trying more things, without proper measurement, wont always lead to good outcomes.
I agree that most things are crapshoots, but I'm a bit skeptical that A/B testing is actually all that necessary or useful for most companies in figuring out what works and what doesn't. For most companies, finding statistically significant but otherwise hard to see wins doesn't really matter that much. To quote a leader from Lyft from Sean Taylor's last blog post, "when a project we work on succeeds, we don’t need statistics to know it." (https://notes.causal.engineering/archive/locally-optimal/) If you take some swings, you'll still be able to tell what's a big win without doing all that much data work.
Having provided analysis of fashion (mainly footwear ) businesses for over 40 years here's my 2c.
For a business to be successful the owners need to understand the business/industry really well and that's where they spend their time.
Companies and people like me can then use KPIs and processes developed over the decades to make the business work well operationally, and then apply analysis from the formulas derived over years of research to show how and where the business can grow.
To me, these are critical. Industry knowledge, well run operations, proper analysis.
eg one of the leading causes of retail failure is accumulation of unsaleable stock. It's incredibly misleading because assets accumulate when they are not assets and retailers struggle to convince themselves to write them off.
If you run B&M as well then my measure - stock vitality - is critical to using online to help clear stock from where it won't sell and therefore allow stocking a location with stock that might sell.
If you run an online only enterprise and I ask how are you going to add another $1m to turnover it's tricky. More advertising, Google adwords, etc all of which can cost more than the increase in notional profit.
Same question and you have B&M? Open another store.
Until ML and AI insisted they could do a better job and sort of won the marketing war we were adding significant profits to our clients. We still do, but those who believed the data approach have never been able to quite get there. In retail you see so many fads over the years.
So the company I would back is the one where the owners know their product space and customers really well, look to business operations as critical support, and are prepared to bring in outside help to achieve their goals.
Several companies have dragged me out of semi retirement for exactly this reason.
That gets at one very obvious element of this that I completely ignored, which is the choice of investment would also be somewhat industry specific. Certainly there are some industries where experience matters much more than others. As might be the case with fashion or retail (neither of which I know much about), there could be 1,000 gotchas that first time execs would trip over. For other types of companies, there could be a lot fewer (or more).
Loved this article, Ben.
I followed you the whole way through on this one, but got stuck at the positioning of Long View vs. Square Corner. I had my rankings flipped from yours (I had Long View at #3 and Square Corner at #5). From my understanding, your opposing ranking is suggesting that military precision in communication will outperform experience in the industry.
I saw the article you linked on "speed" and the importance of speed in analytical decision making. However, I'd be curious to hear any more thoughts you have on ranking Speed vs. Experience? I've always thought Speed is important, but have weighted Experience more heavily.
Side note - laughed at "a budding oak sprig that symbolizes your commitment to helping small saplings grow into enduring landmarks, and your “commitment” to sustainable investing"
Thanks! For Square Corner, I meant that the team is ruthless in execution (hits delivery dates, on top of customer communications, etc). I think that can cover a lot of warts and "bad" decisions.
My view on experience is half of the reason it's useful is because it enables you to be fast. Experienced leaders don't have to figure out how to do things from first principles; they can skip straight to the end. But that only applies to decisions that the team has experience with, and I think I'd rather invest in a team that does that by default, in all areas.
Very well put response, thanks!
Brillliant
This was a fun read!
Here's why I recommend subscribing to benn.substack: Usually offers unique insights but is always delightful to read!
So when data pundits talk about the need to “bring value” how do you interpret this when the value doesn’t exist at any point in time ? how do you measure the measurer?
Meaning, how does a data team "bring value" to a company if nothing they do is valuable? I'm not sure I follow.