Do data-driven companies actually win?

Some gut-based judgements on the effectiveness of data.

Imagine, if you can, that you're a venture capitalist. Your Twitter bio says that you're a proud parent, a lucky spouse, and that your DMs are open to passionate entrepreneurs. You don't wear Allbirds or gauche corporate vests—what are you, a consultant?—but you have several jackets that are tastefully branded with your firm's logo, a budding oak sprig that symbolizes your commitment to helping small saplings grow into enduring landmarks, and your “commitment” to sustainable investing. You tell people you work in venture capital but aren't a venture capitalist, not really; you’re more of an operator, a technologist, a product person; you're willing to get in the trenches; you were a founder, too, once; you’re different than other VCs; you get it; you’re in on the joke—“I know, me? A VC? Crazy. They must not have looked at my tweets, lol”—but couldn’t turn down the job because the other partners are actually great, no, but for real though, and if you think about it, isn’t the job just learning, building, and supporting great founders? You parked a Substack domain—freeiumthinking.substack.com—that you plan to one day use to write a loosely technical and occasionally political blog about the SaaS industry, like, you tell yourself, a blend of Ben Thompson’s substance and Matt Levine’s style. You live in San Francisco, work in Palo Alto, daydream about Miami, and are moving to Austin. You listen to the All-In Podcast. You are jealous of its success.¹

Amid the various pitches that land in your inbox, an odd coincidence arrives. Five nearly identical companies reach out to you at once. They're all launching a new clothing line for working from home.² Their operational models are also the same: They sell on Shopify; they use the same ad agencies to manage their marketing campaigns; they all seem capable of running a competent business; to your MBA-trained eyes, all their outfits look like a blue collar stolen valor J.Crew line designed for Ron Swanson.

They aren’t exactly the same though—different types of experts run each company. The founders of the first company, Long View, have been working in fashion for decades. We know the market, they say; our experience, and the intuition we’ve developed on top of it, will make us successful.

The second company, Bolder, is led by executives who’ve been in the industry for less time. But true to their name, they believe in moving fast and making things, in not overthinking strategy, and that decisiveness is often more important than being right. As the last slide of their pitch deck says, “We fail when we look back.”

Prodigious Daughter, the third company, is run by a thirty year-old wonder kid. In just a few years, she’s already put her stamp on fast fashion. Though her company is average in other ways, it has her generational talent.

Square Corner emphasizes operational excellence. Their leadership team writes emails with military precision, never never misses 7:30 a.m. standup, and always sends out board meeting slides, a pre-read, and a Loom explainer video exactly five days early.

The final company, MTRX, believes data will be their competitive edge. Their prior experience, on par with that of Bolder’s founders, has taught them that fashion is fickle and hard to predict. The most iconic brands, they say, will be built by companies that find and respond to new opportunities in the market before anyone else does. Fashion may be art, but running a business is a science. 

Who do you invest in? Join the fantasy league!³ 

The real tradeoffs

If you work in the data industry, chances are you’ve implicitly made the case for investing in MTRX. In nearly every pitch deck for a data product, or in every breathless (and wildly seemingly exaggerated?) McKinsey survey about the future of enterprise information technology, we say that data-driven companies win. In today’s modern economy, as the bit goes, analytical proficiency is table stakes. Haven’t you read Andy Grove? Snap to a slide of Airbnb’s market share. Snap to story about Jeff Bezos bulldozing Little Annie’s Bookstore & Bakery with a binder full of charts. Use data, be smarter, or die. 

When presented in this context, we tend to gloss over statements about the existential importance of data as obvious truisms. We can’t really help it; many of us bet our careers on data being necessary for corporate survival.⁴ But if we divorce ourselves from our employer and our career ambitions, and truly put these beliefs to the test—if, say, we have to choose an apparel company to invest in, where other companies have something the data-driven one doesn’t—do we still stand by them? 

In other words, are we willing to bet our hard-earned cash that data actually is a material competitive advantage?

On one hand, the answer is obviously yes. A company that tracks how its ads are performing clearly has an edge over one that aimlessly lights marketing dollars on fire. A company whose account managers can see when user activity is declining can clearly be more responsive than one that relies on tracking product adoption through face-to-face meetings. A company with an informed executive team is clearly less likely to steer the business over a financial cliff than one where everyone is driving blind. 

On the other hand, these examples cheat the question. When the Harvard Business Review says that “legacy companies need to become more data driven—fast,” they don’t mean we need to build a few dashboards to monitor basic business functions.⁵ They’re arguing that companies need to use “machine learning analytics and [draw] upon thousands of data elements” to stay ahead of the competition. 

Moreover, the interesting question isn’t if a good data function is a net positive, which is a remarkably low bar for something to clear.⁶ The more interesting question is how much better does data make us? Or, to put it another way, if we’re competing against companies that have more experience, are more decisive, have better instincts, or operate more crisply than we do, can data cover these gaps?

That’s the point of the fashion company thought experiment: to force us to assess where we’d rank a proficiency in data among other potential areas of expertise. Because if data is actually that valuable—if it’s truly a competitive edge—we’d trade away other advantages for it. 

Place your bets

Honestly, if it’s my money, MTRX ain’t getting it. My stack rank is 1) Prodigious Daughter, 2) Bolder, 3) Square Corner, 4) MTRX, and 5) Long View. For early to mid-stage companies, give me talent and intuition over everything; if not that, I’ll trade away analytical rigor for speed and decisiveness. 

Still, I don’t think that the statement that “data-driven companies win” is entirely wrong—it’s just data operates through a different mechanism than we might assume. 

Most often, we say that data helps us make better decisions. We can devise better strategies, and be smarter operators. The implication here is that when we’re faced with a choice of what to do, with data, we’re wise; without it, we’re foolish. 

I think this dramatically overstates data’s usefulness. Business problems are extraordinarily complicated, and analytical recommendations are mostly educated guesses. Great data teams likely make somewhat better guesses, but at the end of the day, we’re all still gambling. 

This, however, points to a useful analogy. When people count cards, they track a deck’s “count,” which tells them if a deck is hot or cold. Cold decks slightly favor the house; hot decks favor the player. Even when played against a hot deck, on any one hand, the best card counters in Vegas aren’t going to be much better than the random dope who wandered in off the strip with a yard of margarita from Señor Frog's. But over the course of many hands, card counters can run the table.

Data offers the same promise. Its constant presence in an organization is like knowing the count of the deck. Though it makes us a bit more informed in each decision, the effect is only felt in the aggregate, as the small edge compounds over time.

This has three big implications:

First, we can’t judge data’s effectiveness on big strategic decisions. Businesses, like blackjack tables, are inherently uncertain environments. No amount of analysis can change that. Sometimes, even when you have a lot of chips on the table, you just get unlucky.⁷ 

Second, being data-driven is a long game. It takes time for an advantage to accumulate. Despite analysts’ romanticized ideal of finding that insightful needle in a quantitative haystack, data’s usefulness doesn’t come from eureka moments.⁸ It comes from logging hours and days at the table, gradually collecting small wins. 

Finally, using data effectively means using data a lot. The more hands a card counter can play, the better off they’ll be. Similarly, the more decisions that data nudges, the more its impact will be.

All of this, I think, suggests that data teams are most valuable when they provide an ambient awareness of the current count of the blackjack deck. More concretely, data teams should keep people as informed as they can about the environment everyone is making decisions in—where key metrics are, how they’re trending, the basic facts about performance across segments and product lines, and so on. 

Each time someone makes a decision with this awareness, they’re tilting that choice in the their favor. The bump may be small, but it adds up. And as much as we data folk might like to imagine ourselves as having the ability to swoop in on a few critical decisions and dramatically shift the odds, we can’t, no more than a great card counter can parachute in to win a particular hand. 

That’s why I’m skeptical of MTRX. Companies, especially young ones, that orient themselves around data can often put too much faith in its magical transformative powers—sometimes to disastrous effect. That’s not how it works. Data is a competitive edge, but it’s earned, from months and years of grinding at the table, counting cards, playing hand after hand, slowly bringing down the house.

---

This may come as a surprise to many of you, but I'm employed. I care a lot about the success of that employer, both because some great people work there, and because it's my path to making some money, to moving to the mountains, and to becoming some weird techno-hermit who can spend most of his time writing cabin-fever fueled diatribes about esoteric corners of the data industry and long overdue odes to Pitbull.

So, for entirely selfish reasons about chasing the life that I want to have, I'm going to shill for that employer for a moment: Check out Mode. Or try a demo. I would like the good people there to sell it to you, which helps them, helps me, and, if what I learned in college about consumer surplus is true, would help you too.

1. The future proprietor of benn.ventures will regret having written this paragraph. 

2. So professional tops and leisure bottoms? UNTUCKit meets Chubbies (borderline NSFW)? I have no idea. I write a loosely technical and occasionally political blog about the data industry; I still own an American Eagle shirt; my fashion icon is Pete Davidson. Don’t ask me about clothes. You might as well ask a shark which vegan restaurant serves the best root vegetables.

3. I know, dear reader, I’ve asked for several of these surveys, and haven’t shared any of the results. I will. It’s the absentee ballots, ok? We can’t report anything until we get the absentee ballots back from Maricopa County. But they’re coming, soon. We just need a bit more time.

4. We didn’t drink the data Kool-aid; we drowned in it.

5. No shade on monitoring basic business functions, though. It’s still quite hard.

6. Other things that might clear the “do no harm” bar that a company could invest in: A froyo machine in the kitchen. Branded Crocs. Ten free Putt-Putt tokens for every new employee.

7. Sarah Catanzaro expressed the same skepticism a few months ago.

8. For what it’s worth, I’ve changed my mind on this. I used to think that data was useful because it’d occasionally uncover a company-altering discovery. While that can happen, these big breaks can also happen because you get lucky, or because someone has a good instinct about an opportunity. I now believe the biggest determinant in how frequently companies hit home runs is how often they swing, not the type of bat they use when they do.