Compacting...

“You can only feel it.”

There’s a difference between knowing a path and walking the path.

Here is how it happened:

Two days ago, Anthropic released an enormous study on what people think about AI. Though the results of the study are interesting—people are both excited and nervous; they feel its power and its burdens;¹ AI is the light and the shade²—I was more immediately struck by the study’s methodology. In one week in December, Anthropic interviewed 80,508 people.³ Interviews were conducted via a choose-your-own adventure chatbot. “Interview transcripts were processed through a suite of hand-validated Claude-powered classifiers.” The groupings that Anthropic used to organize the responses were “designed first by analyzing answers with a bottom-up clustering algorithm,” and “categories were derived from clusters that surfaced during initial analysis.” And the final report was, no doubt, authored with the help of AI, and illustrated with graphics that were surely built using Claude Code. It was, Anthropic said, “the largest and most multilingual qualitative study ever conducted,” and it was done in three months.

An immediate story emerged: This is the future we’ve been talking about.

From two years ago:

If you had a day to save your [business], where would you go looking for answers? In your comprehensive database of financial indicators, operational KPIs, customer behaviors, and market trends, where every metric and insight is a query away? Or in 25 straight hours of unedited video interviews [with 750 customers]?

Two things about this situation seem obvious. First, the actual solution to your [business’] problems is in the customer interviews…And second, despite that, you’d probably use the data, not the interviews. You have 24 hours! There are 25 hours of videos! You only have time to watch a few! You would be insane…if you proposed a bunch of changes based on a handful of randomly selected snippets of feedback. …

[If AI revolutionizes how we work with data, it won’t come from analyzing numbers.] It’ll come from pointing AI at a new well—the unstructured video interviews in Dropbox—and letting us do the basic things with it that we’ve never been able to do before.

From last year:

The enthusiasm that SaaS startups had for analytics, experimentation, and rigorous quantitative thinking has been almost wholly replaced by a demand for people with taste and “agency.” One popular AI company…[has] formal evals to measure how their product is performing, they said, but decisions are ultimately made based on how new features feel.

None of this is to say that data is going away. But it is falling out of fashion. It is fading into the background. In data we trusted; now, God is in the vibes. …

If AI is good at anything, it is good at interpreting the vibes. It is good at aggregating massive amounts of text—and increasingly, of video and audio—into its approximate average. Give it your support tickets and customer communications, and ask it questions about what it read. Don’t classify and categorize images; just ask an AI model what it thinks it sees.

And from earlier this year, after OpenAI released a similar study in which they classified users’ ChatGPT conversations by asking ChatGPT to do it:

If two companies handed their decision-making over to ChatGPT, which one would you bet on? The one that attempted to map every email, Slack message, and database entity into a complex ontological simulacrum and a “semantic mesh,” or the one that figured out how to collect a giant folder full of transcribed voice notes of people describing why they did everything they did? Which one would you trust more: Our ability to model how 1,000 people collectively think, or a state-of-the-art AI, looking for patterns in a large corpus of unstructured text?

There’s something uncomfortable in the latter proposal. We’re used to solving problems with rules and imperative logic. But computers are pretty weird now. And the best companies—in this domain, and many others—seem likely to be those that embrace that, do the dumb thing—build a text box; collect the data—and convince people to always be writing stuff down in it.

This will be an easy post, I thought. A clean narrative; a clear affirmation of what’s coming next. Write it up; declare something dead; announce the beginning of a new era. Talk about the rise of a new market; brand it with a punchy name; call it “big vibes;” call it “ambient analytics;” call it the “enterprise context engine.” Issue a call for new tools: A new Segment for logging unstructured events and observations; a new Mixpanel for building dashboards on top autonomously conducted and automatically analyzed user interviewers; a new Fivetran and a new dbt for moving and transforming video recording and customer interviews into a library of markdown files and “context.” Close with a prediction that someone from Anthropic will leave to turn their internal classifier into a commercial product. Are you an ambitious founder building in this space? We would be excited to talk to you.

I wrote down some notes. I started to fill in the details. Just how big is that Anthropic study? They provide a convenient comparison: The largest prior studies that they found, one conducted by the USC Shoah Foundation Visual History Archive and another run by the World Bank, both interviewed 60,000 people. How much faster did Anthropic run their 80,000 person study? The World Bank’s project, called “Voices of the Poor: Crying out for Change” and conducted in the late 1990s, took years. They planned it for six months;⁴ for six more months, interviews were conducted by teams of local researchers deployed into 23 different countries, and, “on average each research team member worked 14-15 hours per day.” “Notes were written up every day, often until dawn,” with researchers producing “close to 10,000 pages of field notes and national synthesis reports.” “In September 1999 a preliminary global synthesis was complete;” a final 336-page book was published in August of 2000.

The final tally: The World Bank—60,000 interviews, from 23 countries, tens of thousands of hours of work done by almost 400 people, taking two years from start to finish. Anthropic—81,000 interviews across 159 countries and 79 languages, in three months, done by 25 people. It is a breathtaking acceleration; a staggeringly short ride in an incomprehensibly fast machine. Think of what is possible now. Think of what we can learn.

But then I began reading the World Bank’s report:

In Uzbekistan the researchers write that participation in the study helped them see their own country with new eyes: “honestly speaking, the sympathy and sense of sharing the destiny of each person encountered which arose during the research process was an experience never achieved in any of our previous studies, either qualitative or quantitative.. .The sensation of insight and sympathy for our own people is the most important finding of this study.”

And:

In Bosnia and Herzegovina, the team leader writes, “Two of our note-takers, young men, were strongly affected by the process. Milos crying silently while taking notes during one discussion group.”

And:

The study demonstrates the powerful impact participatory appraisals can have on those who facilitate them. … Staying in poor communities for even short times and serving as field facilitators in participatory poverty studies create experiential opportunities to listen and learn face-to-face from poor people.

And:

Poverty is like heat: you can not see it, you can only feel it; so to know poverty you have to go through it.

– A poor man, Adaboya, Ghana

Of course, we’ve known this for 70 years⁵—that distance begets prejudice, and immersion begets understanding. You cannot see some things. You can only feel them.

—

I spent this past week driving through the south, from North Carolina to Mississippi and back again, with an old friend from high school. When we crossed over the border between Georgia and Alabama, it was the first time he’d ever left the time zone he was born in.

Later on the drive, he told me he had changed in other ways too. Growing up, he said, he had been casually ignorant about other cultures, and that metastasized into a lazy racism. He was white; other kids were black, Asian, or “Mexican;” there were no other identities. But in recent years, he’d gotten a job working with people from South and Central America, and, “How I was before—that wasn’t right of me,” he said. “I was a product of my raising,⁶ but now, after getting to know more people, I know it was wrong. And it’s important for me to fix it.”

—

There are two sides—a light and a shade, if you will—to the capabilities that powered Anthropic’s study. On one hand, they demonstrate how much more we can learn with AI. When a machine can have conversations and another machine can summarize them, we can hear far more voices. Companies can hear from huge percentages of their customers; the World Bank’s 60,000 interviews can become hundreds of thousands. From an MIT Generative AI Impact Consortium proposal:

As development economists, we perceive economic challenges faced by the poor and design research around these perceptions. We don’t listen sufficiently to the perspectives of the poor.

This gap is not due to a lack of intent, but rather, a lack of technology. When researchers collect qualitative data, they can only process it by reading transcripts or listening to interviews with their own preconceived notions and without the capacity to feasibly process 1000s of hours of conversations. These limitations have prevented researchers from asking the poor directly, at scale, about the challenges they face…

In this project, we propose developing a Generative AI tool that can listen to, and analyze thousands of hours of conversations to automatically discover insights into the lives of the poor that we did not expect.

On the other hand, is this listening? We have always known that facts and figures are sterilizing. “I am not a statistic,” people say, because statistics anonymize us. Numbers can chart poverty, but they cannot make us feel it. With AI, interviews can also be conducted at the same remove, turned into amalgamated aggregations, and anonymized too.

And if we don’t feel what people say, do we understand it? If we don’t grapple with it, does it change us? From Ezra Klein, in a conversation with David Perell:

I used to conceptualize knowledge the way you see it in the movie The Matrix, where it’s like I wanted the port in the back of my mind that the little needle would go into, and then I had read John Rawls’s “Political Liberalism.” I thought that what you were doing was downloading information into your brain.

And now I think that what you are doing is spending time grappling with the text, making connections. It will only happen through that process of grappling. So, the idea that you could speed run that, the idea that it could just be summarized for you…

Part of what is happening when you spend seven hours reading a book is you spend seven hours with your mind on this topic. The idea that [large language models] can summarize it for you…you didn’t have the engagement. It doesn’t impress itself upon you. It doesn’t change you. What knowledge is supposed to do is change you, and it changes you because you make connections to it. …

Not very much that AI has given me has really changed me very much.

Books are the antidote to bulleted summaries. Interviews and face-to-face conversations have long been the antidote to dashboards and data. Or, more dramatically:

You’re an orphan right? Do you think I’d know the first thing about how hard your life has been, how you feel, who you are because I read Oliver Twist?

A few lines later, Robin Williams’ character accuses Matt Damon’s character of being afraid to talk to him. “You’re terrified of what you might say,” he says. To its credit, AI could solve that half of the problem. People may be more comfortable talking to robots,⁷ and the best models may become as good of a conversationalist as any doctor or researcher—and they can certainly be a more prolific.

But what about the other half? If AI intermediates every conversation, if every expression is reduced to a transcript, and every transcript is compacted into a few bullets and pull quotes, will we still hear other people? Will we still understand what they’re really saying?⁸

From a participant in a discussion group in Zawyet Sultan, Egypt:

Only God listens to us.

Soon, the machines will too. We’ll find out if that counts.

1. With great power comes great responsibility to more rapidly increase shareholder value.

2. A fitting analogy, I suppose.

3. They actually interviewed 112,846 people; about 32,000 were discarded for being “spammy, unserious, or extremely minimal.”

4. The idea for the study “emerged in the summer of 1998,” and “three methodological workshops were held in August and December 1998 and in January 1999.”

5. A 2004 study confirmed the durability of this theory: “With 713 independent samples from 515 studies, the meta-analysis finds that intergroup contact typically reduces intergroup prejudice.”

6. I’m not sure which is more embarrassing: That it took him to say “product of my raisin’” in that car for me to finally realize that this lyric is “raisin(g)” and not “raisin(fruit),” or that, when he said “product of my raisin’,” I immediately thought of that song.

7. From the appendix to Anthropic’s study:

One thing we didn’t fully anticipate was how candid people would be. Respondents shared things—grief, mental health crises, financial precarity, relationship failures—that our human user researchers rarely encounter in traditional interviews. While this might be due to the nature of the questions we asked, we also think this reflects something real about the AI interviewer format: there’s little social cost to vulnerability when the “someone” on the other end isn’t a person.

8. Though this is framed around the World Bank’s research, the same questions apply to mundane corporate problems as well. A lot is communicated in user conversations and on support calls that isn’t in a transcript, and we potentially develop a much more human understanding of customers and employees by talking to a handful of them than we do by using a chatbot to interview tons of them.