How to find insights

Excel, still invincible!

Not Secretariat, but his descendant.

Surely, after so many years, this isn’t the answer, is it?

—

Since the time when people learned to count, the holy grail for every person who counted things was to “find valuable insights.”¹ Children grew up reading almanacs and the backs of baseball cards, and developed a dream—to one day have a job that let them ask fun questions, and find interesting answers. But once they became professional analysts, they spent their time reconciling spreadsheets and making dashboards for other people. So they waited—for the grunt work to be done, for the people to be self-served, and for their days to be filled with something more fulfilling: proactively searching for strategic business opportunities. They didn’t have time for it yet, but in two weeks, they told themselves, after they finish that one last project, “the gridlock will clear up, and there will be nothing left but an open road, and a whole lot of speed.”

The gridlock remained. The last project finished; a new one replaced it. The questions got answered; people still had more. Two more weeks, they said.

Some of them got so tired of waiting that they left their jobs to build tools to finally set their brethren free. We must move away from the crude world of emailed Excel files and manual reporting. They built fancier and fancier tools, with more and more capabilities, all aimed at finally making it possible for anyone to answer questions on their own. We will democratize data, they said. We will set your data free.

They did not. The new tools could not automatically answer everyone’s questions; even when people could answer their first question, they always had a second one they needed help with. Self-serve is still a problem, evermore.

But then—AI. OpenAI built an internal data agent; “teams across Engineering, Data Science, Go-To-Market, Finance, and Research at OpenAI lean on the agent to answer high-impact data questions.” At Ramp, “an AI agent…is now answering thousands of questions per month.” At long last, something can answer the boring questions—at Anthropic, “95% of business analytics queries are automated via Claude, with ~95% accuracy in aggregate”—and the analysts can be free to do to fun stuff—“by giving this often rote, repetitive work to Claude, our data science team can focus on more strategic work like causal modeling, forecasting, and machine learning.”

It seems inevitable, after trying for so long: Analyst will finally work on the frontier, sharing breakthroughs with one another, while the world’s most advanced technology, layered on top of decades of advances in analytical software, delivers answers to everyone else.

—

But I’m not sure that it’ll happen on its own. First, in a world where AI really can answer 95% of the business analytics queries, “the result is a 10-20x increase in the number of questions people ask.” If people are asking an analyst to answer five questions a day, and now people are asking 100 questions, and the AI bots can answer 95 percent of them, that leaves the analyst to answer…five questions a day.

Second, more questions mean more follow-up questions. They mean more chances for things to look out of place, and more opportunities for there to be inconsistencies that need to be reconciled. So long as some numbers look wrong—and even the right numbers sometimes look wrong—people are going to want someone with “data” in their title to double-check them.²

So how do teams avoid that trap? Well! For better or for worse, AI is getting pretty good at figuring stuff out. Anthropic’s Fable 5.1, out on Tuesday, “found the cause of a rare crash in [one of the world’s largest hedge fund’s] internal systems that none of its engineers…had been able to explain after several years of trying.” OpenAI’s Astra, out yesterday, “improved a [mathematical finding about prime numbers] that had remained unchanged for more than 80 years.”

And specifically—also for, uh, better or for worse—AI seems particularly good at figuring stuff out when a bunch of different AI agents work on it together. The OpenAI agents that broke into Hugging Face were successful, in large part, because so many of them were working together to experiment with new ideas. “Roughly 700 agents in total actively participated in the attack on Hugging Face.”

Though AI still probably can’t go and find insights entirely on its own—things are interesting because of their relationship to us; “insight isn’t the data; it is what’s not in our heads,” and we can never write down everything in our heads—you can imagine something close. People offer their theories. Agents spider off to research them, and come back with their ideas. People nudge, steer, and give their feedback, like a CEO flipping through a deck of slides and handwaving at what looks right or wrong. The “swarm” returns to their desks, pulls on every thread and hunts down every clue, and comes back with more findings. It could work.

All we need are some tools to help the agents do their work. They need to be able to check out slices of data from a centralized store. They need to be able to load that data into a computational sandbox. They need to perform some calculations on it. They need ways to make some charts, and write up what they find. They need to save all of those things into a single immutable snapshot. That snapshot needs to be easy to store and share. It needs to be easy for other agents to find and read. They need to be able to pick up where the prior agent left off, and advance the work forward. And all the sandboxes need to be stored in an organized library that can accumulate more and more findings.

On one hand, that sounds like a huge new expanse of technology—a million agent sandboxes! A portable computation engine! Precise snapshots! Limitless copies! Swarms of agents all working in chaotic coordination! For us to have that, some people need to leave their jobs again, build new tools, and move us away from the crude world of SaaS apps and towards scalable agentic infrastructure, or something.

On the other hand, that’s just…Excel?

Excel is a self-contained computational sandbox. It contains built-in methods for manipulating and displaying data. It is very easy to snapshot (you save the file). It is very easy to duplicate (you copy the file). It is very easy to understand (you open the file). It is very easy to share (you email the file). It is very easy to store (you put the file in a folder). And if AI is good at working with anything, it’s good at working with files.

Anyway, the right thing here probably isn’t literally Excel. But the next generation of analytics tools—or all tools, really—might look more like a file than a fancy SaaS product. It could be a standardized collection of open-source assets: A DuckDB database, a lightweight transformation framework that manipulates that data with code,³ a few markdown docs with embedded queries and visualizations, and some glue that lets you execute everything—import data into DuckDB, run the transformation pipeline, build the charts, render the markdown, spit out a webpage.⁴ Dozens of agents—Claude Code, Codex, Cursor, anything—could copy it; read it; cheaply explore to their hearts’ content by working against the local data in the file; and then record their findings, save the file, and put it back in the folder. And the next agents that come along can scan through the files, read the notes, and keep going.

To the extent the analysts have done “strategic exploratory work,” that’s usually how it’s happened—across dozens of slowly advancing files, one finding stacked on top of the other, until, after days or weeks or months of poking, something useful falls out. So why would AI do it any differently?

We always wanted to find interesting insights. It would be fitting if, in the end, at long last, Excel really was how we did it.

1. If you’re an optimistic analyst, you do this because you want to be Jonah Hill, uncovering secret ways to win. If you’re a pessimistic analyst, you do this because you want to be Wharton Spock, telling everyone how they’re the ones who are hopelessly optimistic, and everything is actually way worse than anyone could possibly imagine.

2. You could also probably argue that data teams aren’t really designed to do “more strategic work” anymore. They shot for the stars, landed on the moon—building dashboards and stuff—and then colonized it. They’re pretty moon-native now. After years of trying to help other people answer questions, a lot of teams might not necessarily remember how to ask their own.

3. A tiny dbt? A dbtini?

4. This is fairly similar to a Python notebook or an R Markdown file, but with data and a runtime engine attached. If you share a Python notebook with an agent, it’s an instruction for how to work with data. If you share an Excel file, it’s everything: the data and the instructions. And popular things are “sandboxable” things.