“You can save it, and know, with total confidence, that when you open it tomorrow, next week, or next year, it’ll be the same as it was when you last left it.” This is 100% why excel exists and will continue to exist. And the people that most want this functionality are the company data historians or what most people call accountants. 😁.
Yes - I remember my initial shock learning this fact. I was so pumped showing accounting the almost live data... they were like - where is the excel download button.
That's probably kinda true, though I will say, at least from my experience, a lot of people intuitively understand *and want* the analog version of version control (a bunch of different saved files) much better than the proper, formal version.
I really like this way of putting it. For all the "modern data stack in a box" stuff, excel is actually a data stack in a single file. And that's... pretty impressive.
I've designed/built distributed graph database systems used by enterprises and governments. But the most valuable work I've done might have been the two or three "here, let me enable your critical business functionality with this one complex Excel file" solutions. I did it that way because it was cheapest/fastest/easiest way to solve a problem which didn't need any more complexity. Its terrible b/c of the lack of source code control, lack of testing, etc. But in some cases it's sufficient, and that's good enough
I remain just so confused by the persistence of the “Excel can’t connect to live data” meme. And I’m not even saying that Excel’s power query live data connection implementation is good (or bad!). Just that it plainly exists and has existed for like a decade. It’s like everyone in modern data stack land is unaware that there have been updates since the physical CD version of Excel 2007.
So I had a footnote that I cut on this line - "You can save it, and know...it’ll be the same as it was when you last left it" - that said, technically, this isn't at all true, because 1) Excel can connect to databases and other live sources, 2) you can also have connected workbooks or automatically updating macros, or 3) someone else can just change the file. But I'd generally agree with Ethan that most Excel files don't use it. My point wasn't that Excel can't do it, but that's not really how Excel is used, even in corporate environments. Compare that to something like Tableau, which I'd guess is more often than not connected to live data in corporate environments.
The interesting question to me there is, if live connections were more popular, would Excel get dragged into building more BI features like dashboards and reports?
My take on that would be no, mainly because I think Microsoft has pretty much solved for those BI needs already with Power BI. I guess maybe it’s not widely known just how integrated the backends of modern Excel and Power BI are, but it’s real easy to model out a dataset in PBI, publish it to the power platform, schedule recurring updates, and then let your users either access it directly in drillable pivot tables in Excel or build PBI reports on top of it. And a super helpful corollary of this situation is it defangs the “I want to export the data to Excel” users who would kneecap any other BI approach.
Yeah, as I understand it, that's basically where power BI came from, was as a kind of Excel-like interface that's actually a proper BI tool. Which I guess might be the answer to everyone else's problem of turning into a BI tool - the solution isn't to become a BI tool, but to build something totally new that's kinda like your first tool. (Which is great, if you happen to have a few thousand people sitting around looking forwards something to do.)
Comment on the AI section. AI is chaotic right now. LLMs are this weird innovation that are by default maybe 80% useful for a million different things. And each startup is trying to find one thing and make LLMs as close to 100% just for their thing as possible. For example seek.ai is working on “generative AI DB queries”. Haven’t used it personally - but have heard it’s pretty cool. Little do they know they are building a BI tool. 😁. Anyways the scary part to me is most people are using OpenAI as their backend and as we saw this week OpenAI can wipe out your startup in a minute with a product change or update. Just wait for them to announce a slack integration...
Yeah, the losing game to me is building things that are roughly AI infrastructure (ie, stuff to create your own chatbot or whatever), because I still think "OpenAI as AWS" is still basically right: https://benn.substack.com/p/the-public-imagination (though I think this missed somewhat in that it mostly predicted OpenAI would build lots of media-type-to-media-type services, whereas it seems like they're more focused on tools for helping people build and tune their own AIs).
That said, I do think that app builders are still relatively safe. To build a LLM analyst bot thing, there's a lot of stuff that has to built on top of the LLM itself, and I don't really see OpenAI doing that work. At worst, I could see them making it easier for people to build that stuff, and making the "effort" moat shallower. But, that's also pretty similar to AWS, where it made it easier for everyone to build web-based software, but didn't make that software any less lucrative if you did it well.
Ya... some kind of infra+marketplace like AWS seems very much the plan to me. LLM AI stuff is so weird to me. It feels a mile wide and inch deep right now and feels hard to predict where there are layers of loose soil vs layers of solid rock to drill through as far as the applications.
This seems like the really weird thing to me. It's like, programming a person, where you think it'll probably do what you want, but you can never *really* know.
You've written multiple times that data companies end up being dragged into the BI black hole. But why is that such a bad thing? If people ask for dashboards, we build them dashboards. Seems like the right thing to do. What am I missing here?
My answer is basically this. I think that a lot of companies start thinking they don't have to build BI, and end up slowly drifting into it. Which isn't, like, capital-B Bad, but it can make life very hard for the company because 1) it's a very crowded market, and 2) building BI tools is very very big lift. There are dozens of "table-stakes" features, and it takes a decade-plus to build something that's moderately mature. If you're a company that's planned your roadmap and GTM motions around the idea that you can build a couple differentiating features and be off to the races, and then you find yourself actually needing to slog through a messy market and a huge todo list of features, a whole lot can go sideways.
I would add, I think companies drift toward BI because there is always some data wall that gets hit - be it collaboration (BI tools aren't great here either to be fair), Spreadmarts are getting to unwieldly, there's always a demand somewhere that Excel is seen as not able to manage, even though export to Excel is always the most popular feature. I don't think there's a single tool on the market in BI that will be the all satisfying, killer of other BI tools, and in reality our industry has to rely on this "evolving" space, as it's what we do for a living. So you end up in this continuous loop of innovation, whilst Excel is still a prominent feature in any given BI tool, be it the ability to use it as a data source, or export whatever beautiful dashboard you've built into the meta data components that underlie it.
I used to get figurative hives when it comes to the prevalent use of Excel, but in recent years, I've accepted it's importance in the BI continuum.
Yeah, the analogy I've used with this is that BI is like market for note-taking apps, where everyone has their own preferences for how they want to do it. You can never really consolidate that market, because it's all opinion-driven by what people like, and not something where people just need to check a list of functionality boxes. Everyone's wants 50 features from a possible set of 1,000 - it's just 50 different ones for every person (except export to excel; that one's on everyone's list).
The Snowflake optimization person fundamentally misunderstands the nature of employment. If he truly wants to "privatize the gains", he needs the start his own business and reap the gains through equity ownership.
So that is one sorta interesting thing here that occurred to me - startups in Silicon Valley are built almost entirely on a system of privatized gains and socialized losses. Founders take next to no personal financial risk, but have very high upside. I'm not sure how we pulled that off so well, and in such a socially acceptable way.
“You can save it, and know, with total confidence, that when you open it tomorrow, next week, or next year, it’ll be the same as it was when you last left it.” This is 100% why excel exists and will continue to exist. And the people that most want this functionality are the company data historians or what most people call accountants. 😁.
Yeah, I think a lot of data companies see "always up-to-date data" as a benefit, when, for a lot of people, it's a bug.
Yes - I remember my initial shock learning this fact. I was so pumped showing accounting the almost live data... they were like - where is the excel download button.
The first most used feature in every BI tool is connecting it to a database. The second most used one is exporting results to Excel.
Fascinating. Our company is built around reliable versioning with user-friendly snapshots. Maybe there’s an opening.
Interesting. Maybe I should give you and demo and see whether it “counts” :-}
If you got videos or something, I can look, but I also, don't trust my opinions on these things.
That's probably kinda true, though I will say, at least from my experience, a lot of people intuitively understand *and want* the analog version of version control (a bunch of different saved files) much better than the proper, formal version.
Ya - and want to be able to adjust the data manually - cell by cell
Love the blog almost as much for the links and footnotes as the info in the blog itself
Excellent insight into Excel. I've long thought that it's brilliance is b/c it combined 3 critical functional areas:
A data layer - addressable data (cells, lookups, pivot tables for the more sophisticated), and schema-lite, ...
A compute layer - basic math operations, macros, full coding language support, ...
A presentation layer - cell formatting + all the charts
The pièce de résistance is that it is fully portable and basically works the same from machine to machine, version to version
I really like this way of putting it. For all the "modern data stack in a box" stuff, excel is actually a data stack in a single file. And that's... pretty impressive.
I've designed/built distributed graph database systems used by enterprises and governments. But the most valuable work I've done might have been the two or three "here, let me enable your critical business functionality with this one complex Excel file" solutions. I did it that way because it was cheapest/fastest/easiest way to solve a problem which didn't need any more complexity. Its terrible b/c of the lack of source code control, lack of testing, etc. But in some cases it's sufficient, and that's good enough
I like this new format of covering multiple topics. Very Matt Levine-esque 😊 who is also my other favorite writer.
quarterly investor update will be
"we are not a bi tool"
"we are not a bi tool"
"we are not a bi tool"
"we are not a bi tool"
"we are not a bi tool"
"we are not a bi tool"
"we are not a bi tool"
"we are not a bi tool"
"we are reinventing bi"
I remain just so confused by the persistence of the “Excel can’t connect to live data” meme. And I’m not even saying that Excel’s power query live data connection implementation is good (or bad!). Just that it plainly exists and has existed for like a decade. It’s like everyone in modern data stack land is unaware that there have been updates since the physical CD version of Excel 2007.
what percent of excel documents do you think use it?
0.001%?
0.000001%?
Do you mean like people are reading old Excel documentation that pre-dates power query?
So I had a footnote that I cut on this line - "You can save it, and know...it’ll be the same as it was when you last left it" - that said, technically, this isn't at all true, because 1) Excel can connect to databases and other live sources, 2) you can also have connected workbooks or automatically updating macros, or 3) someone else can just change the file. But I'd generally agree with Ethan that most Excel files don't use it. My point wasn't that Excel can't do it, but that's not really how Excel is used, even in corporate environments. Compare that to something like Tableau, which I'd guess is more often than not connected to live data in corporate environments.
The interesting question to me there is, if live connections were more popular, would Excel get dragged into building more BI features like dashboards and reports?
My take on that would be no, mainly because I think Microsoft has pretty much solved for those BI needs already with Power BI. I guess maybe it’s not widely known just how integrated the backends of modern Excel and Power BI are, but it’s real easy to model out a dataset in PBI, publish it to the power platform, schedule recurring updates, and then let your users either access it directly in drillable pivot tables in Excel or build PBI reports on top of it. And a super helpful corollary of this situation is it defangs the “I want to export the data to Excel” users who would kneecap any other BI approach.
Yeah, as I understand it, that's basically where power BI came from, was as a kind of Excel-like interface that's actually a proper BI tool. Which I guess might be the answer to everyone else's problem of turning into a BI tool - the solution isn't to become a BI tool, but to build something totally new that's kinda like your first tool. (Which is great, if you happen to have a few thousand people sitting around looking forwards something to do.)
Comment on the AI section. AI is chaotic right now. LLMs are this weird innovation that are by default maybe 80% useful for a million different things. And each startup is trying to find one thing and make LLMs as close to 100% just for their thing as possible. For example seek.ai is working on “generative AI DB queries”. Haven’t used it personally - but have heard it’s pretty cool. Little do they know they are building a BI tool. 😁. Anyways the scary part to me is most people are using OpenAI as their backend and as we saw this week OpenAI can wipe out your startup in a minute with a product change or update. Just wait for them to announce a slack integration...
*a push a button slack integration that makes all your gpts available to your entire team.
Yeah, the losing game to me is building things that are roughly AI infrastructure (ie, stuff to create your own chatbot or whatever), because I still think "OpenAI as AWS" is still basically right: https://benn.substack.com/p/the-public-imagination (though I think this missed somewhat in that it mostly predicted OpenAI would build lots of media-type-to-media-type services, whereas it seems like they're more focused on tools for helping people build and tune their own AIs).
That said, I do think that app builders are still relatively safe. To build a LLM analyst bot thing, there's a lot of stuff that has to built on top of the LLM itself, and I don't really see OpenAI doing that work. At worst, I could see them making it easier for people to build that stuff, and making the "effort" moat shallower. But, that's also pretty similar to AWS, where it made it easier for everyone to build web-based software, but didn't make that software any less lucrative if you did it well.
Ya... some kind of infra+marketplace like AWS seems very much the plan to me. LLM AI stuff is so weird to me. It feels a mile wide and inch deep right now and feels hard to predict where there are layers of loose soil vs layers of solid rock to drill through as far as the applications.
This seems like the really weird thing to me. It's like, programming a person, where you think it'll probably do what you want, but you can never *really* know.
Yes! So weird. I think this might make the new hot skill management and the hot degree liberal arts.
You've written multiple times that data companies end up being dragged into the BI black hole. But why is that such a bad thing? If people ask for dashboards, we build them dashboards. Seems like the right thing to do. What am I missing here?
It isn’t “bad” exactly, but it is dangerous because:
- it happens by accident
- you end up competing with every other BI tool
- BI tools are notoriously unsatisfying
If you want to play that game, fine, but you better own it up front.
My answer is basically this. I think that a lot of companies start thinking they don't have to build BI, and end up slowly drifting into it. Which isn't, like, capital-B Bad, but it can make life very hard for the company because 1) it's a very crowded market, and 2) building BI tools is very very big lift. There are dozens of "table-stakes" features, and it takes a decade-plus to build something that's moderately mature. If you're a company that's planned your roadmap and GTM motions around the idea that you can build a couple differentiating features and be off to the races, and then you find yourself actually needing to slog through a messy market and a huge todo list of features, a whole lot can go sideways.
I would add, I think companies drift toward BI because there is always some data wall that gets hit - be it collaboration (BI tools aren't great here either to be fair), Spreadmarts are getting to unwieldly, there's always a demand somewhere that Excel is seen as not able to manage, even though export to Excel is always the most popular feature. I don't think there's a single tool on the market in BI that will be the all satisfying, killer of other BI tools, and in reality our industry has to rely on this "evolving" space, as it's what we do for a living. So you end up in this continuous loop of innovation, whilst Excel is still a prominent feature in any given BI tool, be it the ability to use it as a data source, or export whatever beautiful dashboard you've built into the meta data components that underlie it.
I used to get figurative hives when it comes to the prevalent use of Excel, but in recent years, I've accepted it's importance in the BI continuum.
Yeah, the analogy I've used with this is that BI is like market for note-taking apps, where everyone has their own preferences for how they want to do it. You can never really consolidate that market, because it's all opinion-driven by what people like, and not something where people just need to check a list of functionality boxes. Everyone's wants 50 features from a possible set of 1,000 - it's just 50 different ones for every person (except export to excel; that one's on everyone's list).
Excellent post! ✨
The Snowflake optimization person fundamentally misunderstands the nature of employment. If he truly wants to "privatize the gains", he needs the start his own business and reap the gains through equity ownership.
So that is one sorta interesting thing here that occurred to me - startups in Silicon Valley are built almost entirely on a system of privatized gains and socialized losses. Founders take next to no personal financial risk, but have very high upside. I'm not sure how we pulled that off so well, and in such a socially acceptable way.
*chants* capped revenue share
So, I guess my other question about that is, isn't that basically a company-performance-based bonus, like what people in finance get paid?