I totally agree, my path to a career in data started in excel, then excel + vba, then access, then oracle plsql, snuck into JavaScript and python, and building full stack apps. All beginning from the joy of the hesitant but exciting press of the “go button” on a simple script in excel. I’ve helped others on this journey, and it’s fun to see the smile as they understand the potential of the computer for the first time
Benn I love this. I work in a very Excel-focused environment: I'm in a data team working alongside a small army of economic researchers who speak Excel & Powerpoint. Over the last few years my opinion on Excel has gone through something like the Gartner hype cycle, from the peak of possibility, through the trough of disillusionment, and finally on to the plateau of acceptance.
My general feeling on it is in fact both of your viewpoints - the best and worst thing about Excel is that it can do everything. I have seen horrors. However, it is incredibly empowering for smart people who know what they want to achieve and don't want to deal with the orgy of frustration that comes with figuring out how to get started with Python ("What does 'No module named pandas' mean?") and in all honesty don't have the time or inclination to learn how to write code. They have more valuable things to do.
The role I (and my team) fills is often a kind of guide to making Excel behave well in the context of a larger data process. We help people get data into Excel from various databases & APIs, then define some standards on what the output of some spreadsheeting should look like so that it can feed into some other downstream process. A kind of data sherpa role. And it works! We end up with analysts who can work in the way they're comfortable, can flex the middle part of their process as much as they like, and then produce results that fit into databases and get surfaced to paying customers in products built by 'real' developers.
Also I am 100% with you on the trouble with AWS. I came from that Windows & Excel background and started working more with Python & SQL and the barrier to entry to working in the cloud is enormous. You need to learn how to write the kind of code that works in the cloud, what event-based architecture is, how to provision infrastructure, how to use Linux, what CI/CD is, what containers are and why they're useful... the list goes on.
Agreed on Excel being really powerful for those sorts of folks. Though I would disagree on one point: "It is incredibly empowering for smart people who...don't have the time or inclination to learn how to write code."
But they DID learn how to write code, in Excel. It's just not writing code the way it's usually thought of. And that's what makes Excel so unique to me. It lets people write code and create programs without even realizing that they're doing it. Which raises the question for me - is there a way to do the same thing, but for a wider range of jobs than what Excel enables?
Re: barrier to entry — it definitely sucks if you come from the Excel/Access world to building things with Python.
I find Google Cloud Function’s UI to be less daunting than AWS Lambda, so that’s helped.
I now do lots of development using a platform called Anvil (https://anvil.works) because it abstracts away the devops work while letting you build full-fledged web apps using only Python. 10/10, I recommend.
Ah, this is cool. This is close to what I want, but instead of apps with nothing but Python, it's more like, scripts for automating things with nothing but Python.
Music to my ears, Ben! You know, there are many people whose entire careers are driven by the line of thoughts you mentioned here--billions of people programming computers without coding. I've been on this path for a long time and it's been worth it! I wish there was a community of people where we could discuss these ideas and and try them. Anyone knows of such community?
1) Is this sort of like VBA, the built-in Excel language that you can "macro" tasks to make them endlessly adjustable.
2) Is this like RPA, the "low-code" solution that creates logical workflows on any computer UI use case based upon your task-steps?
To me, those are the two most similar sorts of things.
But I also want to push-back and say that there are advanced spreadsheet users and less capable users.
It is easy to say Excel is great and mighty and powerful, and it is true, but the detractors are 1000% correct. A great Excel user is like a great programmer, and many of them are well-compensated (think consultants and accountants). A bad one... Well... They are why some analytics professionals despise spreadsheets.
And the more powerful the tool, the more opportunities to misuse it.
(Also, TBH, I think a real opportunity to bridge the gap between Excel lovers and haters would be better tracking of the history of data transformations in Excel. This would allow mistakes to be audited more easily, and allow better transformation/auditing of Excel processes into code. G-Suite today often tracks the full history, but imagine being able to understand processes as gscript code and auditing problems rather than having to check each historical log as to what went wrong. Then imagine being able to plug this into Github for established spreadsheet processes for end to end tracking.)
Is it VBA? Kinda, though VBA is very much built for power users. My question would be, what would it look like to build something where VBA is the central experience, and made for less advanced users?
Is it RPA? Yes, ish. The difference between this and RPA is 1) they're very enterprise/IT oriented, and 2) they tend to bury the code. Is there a consumer-focused version of RPA that leans into a code-based way of building the automations, as opposed to one that tries to be entirely point and click?
VBA is currently the easiest language to go from 0 to doing something useful. The record function means you can just adjust your recordings to create new code.
My guess is that Gscript, by having modern syntax, will help Sheets users immensely.
And that is where I am confused. VBA is a code-based way of building automations. Like an RPA, it records all steps. Unlike RPA, you can edit and modify code. However, you could literally design a task automation entirely by point and click within VBA.
And my apologies for missing a nuance. My first professional coding was VBA. Way easier than Python. In Python, you have to learn libraries & syntax to get started. In VBA, if you don't know how to code something, you just record doing it in Excel and modify the code created as needed.
I think VBA is the right idea; my main point is that it's kind of buried experience that feels like it's for power users. I think it'd be cool if Excel built Excel in a way where VBA was as central as the standard formula language.
It's mostly a joke. I find Slack really problematic to use as the primary tool for communication at work for a handful of reasons. The ideal tool to me would be something in between Slack and email.
great article! In my own observations, both excel and your typical source code project contain the same two elements, data and the logic that interoperates with it. They are intertwined. However with excel, we see the data front and center, with the logic hiding in the background (i.e. behind the cell) whereas in software programs, emphasis is the reverse. The data structures used are often hidden (i.e. information hiding) and code/interfaces are front and center. each is emphasizing one side of the coin over the other. Seems like numbers are more familiar than algorithms to most people.
What are your thoughts on me powerautomate in this space? It does many of the things I often do in excel (collect and mashup datasets, send custom reports to the right people, etc). I work in a large finance org, and they’ve banned anyone not in the central team from writing power apps. Which makes me think there are enough tech professionals who still think excel is evil, and we shouldn’t let anyone who isn’t already a coder anywhere near anything that looks like coding (and in their view I know this includes excel)
Although thinking about it, it’s not quite what you’re thinking. If you do come up with something amazing getting people to use it is the first, almost smaller problem, than IT people letting people use it in anger ;-)
Yeah, someone else asked about the RPA tools like power automate and UiPath. I think those are close, but they seem to be very enterprise/IT oriented, and they seem to bury the code. My question would be, is there a consumer-focused version of those tools that also leans into (simple) code?
Another one to check is Pulumi. Even more ambitious in scope. I had fun working with Serverless on a very simple project, but have not tried Pulumi yet.
I was going to mention SAM : https://github.com/aws/aws-sam-cli I was able to setup a python hello world via Sam cli on cloud nine, in under ten minutes from restarting my pc
I eventually got things running with SAM, but it definitely didn't take me just 10 minutes.
(Which, I think is part of the point. For engineers, I think something like this is probably pretty easy. For non-engineers like me, it's a pretty tedious exercise, which often keeps people from doing it at all. What I want is a way for someone to set up a lambda without needing any prior engineering knowledge.)
I wrote a bit about this here almost 2 years ago (woah): https://www.forbes.com/sites/googlecloud/2020/03/11/beyond-spreadsheets/. Basically, spreadsheet pros are already coders (in my view), and it's weird (to me) that tech puts them on the other side of some wall and calls them "business users". It should be easier for these folks to be able to use more and better coding tools, like notebooks and such. Code is good!
Oh nice, this post is great. And I really like emphasizing the people who use Excel do write code, just not in the way that traditional engineers do.
The notebook-as-Excel replacement is interesting. I'd think that'd be a really tough sell, which may actually get at another way to think about this: What would you have to build to convince the "pry it from my cold dead hands" Excel people to use something like notebooks?
There was also an interesting tool on PH the other day that lets you view Gsheet formulas as code blocks that could represent an interesting approach. (I’ll edit with a link when I find it).
Hex's graph is a bit of step in that direction, though, yeah, it's more focused on companies, and (like any notebook) it's still raw Python, instead of Python extending the UI.
The other thing is that notebooks are often focused on data analysis, which isn't quite what I'd imagine this to be for. It's less about crunching numbers, and more about just making computers do stuff for you.
Interesting. These seem little more like Retool to me. One other element to all of this is lot of the tools in this "easy tool space" are clearly building something to sell to companies, which also changes the approach a good bit. What I really want is a straight consumer product that doesn't assume you've got a database, a place to host anything, or very much money.
This is a very different problem to me though. Combining a bunch of workplace apps together is nice, but what I want is a way for more people to be able to easy automate stuff with code. That's less about adding more features to Excel, or combining it with Word or making it work with databases; it's more about learning from how Excel made us all able to automate math with code, and trying to extend that to operations other than arithmetic.
What you are able to do with Notion databases is for many people exactly what they need: https://www.notion.so/help/category/databases. And there is an API now, this will probably open up a lot more use cases.
I suspect that's gets at part of the problem. Building something like this would be pretty ambitious, and there's a lot of offramps for shorter things that could make money a lot faster. I think you'd need a mix of patience and a focus on building for a consumer (eg, not some research project), which is probably a tough combination.
I totally agree, my path to a career in data started in excel, then excel + vba, then access, then oracle plsql, snuck into JavaScript and python, and building full stack apps. All beginning from the joy of the hesitant but exciting press of the “go button” on a simple script in excel. I’ve helped others on this journey, and it’s fun to see the smile as they understand the potential of the computer for the first time
woah - this is pretty much exactly what we're building... product not publicly launched yet, but v keen to pick your brains and give you a demo?
Sure! Feel free to reach out over any email, twitter, or linkedin: https://benn.substack.com/about
Benn I love this. I work in a very Excel-focused environment: I'm in a data team working alongside a small army of economic researchers who speak Excel & Powerpoint. Over the last few years my opinion on Excel has gone through something like the Gartner hype cycle, from the peak of possibility, through the trough of disillusionment, and finally on to the plateau of acceptance.
My general feeling on it is in fact both of your viewpoints - the best and worst thing about Excel is that it can do everything. I have seen horrors. However, it is incredibly empowering for smart people who know what they want to achieve and don't want to deal with the orgy of frustration that comes with figuring out how to get started with Python ("What does 'No module named pandas' mean?") and in all honesty don't have the time or inclination to learn how to write code. They have more valuable things to do.
The role I (and my team) fills is often a kind of guide to making Excel behave well in the context of a larger data process. We help people get data into Excel from various databases & APIs, then define some standards on what the output of some spreadsheeting should look like so that it can feed into some other downstream process. A kind of data sherpa role. And it works! We end up with analysts who can work in the way they're comfortable, can flex the middle part of their process as much as they like, and then produce results that fit into databases and get surfaced to paying customers in products built by 'real' developers.
Also I am 100% with you on the trouble with AWS. I came from that Windows & Excel background and started working more with Python & SQL and the barrier to entry to working in the cloud is enormous. You need to learn how to write the kind of code that works in the cloud, what event-based architecture is, how to provision infrastructure, how to use Linux, what CI/CD is, what containers are and why they're useful... the list goes on.
Agreed on Excel being really powerful for those sorts of folks. Though I would disagree on one point: "It is incredibly empowering for smart people who...don't have the time or inclination to learn how to write code."
But they DID learn how to write code, in Excel. It's just not writing code the way it's usually thought of. And that's what makes Excel so unique to me. It lets people write code and create programs without even realizing that they're doing it. Which raises the question for me - is there a way to do the same thing, but for a wider range of jobs than what Excel enables?
Re: barrier to entry — it definitely sucks if you come from the Excel/Access world to building things with Python.
I find Google Cloud Function’s UI to be less daunting than AWS Lambda, so that’s helped.
I now do lots of development using a platform called Anvil (https://anvil.works) because it abstracts away the devops work while letting you build full-fledged web apps using only Python. 10/10, I recommend.
Ah, this is cool. This is close to what I want, but instead of apps with nothing but Python, it's more like, scripts for automating things with nothing but Python.
Music to my ears, Ben! You know, there are many people whose entire careers are driven by the line of thoughts you mentioned here--billions of people programming computers without coding. I've been on this path for a long time and it's been worth it! I wish there was a community of people where we could discuss these ideas and and try them. Anyone knows of such community?
Sadly, I don't. (I mostly just yell out here in the void; I have no idea where the cool people actually hang out.)
Two random thoughts:
1) Is this sort of like VBA, the built-in Excel language that you can "macro" tasks to make them endlessly adjustable.
2) Is this like RPA, the "low-code" solution that creates logical workflows on any computer UI use case based upon your task-steps?
To me, those are the two most similar sorts of things.
But I also want to push-back and say that there are advanced spreadsheet users and less capable users.
It is easy to say Excel is great and mighty and powerful, and it is true, but the detractors are 1000% correct. A great Excel user is like a great programmer, and many of them are well-compensated (think consultants and accountants). A bad one... Well... They are why some analytics professionals despise spreadsheets.
And the more powerful the tool, the more opportunities to misuse it.
(Also, TBH, I think a real opportunity to bridge the gap between Excel lovers and haters would be better tracking of the history of data transformations in Excel. This would allow mistakes to be audited more easily, and allow better transformation/auditing of Excel processes into code. G-Suite today often tracks the full history, but imagine being able to understand processes as gscript code and auditing problems rather than having to check each historical log as to what went wrong. Then imagine being able to plug this into Github for established spreadsheet processes for end to end tracking.)
Is it VBA? Kinda, though VBA is very much built for power users. My question would be, what would it look like to build something where VBA is the central experience, and made for less advanced users?
Is it RPA? Yes, ish. The difference between this and RPA is 1) they're very enterprise/IT oriented, and 2) they tend to bury the code. Is there a consumer-focused version of RPA that leans into a code-based way of building the automations, as opposed to one that tries to be entirely point and click?
Power Excel users, maybe???
VBA is currently the easiest language to go from 0 to doing something useful. The record function means you can just adjust your recordings to create new code.
My guess is that Gscript, by having modern syntax, will help Sheets users immensely.
And that is where I am confused. VBA is a code-based way of building automations. Like an RPA, it records all steps. Unlike RPA, you can edit and modify code. However, you could literally design a task automation entirely by point and click within VBA.
And my apologies for missing a nuance. My first professional coding was VBA. Way easier than Python. In Python, you have to learn libraries & syntax to get started. In VBA, if you don't know how to code something, you just record doing it in Excel and modify the code created as needed.
I think VBA is the right idea; my main point is that it's kind of buried experience that feels like it's for power users. I think it'd be cool if Excel built Excel in a way where VBA was as central as the standard formula language.
Hi Ben - What did you mean by "Slack, but more like email." (footnote 7)?
It's mostly a joke. I find Slack really problematic to use as the primary tool for communication at work for a handful of reasons. The ideal tool to me would be something in between Slack and email.
Got it. Thanks for the context!
Most people don’t understand that this is what the Salesforce Platform actually is.
great article! In my own observations, both excel and your typical source code project contain the same two elements, data and the logic that interoperates with it. They are intertwined. However with excel, we see the data front and center, with the logic hiding in the background (i.e. behind the cell) whereas in software programs, emphasis is the reverse. The data structures used are often hidden (i.e. information hiding) and code/interfaces are front and center. each is emphasizing one side of the coin over the other. Seems like numbers are more familiar than algorithms to most people.
That makes sense to me. Someone made a similar point on Twitter too, arguing that that's a big reason why Excel has been so successful: https://twitter.com/SriniKadamati/status/1492308599653154817
Thanks Benn, great read as usual!
This seems appropriate https://youtu.be/UBX2QQHlQ_I
:D
What are your thoughts on me powerautomate in this space? It does many of the things I often do in excel (collect and mashup datasets, send custom reports to the right people, etc). I work in a large finance org, and they’ve banned anyone not in the central team from writing power apps. Which makes me think there are enough tech professionals who still think excel is evil, and we shouldn’t let anyone who isn’t already a coder anywhere near anything that looks like coding (and in their view I know this includes excel)
Although thinking about it, it’s not quite what you’re thinking. If you do come up with something amazing getting people to use it is the first, almost smaller problem, than IT people letting people use it in anger ;-)
Yeah, someone else asked about the RPA tools like power automate and UiPath. I think those are close, but they seem to be very enterprise/IT oriented, and they seem to bury the code. My question would be, is there a consumer-focused version of those tools that also leans into (simple) code?
https://www.serverless.com/ solved some of the AWS Lambda issues you described.
Oh nice, I did not know this was a thing. I'll check it out..
Another one to check is Pulumi. Even more ambitious in scope. I had fun working with Serverless on a very simple project, but have not tried Pulumi yet.
https://www.pulumi.com/serverless/
https://www.pulumi.com/docs/intro/vs/serverless/
Huh, it's almost like Heroku, but for serverless stuff? That's cool, but does indeed seem ambitious.
I was going to mention SAM : https://github.com/aws/aws-sam-cli I was able to setup a python hello world via Sam cli on cloud nine, in under ten minutes from restarting my pc
I eventually got things running with SAM, but it definitely didn't take me just 10 minutes.
(Which, I think is part of the point. For engineers, I think something like this is probably pretty easy. For non-engineers like me, it's a pretty tedious exercise, which often keeps people from doing it at all. What I want is a way for someone to set up a lambda without needing any prior engineering knowledge.)
That is fair! I’ve been playing with them for a while, and I was following a really nice tutorial here: https://github.com/walmsles/lambda-powertools-coding-workshop which I admit again is aimed at devs
Great post! It reminds me of the “beyond spreadsheets” ideas kicking around for awhile in my brain...
...please go on
I wrote a bit about this here almost 2 years ago (woah): https://www.forbes.com/sites/googlecloud/2020/03/11/beyond-spreadsheets/. Basically, spreadsheet pros are already coders (in my view), and it's weird (to me) that tech puts them on the other side of some wall and calls them "business users". It should be easier for these folks to be able to use more and better coding tools, like notebooks and such. Code is good!
Oh nice, this post is great. And I really like emphasizing the people who use Excel do write code, just not in the way that traditional engineers do.
The notebook-as-Excel replacement is interesting. I'd think that'd be a really tough sell, which may actually get at another way to think about this: What would you have to build to convince the "pry it from my cold dead hands" Excel people to use something like notebooks?
I think Hex’s graph view is a step in the right direction (https://learn.hex.tech/docs/develop-logic/compute-model/graph-view) though, admittedly, this is still aimed at companies.
There was also an interesting tool on PH the other day that lets you view Gsheet formulas as code blocks that could represent an interesting approach. (I’ll edit with a link when I find it).
Hex's graph is a bit of step in that direction, though, yeah, it's more focused on companies, and (like any notebook) it's still raw Python, instead of Python extending the UI.
The other thing is that notebooks are often focused on data analysis, which isn't quite what I'd imagine this to be for. It's less about crunching numbers, and more about just making computers do stuff for you.
A couple of interesting takes on automation if you haven’t seen them: Wayscript (wayscript.com) and Airplane (airplane.dev)
Interesting. These seem little more like Retool to me. One other element to all of this is lot of the tools in this "easy tool space" are clearly building something to sell to companies, which also changes the approach a good bit. What I really want is a straight consumer product that doesn't assume you've got a database, a place to host anything, or very much money.
Lol anyone remember AlphaSheets?
I think Notion is also on this path.
Have they publicly done anything in this direction? It still seems to be a "combine all docs into one thing" thing to me.
Also read the 'origin' story, they clearly want Notion to be the Everything Tool. https://www.notion.so/about
This is a very different problem to me though. Combining a bunch of workplace apps together is nice, but what I want is a way for more people to be able to easy automate stuff with code. That's less about adding more features to Excel, or combining it with Word or making it work with databases; it's more about learning from how Excel made us all able to automate math with code, and trying to extend that to operations other than arithmetic.
What you are able to do with Notion databases is for many people exactly what they need: https://www.notion.so/help/category/databases. And there is an API now, this will probably open up a lot more use cases.
I suspect that's gets at part of the problem. Building something like this would be pretty ambitious, and there's a lot of offramps for shorter things that could make money a lot faster. I think you'd need a mix of patience and a focus on building for a consumer (eg, not some research project), which is probably a tough combination.