Oracle had much of the same thinking as far back as 20+ years ago (maybe more, but I wasn't around then...). So if we're on making comparison, I think the right comparison is that we're seeing the next iteration of Oracle's proprietary in-database products.
The question is how open will that implementation be? And will apps built for Snowflake work on GCP, AWS stacks? I suspect no. In the end, I am not sure this is as exciting as it sounds. I think we're just looking at a proliferation of in-database functionalities that are required for a mature enterprise database product. And the standard there for decades has been Oracle.
What we're actually probably going to see is a few iterations of implementations with relatively open standards. Something that will give Snowflake's competitors their own good ideas to make in-database app products. Then competition on in-database apps kicks off, and Snowflake will start making decisions. towards making App implementation much more proprietary, using custom languages totally incompatible with anything else.
I think the more interesting future is emergence of another Streamlit, but one like DBT that is able to raise >$300mil and build an an entire cloud platform around the idea of community contributed apps. That would be an interesting future.
Yeah, none of this is entirely novel. But small differences can make or break results. Why did Slack become Slack and Hipchat died? It wasn't really anything structural; they just made a better product and caught the market at the right time.
Would something like this work with Snowflake even though it didn't at Oracle? Certainly could. Most "new" tech is some old idea recut with new paradigms (same as before, but in the cloud! same as before, but selling to analysts and not DBAs!). Sometimes that change makes all the difference; sometimes, it flops just the same.
As for creating a platform around community apps, I don't see why that's not Snowflake. They're already where the tech lives; I'm not sure why you'd need another entity to host the community. Back to the iPhone analogy, that'd be like another company building the app store. I guess that's possible, but it makes a lot more sense for Apple to do it.
There is a difference b/w Snowflake and Apple in the iPhone analogy. The difference is small, but is important: monetization.
Apple Store is enabler for new Business Models, for new Businesses. The purchase of the app is just the beginning to the monetization. The app is the end-user interface, so it can do anything.
In the case of databases, I am not sure there is an enabler for new businesses. Say I am interested in the privacy app for emails. I think it is unlikely that a new business will born from a successful privacy app for emails on Snowflake. A more realistic scenario is that there is already a privacy vendor, which will build an integration for Snowflake.
The better comparison is with Chrome/Firefox Web Store of Extensions. Some succeed in monetizing, but the majority won't. The successful ones (Evernote, Loom) exist as standalone applications, and integrate with a browser in addition to the application itself. But even that is a misleading comparison. These extensions are embedded in my UI experience.
So Snowflake Store is primarily an ecosystem play, to get existing vendors with their own users, to make a bigger bet on Snowflake as a DW partner. For companies such as Mode, to create Snowflake-specific apps, doubling down on that partnership. Or to promote 3rd party apps as part of Mode-onboarding experience.
In that sense, this is not much different than the Looker marketplace. Powerful as an idea, but not an enabler of new businesses - just a way to integrate closer with an ecosystem.
Maybe what we really need is new “atomic unit” of data. Not tables. Not dashboards. Not even apps. Let’s call it an INS (for insight, insert, or inside joke). A true Data OS would make it easy for humans (or machines) produce, consume, and reuse INS without having worry about the complexity of formats, compute, or even platform.
Great thought piece. I'm with you on the utility of data apps, a vertical approach that delivers immediate business value versus a general horizontal BI tool that says "you can build anything you want, go for it."
One of my challenges in delivering data apps has been the fact that different customers have different data stacks, especially database preference. Perhaps this iSnowflake approach simplifies that, but does it then push them to a single vendor that data app creators sort of homogenize around?
Could an inverted approach that is database agnostic provide much of the same value? A lot of the data app utility is in the data transformation component - getting data from the disparate systems of all the potential data app customers to be used by a generic data app UI. For example, deliver a data app with dbt while paying special attention to keeping it database agnostic, then build your data app UI (Mode, custom app, Tableau, etc.) based on those models. Free, agnostic, open source.
For many, is there value in a $1500 phone that is roughly the same as a $500 phone? I know this is blasphemy, but how much of that $1500 is sales and marketing versus real value? Also, how do marketplace app creators feel about their end of the deal?
This is exactly the approach we are taking at Starburst. The notion that a single data platform (i.e. Snowflake) should own all your data and now all your data apps is counter to good engineering practices and promotes vendor lock-in in our view. Why should Snowflake's product roadmap dictate the way an analytics engineer thinks about their development framework?
Optionality and the ability to choose the right data stack for the right use case is foundational to creating apps that have the right performance, complexity, cost tradeoffs. Additionally, as a data app evolves or gains adoption over time, it may have different needs that would be best served by broader variety of database architectures and vendors in the future.
While starting with everything on a single vendor is likely going to be easier in the short term for smaller teams who don't have the time or resources to customize their environment, it often leads to pain and greater cost of paying down tech debt in the long run.
Isn't that inevitable though? If these apps exist, someone has to provide the platform for it. While it could be an "open source" thing that supported by a megavendor, most people building on top of that platform would just take it as is I would think (a la Android).
I get the desire for an open thing and all of that; I just don't see how it materializes without some entity with a lot of money financing it.
Great point. Note that the “real” value of a $1500 phone is a) belonging to a community with certain values, and b) acquiring status within that community. It is a fascinating question whether Snowflake (or anyone) can foment such an exponentially adaptive community in the data space…
So, I used to think that something like dbt could do this (and wrote about that a bit here: https://benn.substack.com/p/the-data-os). In theory, I think it could work, but I’m not sure how practical it is.
The issue, I think, is that the major databases that might want an “OS” (Snowflake, Databricks, BigQuery) are all pretty different. For a third party to build the OS, they’d have to build very complicated software for a bunch of very complicated tech, all from a distance. And to the end user, I’m not sure that’s any better than those OS’s being built by the vendors themselves, since there wouldn’t be any way to make them all unified. Sure, a third party might get closer than independent vendors, but I suspect people would rather trade that off for the more native OS.
Also, to overextend the phone analogy, Android and iOS provide two other kind of interesting comparisons:
First, even though the two are different, they do move toward common patterns. I’d suspect there’s be some of that in this case two, where different vendors start to mimic one another to some degree. You already see that in web services and between Snowflake and Databricks; I see no reason why it wouldn’t happen here.
Second, as a partial counterpoint to what I’m saying, Android did manage to unify a bunch of disparate hardware around a single OS. But 1) phones are probably less complicated than Snowflake, and 2) Android is the biggest player in the ecosystem. Phone manufacturers were pulled to Android. In this case, the database vendors are the big players, so they have a lot less incentive to adhere to a community standard.
How to address these issues is a tough question. I really don’t understand/cannot see how to embed or to make the privacy/compliance team to work together.
How do you think that this topic would translate for compliance/security people within companies? I strongly believe that the hardest part is to connect who draw these kind of rules (lawyers, cfo, board member etc) with products such as snowflake and the IT.
COngrats and hope you've won some money betting on the roulette.
Security and Privacy will have to be part of the Data OS, but rather than having an additional layer between the data and the user, we'll see solutions that translate centrally managed privacy and security policies expressed in human readable language to controls native to the Data OS. As a privacy officer you'll have your access, retention, consent, and masking policies consistently applied across the data irrespective of where it is stored (Snowflake, S3, Kafka,..), and how it is accessed (SQL, API, Data App, Reverse ETL,...).
The main challenges I'm seeing in achieving this are:
1) finding the common denominators across the vendors' security capabilities to achieve consistent controls. The space of possible controls will be limited to the lowest common denominators among these capabilities. I'm expecting the market to evolve to standards.
2) getting high quality metadata to automate these controls. Still a big question mark.
I think this is basically right. Platforms like Snowflake could govern what apps could do (can the app send data out of Snowflake? Does it have read access to other things in Snowflake? what snowflake resources can it control? etc) much like how phones govern what apps can do. In a lot of ways, I think that's a very big improvement over what we have today, where you can connect tools to snowflake, and have to trust each one individually. This framework would say, no, you don't even need to trust the app, because it can only do what snowflake says it can do.
That's not a solution without its challenges (as Bart-Vee says, there's a standardization problem, and some technical hurdles). But if done right, I think it makes IT's job easier rather than harder.
'Data app' is a weird term. Fundamentally, all apps are data apps. And if we can eventually write a CRM in Snowflake, that's not some special 'data' version of a CRM. It's just a CRM.
Mostly agree (and I talked about something related here: https://benn.substack.com/p/the-future-of-operational-analytics). Though I do think it's useful to have some way to distinguish between a CRM that is entirely standalone and one that uses an external data source as a backend.
I gotta be honest: it feels like we are still in the VMS era of the Data OS, where the first-party services are very different than third-party apps. I don’t feel like I’ve seen any full-stack data platform that is designed for extensible abstraction the way UNIX was.
Maybe we have to wait half a decade for Snowflake and DataBricks to hack their way into pseudo-data-apps before someone can design the right building blocks to self-assemble into a coherent stack...
Yeah, it's still very early I imagine. But Snowflake's and Databricks' approach seems markedly different than the early approaches of Redshift, BigQuery, etc, so it seems like we're at least moving in that direction. It'll happen in fits and starts, but seems like we're moving.
"...for Snowflake Summit, the first data Megaconference1 since 2019"? Sir. The second. The second data Megaconference since 2019. Tableau Conference was May 17-19! 🤓
> What if apps aren’t simple widgets for consumption, but are foundational programs that fundamentally change how Snowflake operates?
Um, sure. But “apps” is probably not the optimal term for that, or even the right layer to focus on. What you’re really talking about is a “Data OS” whose programs can be apps, daemons, scripts, extensions, etc. Right?
I’d say it’s more that Snowflake is the OS (in a loose sense) and that you run programs inside of it. That’s probably richer than an app as we think of it in the colloquial sense today.
Oracle had much of the same thinking as far back as 20+ years ago (maybe more, but I wasn't around then...). So if we're on making comparison, I think the right comparison is that we're seeing the next iteration of Oracle's proprietary in-database products.
The question is how open will that implementation be? And will apps built for Snowflake work on GCP, AWS stacks? I suspect no. In the end, I am not sure this is as exciting as it sounds. I think we're just looking at a proliferation of in-database functionalities that are required for a mature enterprise database product. And the standard there for decades has been Oracle.
What we're actually probably going to see is a few iterations of implementations with relatively open standards. Something that will give Snowflake's competitors their own good ideas to make in-database app products. Then competition on in-database apps kicks off, and Snowflake will start making decisions. towards making App implementation much more proprietary, using custom languages totally incompatible with anything else.
I think the more interesting future is emergence of another Streamlit, but one like DBT that is able to raise >$300mil and build an an entire cloud platform around the idea of community contributed apps. That would be an interesting future.
Yeah, none of this is entirely novel. But small differences can make or break results. Why did Slack become Slack and Hipchat died? It wasn't really anything structural; they just made a better product and caught the market at the right time.
Would something like this work with Snowflake even though it didn't at Oracle? Certainly could. Most "new" tech is some old idea recut with new paradigms (same as before, but in the cloud! same as before, but selling to analysts and not DBAs!). Sometimes that change makes all the difference; sometimes, it flops just the same.
As for creating a platform around community apps, I don't see why that's not Snowflake. They're already where the tech lives; I'm not sure why you'd need another entity to host the community. Back to the iPhone analogy, that'd be like another company building the app store. I guess that's possible, but it makes a lot more sense for Apple to do it.
There is a difference b/w Snowflake and Apple in the iPhone analogy. The difference is small, but is important: monetization.
Apple Store is enabler for new Business Models, for new Businesses. The purchase of the app is just the beginning to the monetization. The app is the end-user interface, so it can do anything.
In the case of databases, I am not sure there is an enabler for new businesses. Say I am interested in the privacy app for emails. I think it is unlikely that a new business will born from a successful privacy app for emails on Snowflake. A more realistic scenario is that there is already a privacy vendor, which will build an integration for Snowflake.
The better comparison is with Chrome/Firefox Web Store of Extensions. Some succeed in monetizing, but the majority won't. The successful ones (Evernote, Loom) exist as standalone applications, and integrate with a browser in addition to the application itself. But even that is a misleading comparison. These extensions are embedded in my UI experience.
So Snowflake Store is primarily an ecosystem play, to get existing vendors with their own users, to make a bigger bet on Snowflake as a DW partner. For companies such as Mode, to create Snowflake-specific apps, doubling down on that partnership. Or to promote 3rd party apps as part of Mode-onboarding experience.
In that sense, this is not much different than the Looker marketplace. Powerful as an idea, but not an enabler of new businesses - just a way to integrate closer with an ecosystem.
Maybe what we really need is new “atomic unit” of data. Not tables. Not dashboards. Not even apps. Let’s call it an INS (for insight, insert, or inside joke). A true Data OS would make it easy for humans (or machines) produce, consume, and reuse INS without having worry about the complexity of formats, compute, or even platform.
Great thought piece. I'm with you on the utility of data apps, a vertical approach that delivers immediate business value versus a general horizontal BI tool that says "you can build anything you want, go for it."
One of my challenges in delivering data apps has been the fact that different customers have different data stacks, especially database preference. Perhaps this iSnowflake approach simplifies that, but does it then push them to a single vendor that data app creators sort of homogenize around?
Could an inverted approach that is database agnostic provide much of the same value? A lot of the data app utility is in the data transformation component - getting data from the disparate systems of all the potential data app customers to be used by a generic data app UI. For example, deliver a data app with dbt while paying special attention to keeping it database agnostic, then build your data app UI (Mode, custom app, Tableau, etc.) based on those models. Free, agnostic, open source.
For many, is there value in a $1500 phone that is roughly the same as a $500 phone? I know this is blasphemy, but how much of that $1500 is sales and marketing versus real value? Also, how do marketplace app creators feel about their end of the deal?
This is exactly the approach we are taking at Starburst. The notion that a single data platform (i.e. Snowflake) should own all your data and now all your data apps is counter to good engineering practices and promotes vendor lock-in in our view. Why should Snowflake's product roadmap dictate the way an analytics engineer thinks about their development framework?
Optionality and the ability to choose the right data stack for the right use case is foundational to creating apps that have the right performance, complexity, cost tradeoffs. Additionally, as a data app evolves or gains adoption over time, it may have different needs that would be best served by broader variety of database architectures and vendors in the future.
While starting with everything on a single vendor is likely going to be easier in the short term for smaller teams who don't have the time or resources to customize their environment, it often leads to pain and greater cost of paying down tech debt in the long run.
Isn't that inevitable though? If these apps exist, someone has to provide the platform for it. While it could be an "open source" thing that supported by a megavendor, most people building on top of that platform would just take it as is I would think (a la Android).
I get the desire for an open thing and all of that; I just don't see how it materializes without some entity with a lot of money financing it.
Great point. Note that the “real” value of a $1500 phone is a) belonging to a community with certain values, and b) acquiring status within that community. It is a fascinating question whether Snowflake (or anyone) can foment such an exponentially adaptive community in the data space…
So, I used to think that something like dbt could do this (and wrote about that a bit here: https://benn.substack.com/p/the-data-os). In theory, I think it could work, but I’m not sure how practical it is.
The issue, I think, is that the major databases that might want an “OS” (Snowflake, Databricks, BigQuery) are all pretty different. For a third party to build the OS, they’d have to build very complicated software for a bunch of very complicated tech, all from a distance. And to the end user, I’m not sure that’s any better than those OS’s being built by the vendors themselves, since there wouldn’t be any way to make them all unified. Sure, a third party might get closer than independent vendors, but I suspect people would rather trade that off for the more native OS.
Also, to overextend the phone analogy, Android and iOS provide two other kind of interesting comparisons:
First, even though the two are different, they do move toward common patterns. I’d suspect there’s be some of that in this case two, where different vendors start to mimic one another to some degree. You already see that in web services and between Snowflake and Databricks; I see no reason why it wouldn’t happen here.
Second, as a partial counterpoint to what I’m saying, Android did manage to unify a bunch of disparate hardware around a single OS. But 1) phones are probably less complicated than Snowflake, and 2) Android is the biggest player in the ecosystem. Phone manufacturers were pulled to Android. In this case, the database vendors are the big players, so they have a lot less incentive to adhere to a community standard.
Interesting stuff, thanks!
I'm really so glad that someone else called out how squishy the carpet was. It was ... almost hard to walk on.
At one point I had to walk the length of the hall with a coffee without a lid, and I had genuine concerns that I was gonna make it.
How to address these issues is a tough question. I really don’t understand/cannot see how to embed or to make the privacy/compliance team to work together.
Do you any good solution?
How do you think that this topic would translate for compliance/security people within companies? I strongly believe that the hardest part is to connect who draw these kind of rules (lawyers, cfo, board member etc) with products such as snowflake and the IT.
COngrats and hope you've won some money betting on the roulette.
Security and Privacy will have to be part of the Data OS, but rather than having an additional layer between the data and the user, we'll see solutions that translate centrally managed privacy and security policies expressed in human readable language to controls native to the Data OS. As a privacy officer you'll have your access, retention, consent, and masking policies consistently applied across the data irrespective of where it is stored (Snowflake, S3, Kafka,..), and how it is accessed (SQL, API, Data App, Reverse ETL,...).
The main challenges I'm seeing in achieving this are:
1) finding the common denominators across the vendors' security capabilities to achieve consistent controls. The space of possible controls will be limited to the lowest common denominators among these capabilities. I'm expecting the market to evolve to standards.
2) getting high quality metadata to automate these controls. Still a big question mark.
I think this is basically right. Platforms like Snowflake could govern what apps could do (can the app send data out of Snowflake? Does it have read access to other things in Snowflake? what snowflake resources can it control? etc) much like how phones govern what apps can do. In a lot of ways, I think that's a very big improvement over what we have today, where you can connect tools to snowflake, and have to trust each one individually. This framework would say, no, you don't even need to trust the app, because it can only do what snowflake says it can do.
That's not a solution without its challenges (as Bart-Vee says, there's a standardization problem, and some technical hurdles). But if done right, I think it makes IT's job easier rather than harder.
'Data app' is a weird term. Fundamentally, all apps are data apps. And if we can eventually write a CRM in Snowflake, that's not some special 'data' version of a CRM. It's just a CRM.
Mostly agree (and I talked about something related here: https://benn.substack.com/p/the-future-of-operational-analytics). Though I do think it's useful to have some way to distinguish between a CRM that is entirely standalone and one that uses an external data source as a backend.
I gotta be honest: it feels like we are still in the VMS era of the Data OS, where the first-party services are very different than third-party apps. I don’t feel like I’ve seen any full-stack data platform that is designed for extensible abstraction the way UNIX was.
Maybe we have to wait half a decade for Snowflake and DataBricks to hack their way into pseudo-data-apps before someone can design the right building blocks to self-assemble into a coherent stack...
Yeah, it's still very early I imagine. But Snowflake's and Databricks' approach seems markedly different than the early approaches of Redshift, BigQuery, etc, so it seems like we're at least moving in that direction. It'll happen in fits and starts, but seems like we're moving.
"...for Snowflake Summit, the first data Megaconference1 since 2019"? Sir. The second. The second data Megaconference since 2019. Tableau Conference was May 17-19! 🤓
Fair, I did exactly zero research to figure out if that statement was actually true.
Great post, by the way. I think you're right on the mark in terms of what comes next for Snowflake.
Thanks!
> What if apps aren’t simple widgets for consumption, but are foundational programs that fundamentally change how Snowflake operates?
Um, sure. But “apps” is probably not the optimal term for that, or even the right layer to focus on. What you’re really talking about is a “Data OS” whose programs can be apps, daemons, scripts, extensions, etc. Right?
I’d say it’s more that Snowflake is the OS (in a loose sense) and that you run programs inside of it. That’s probably richer than an app as we think of it in the colloquial sense today.