“If extracting an answer from a BI tool takes too long—i.e., minutes instead of seconds—people aren’t able to cycle through enough trials to cover meaningful ground. Rather of recognizing this incomplete analysis, we score it as bad analysis, and put guardrails around it, to protect people from making mistakes. Do the mundanes stuff, we say, but data teams will do the meaningful work. Shoot your own home videos, and leave the important content to the professionals.”
That’s gold! I got just a taste of this when Tableau first came out - when I got relatively good at Tableau there were brief moments I could move at the speed of thought, only to crash and burning a few minutes later stuck trying to figure out how to do something silly like format an axis. I agree that LLMs have to potential to have a more sustainable “speed of thought” interface with less crashing and burning.
This is exactly my hope - Tableau gets fairly close, but 1) is still a little too walled in, and 2) has a really steep learning curve. But if you learn it, and are working on something that stays within those walls, you can feel what it *could* be like.
Yes! I invested lots of hours in getting up to that speed that most people won’t want to do. I eventually abandon tableau for most things... I will still use for EDA sometimes because I’m really fast but not much else. - 1) walls are a problem - Salesforce may make that worse 2) Still suffered from the chaos that spreadsheets suffer from b/c adhoc from unknown data source is very easy
I have such mixed feelings about 2. We always complain about that kind of ad hoc mess, but it's also really valuable. It's like, how do we work within the mess, rather than trying to contain it?
Ya - I also suppose it’s kind of like criticizing the papers on my desk for being messy. They are messy right now - but those same papers could be well organized in another sense if they were indexed and searchable.
This reminds me - we have seen a version of this problem before… Do we create an organized, curated index of the internet like a phone book? Or do we give up and solve another way… clearly Google answered that question. BUT I don’t know any company that had a really good internal Google - it was/is much more “phone book” style to organize everything. As you were saying, maybe it’s the perfect time for LLMs to solve the internal and external problems. There is nothing inherently wrong with a spreadsheet if we knew it was trustworthy and we could quickly find it again!
But - if I can answer the question without any constructs like spreadsheets that is also pretty appealing…
>"I agree that LLMs have to potential to have a more sustainable “speed of thought” interface with less crashing and burning."
Benn made a list of all of things in the data space that weren't as good as expected, but now we're going to wait for the next big hype? Because this time it will resolve all our issues? Guess what: LLMs won't be the magic bullet either.
In reality, businesses needing to be "data-driven" has not changed, and is not going anywhere. It's table-stakes for all companies. This has been common knowledge for a century. It's an easier strategy for some business models than others, but it's a universal requirement. It is always a net positive to have more information than less.
The mistake, I believe, is in thinking that everything needs to be "real time" or "self serve", which result in tech and tooling obsession. The vast majority of business do not require it. The idea that you need to be able to cycle through 100 experiments a day is probably where "modern data people" are failing.
I think that gets at a big part of my question here though. If every company needs to be data driven and it’s truly table stakes, what does that actually mean? Ie, if being data driven isn’t experiments, or self-serve, or real-time information to inform decisions, what is that thing that everyone needs?
My point here is that I think I agree with you that businesses don’t need to those things…but if we strip enough stuff away, and are saying being data driven is a basic understanding of a few KPIs and the reporting on the goings-on of a business, is that really data driven in the way it’s commonly used? To use another appliance analogy, it feels like saying “the smart house is the future” and then defining the smart house as having wifi.
Way to articulate the existential questions I have rumbling around in my brain meats after working in data (BI specifically) for the last 23 years or so. A lot has changed, sure, but a lot has really stayed the same, and I doubt the pundit at HBR with a regular "but are you really data driven?" column would still be writing about it if suddenly the world really was all that we have promised it would be. That said, The LLM thing is an interesting one, and as *much* as I raise eyebrows at some of the ethical concerns, this is absolutely the space for the "imperfect" analyst to tinker through to the answers they are looking for. It could potentially be the final blow to that wall that exists between data teams and the business; provided the data isn't garbage. If you take a look at ThoughtSpot, this is what they are doing with their product, albeit to a point. Data modelling still has to be somewhat understood, but otherwise, it positions its tooling as from a Q&A approach.
(And this is in no way an official endorsement/advertisement on my part, I have not stake in the company, just somewhat interested in what they are doing)
As someone who now very much has a stake in ThoughtSpot…that’s more or less my thinking. That’s one of reasons we did the deal, to be honest - if AI is going to change this market, their approach seems to be the closest on to getting there. AI will never be a complete solution (the whole data modeling thing still seems like a pretty complicated problem nobody has a great answer to), but their approach to mixing AI with more traditional modeling seems more promising to me than the question-to-SQL-query path.
When I saw the Mode announcement, I thought, wow, my comment was timely. I have a tremendous amount of reservations around AI, and I felt the most comfortable so far with what ThoughtSpot is doing, mixing it with data modelling, and I get the sense they are taking a very thoughtful approach to it. They only I think they need to be careful with is their "crowdsourcing" approach to rating whether a result/output is "accurate", as human bias can be a hell of a drug.
Yeah, that's fair. There's a whole lot to figure out in all of this (and hopefully, with enough basic guardrails that the figuring it out part is one of gradual improvements, and not giant disasters we learn from).
“If extracting an answer from a BI tool takes too long—i.e., minutes instead of seconds—people aren’t able to cycle through enough trials to cover meaningful ground. Rather of recognizing this incomplete analysis, we score it as bad analysis, and put guardrails around it, to protect people from making mistakes. Do the mundanes stuff, we say, but data teams will do the meaningful work. Shoot your own home videos, and leave the important content to the professionals.”
That’s gold! I got just a taste of this when Tableau first came out - when I got relatively good at Tableau there were brief moments I could move at the speed of thought, only to crash and burning a few minutes later stuck trying to figure out how to do something silly like format an axis. I agree that LLMs have to potential to have a more sustainable “speed of thought” interface with less crashing and burning.
This is exactly my hope - Tableau gets fairly close, but 1) is still a little too walled in, and 2) has a really steep learning curve. But if you learn it, and are working on something that stays within those walls, you can feel what it *could* be like.
Yes! I invested lots of hours in getting up to that speed that most people won’t want to do. I eventually abandon tableau for most things... I will still use for EDA sometimes because I’m really fast but not much else. - 1) walls are a problem - Salesforce may make that worse 2) Still suffered from the chaos that spreadsheets suffer from b/c adhoc from unknown data source is very easy
I have such mixed feelings about 2. We always complain about that kind of ad hoc mess, but it's also really valuable. It's like, how do we work within the mess, rather than trying to contain it?
Ya - I also suppose it’s kind of like criticizing the papers on my desk for being messy. They are messy right now - but those same papers could be well organized in another sense if they were indexed and searchable.
This reminds me - we have seen a version of this problem before… Do we create an organized, curated index of the internet like a phone book? Or do we give up and solve another way… clearly Google answered that question. BUT I don’t know any company that had a really good internal Google - it was/is much more “phone book” style to organize everything. As you were saying, maybe it’s the perfect time for LLMs to solve the internal and external problems. There is nothing inherently wrong with a spreadsheet if we knew it was trustworthy and we could quickly find it again!
But - if I can answer the question without any constructs like spreadsheets that is also pretty appealing…
>"I agree that LLMs have to potential to have a more sustainable “speed of thought” interface with less crashing and burning."
Benn made a list of all of things in the data space that weren't as good as expected, but now we're going to wait for the next big hype? Because this time it will resolve all our issues? Guess what: LLMs won't be the magic bullet either.
In reality, businesses needing to be "data-driven" has not changed, and is not going anywhere. It's table-stakes for all companies. This has been common knowledge for a century. It's an easier strategy for some business models than others, but it's a universal requirement. It is always a net positive to have more information than less.
The mistake, I believe, is in thinking that everything needs to be "real time" or "self serve", which result in tech and tooling obsession. The vast majority of business do not require it. The idea that you need to be able to cycle through 100 experiments a day is probably where "modern data people" are failing.
I think that gets at a big part of my question here though. If every company needs to be data driven and it’s truly table stakes, what does that actually mean? Ie, if being data driven isn’t experiments, or self-serve, or real-time information to inform decisions, what is that thing that everyone needs?
My point here is that I think I agree with you that businesses don’t need to those things…but if we strip enough stuff away, and are saying being data driven is a basic understanding of a few KPIs and the reporting on the goings-on of a business, is that really data driven in the way it’s commonly used? To use another appliance analogy, it feels like saying “the smart house is the future” and then defining the smart house as having wifi.
Ironically it seems that even the world of Microwaves cannot escape the lure of AI:
https://twitter.com/eumweek/status/1671934323048562710
Microwave makers looking for that 20% bump in stock prices.
Way to articulate the existential questions I have rumbling around in my brain meats after working in data (BI specifically) for the last 23 years or so. A lot has changed, sure, but a lot has really stayed the same, and I doubt the pundit at HBR with a regular "but are you really data driven?" column would still be writing about it if suddenly the world really was all that we have promised it would be. That said, The LLM thing is an interesting one, and as *much* as I raise eyebrows at some of the ethical concerns, this is absolutely the space for the "imperfect" analyst to tinker through to the answers they are looking for. It could potentially be the final blow to that wall that exists between data teams and the business; provided the data isn't garbage. If you take a look at ThoughtSpot, this is what they are doing with their product, albeit to a point. Data modelling still has to be somewhat understood, but otherwise, it positions its tooling as from a Q&A approach.
(And this is in no way an official endorsement/advertisement on my part, I have not stake in the company, just somewhat interested in what they are doing)
As someone who now very much has a stake in ThoughtSpot…that’s more or less my thinking. That’s one of reasons we did the deal, to be honest - if AI is going to change this market, their approach seems to be the closest on to getting there. AI will never be a complete solution (the whole data modeling thing still seems like a pretty complicated problem nobody has a great answer to), but their approach to mixing AI with more traditional modeling seems more promising to me than the question-to-SQL-query path.
When I saw the Mode announcement, I thought, wow, my comment was timely. I have a tremendous amount of reservations around AI, and I felt the most comfortable so far with what ThoughtSpot is doing, mixing it with data modelling, and I get the sense they are taking a very thoughtful approach to it. They only I think they need to be careful with is their "crowdsourcing" approach to rating whether a result/output is "accurate", as human bias can be a hell of a drug.
Yeah, that's fair. There's a whole lot to figure out in all of this (and hopefully, with enough basic guardrails that the figuring it out part is one of gradual improvements, and not giant disasters we learn from).