I like the part where you mention dbt Metrics. I fully agree with everything you said and would like to add up that dbt should leave the semantic layer to players who are more experienced with it.
While talking about semantic layers, I think there should be a standardization of this mighty feature. Unfortunately, semantic layers do not follow any standardization now and for that, they become vendor locks.
On dbt leaving it to others, sure, though I'm sure they'd like to (and may well become) one of those players who have experience with it.
And on standardization, that's the dream. My suspicion is that we only get that standardization when a vendor wins the market, and we all gravitate towards it. So in that sense, I think these vendor-led efforts are the road to standardization, even if it doesn't look like it yet.
Having only seen a simple demo of dbt it just does not seem to do much at all. And I do not see how it would possibly scale to create an ETL system for hundreds of tables and tens of thousands of fields.
The core of it is pretty simple, but I think that's why it works: You can start with very basic (and useful) operations, and build in a lot of complexity from there. It's true that it doesn't do a bunch of orchestration for very complex systems well, though I'd imagine they'll get to that at some point.
yes. Thes GUI ETL style tools always sell on the point of "it is simple to use and you can get lower paid people to use it because it's simple". DataStage and Informatica were sold like that. The issue comes when there are a lot of mappings and things are complex. Then the GUI gets in the way and vastly increases costs. We have been through this before.
But dbt is the opposite of that. There is no GUI. It's all code. That's what makes it work to me: You start with basic code, so there is no GUI that gets in the way as you add complexity.
Hi Benn, I have seen all of a 1 hour demo of dbt on their you tube channel, so very little exposure to it. It looked to me like it is accessed via a GUI and that actions in the GUI generated code. Maybe I should re-watch the demo. It didn't look like the user had to write each piece of code inside the GUI. Maybe I should watch the demo again. I watched it about 3-4 weeks ago. It looked very similar to many other things I have seen so maybe I jumped to conclusions.
just so you know. This is the video I watched. dbt 101 they called it. What I see is a GUI and an sql like language with things such as "ref" statements etc. This "sql like language" then gets "generated / compiled" in to SQL and can then be run under the control of dbt as a scheduler.
I thought most of the "sql like" language would be generated and not hand written or "cut and pasted" into the GUI.
Did they miss something serious in explaining what dbt is? Or have I in some way mis-understood? You can email me directly if you like. You should have my email on your email list. If not just go to our contact page on bida.ro and pop in your email and I will write to you there.
About 3-4 weeks ago I thought I would watch a video on dbt just to see what people were talking about. That is the one I watched. It's extremely simplistic compared to what we are doing so I didn't pay much attention.
you must have not read many real MBA case studies. This is nowhere as dry.
Hi Mir, Can you please clarify what is it you were trying to argue for?
I like the part where you mention dbt Metrics. I fully agree with everything you said and would like to add up that dbt should leave the semantic layer to players who are more experienced with it.
While talking about semantic layers, I think there should be a standardization of this mighty feature. Unfortunately, semantic layers do not follow any standardization now and for that, they become vendor locks.
On dbt leaving it to others, sure, though I'm sure they'd like to (and may well become) one of those players who have experience with it.
And on standardization, that's the dream. My suspicion is that we only get that standardization when a vendor wins the market, and we all gravitate towards it. So in that sense, I think these vendor-led efforts are the road to standardization, even if it doesn't look like it yet.
Really enjoy reading your posts, thanks for doing this every week!
Thanks!
Having only seen a simple demo of dbt it just does not seem to do much at all. And I do not see how it would possibly scale to create an ETL system for hundreds of tables and tens of thousands of fields.
The core of it is pretty simple, but I think that's why it works: You can start with very basic (and useful) operations, and build in a lot of complexity from there. It's true that it doesn't do a bunch of orchestration for very complex systems well, though I'd imagine they'll get to that at some point.
Hi Benn,
yes. Thes GUI ETL style tools always sell on the point of "it is simple to use and you can get lower paid people to use it because it's simple". DataStage and Informatica were sold like that. The issue comes when there are a lot of mappings and things are complex. Then the GUI gets in the way and vastly increases costs. We have been through this before.
But dbt is the opposite of that. There is no GUI. It's all code. That's what makes it work to me: You start with basic code, so there is no GUI that gets in the way as you add complexity.
Hi Benn, I have seen all of a 1 hour demo of dbt on their you tube channel, so very little exposure to it. It looked to me like it is accessed via a GUI and that actions in the GUI generated code. Maybe I should re-watch the demo. It didn't look like the user had to write each piece of code inside the GUI. Maybe I should watch the demo again. I watched it about 3-4 weeks ago. It looked very similar to many other things I have seen so maybe I jumped to conclusions.
It's worth checking out. They've got lots of resources and things online, so there's no shortage of material to learn from.
Hi Benn,
just so you know. This is the video I watched. dbt 101 they called it. What I see is a GUI and an sql like language with things such as "ref" statements etc. This "sql like language" then gets "generated / compiled" in to SQL and can then be run under the control of dbt as a scheduler.
I thought most of the "sql like" language would be generated and not hand written or "cut and pasted" into the GUI.
Did they miss something serious in explaining what dbt is? Or have I in some way mis-understood? You can email me directly if you like. You should have my email on your email list. If not just go to our contact page on bida.ro and pop in your email and I will write to you there.
About 3-4 weeks ago I thought I would watch a video on dbt just to see what people were talking about. That is the one I watched. It's extremely simplistic compared to what we are doing so I didn't pay much attention.
https://www.youtube.com/watch?v=FfHk2dyeW1M&ab_channel=dbt