I like the analogy with an OS, but I doubt we will have something like this in the next couple of years, and even more skeptical of dbt's ability to pull this. dbt is a transformation layer with some cool things aside (based on ref it creates the lineage and processing dependencies, some minimal testing capabilities, and a couple more minor things related to deployment, snapshots), but it's a long shot to convert everything there is out there to dbt. dbt is still not suitable for real-time processing, having native transaction or procedural support, mutations, and many more. Oh and btw dbt does integrate today with lots of vendors (Airbyte, Fivetran, Hightouch, Paradime, etc.), and acts as the main hub. It's just is a small, limited hub :)
I think we should have an open protocol and enforce it on the big players (Snowflake,
BigQuery, Databricks). The protocol will define how each vendor will integrate and exchange data with other vendors and a suite of tests to pass for when you integrate. The underlying layer (OS) should be the cloud data warehouse platform, and every such platform to adhere to those standards.
dbt does "in batch" what Trino/Presto does "live". Trino/Presto is able to join across disparate databases, e.g. more than a switchboard (or at least the same level of switchboard as dbt). So either (or both) are viable and great options because they are both SQL-based abstractions.
I also agree the formal definition of the mesh is very lacking on: "What happens on the other side of the mesh, or even what the mesh actually is, isn’t discussed"
Isnt this mess part of the process of technology evolution? Think about how we consumed music in the 90's. Grandparents had record players, because their collection was big and they sounded good. Parents had some records, maybe a few 8 tracks, and a decent tape collection (even if some of those tapes were mixed from the record collection). Poor kids had tapes, wealthy kids had CD's and rich kids had mini-disks. Then the 2000's trashed all that with MP3s, MP4s, Itunes and Napster. Now the cool kids are buying records again.
Also, as I am applying for my first analyst job, I see this same type of tool overload disfunction every day. Each job board wants to process my resume slightly differently. Then the application might happen through the job board itself, or redirect to the company. Sometimes the company processes it internally, sometimes it is processed by a 3rd party where I have to set up an account and password. Sometimes the company I am applying to is a staffing company, a recruiting company or a contract company and they use a 3rd party processor to vet for the 4th? party. Sometimes the posting will be a cross posting from another job board, that is a recruiter posting for a 3rd party that uses a 2nd 3rd party to process the application. Maybe applications would be a good application for "The Mesh"?
Isn't this, in essence, Denodo?
I like the analogy with an OS, but I doubt we will have something like this in the next couple of years, and even more skeptical of dbt's ability to pull this. dbt is a transformation layer with some cool things aside (based on ref it creates the lineage and processing dependencies, some minimal testing capabilities, and a couple more minor things related to deployment, snapshots), but it's a long shot to convert everything there is out there to dbt. dbt is still not suitable for real-time processing, having native transaction or procedural support, mutations, and many more. Oh and btw dbt does integrate today with lots of vendors (Airbyte, Fivetran, Hightouch, Paradime, etc.), and acts as the main hub. It's just is a small, limited hub :)
I think we should have an open protocol and enforce it on the big players (Snowflake,
BigQuery, Databricks). The protocol will define how each vendor will integrate and exchange data with other vendors and a suite of tests to pass for when you integrate. The underlying layer (OS) should be the cloud data warehouse platform, and every such platform to adhere to those standards.
Great article!
To me, SQL is the "common utility".
dbt does "in batch" what Trino/Presto does "live". Trino/Presto is able to join across disparate databases, e.g. more than a switchboard (or at least the same level of switchboard as dbt). So either (or both) are viable and great options because they are both SQL-based abstractions.
I also agree the formal definition of the mesh is very lacking on: "What happens on the other side of the mesh, or even what the mesh actually is, isn’t discussed"
Isnt this mess part of the process of technology evolution? Think about how we consumed music in the 90's. Grandparents had record players, because their collection was big and they sounded good. Parents had some records, maybe a few 8 tracks, and a decent tape collection (even if some of those tapes were mixed from the record collection). Poor kids had tapes, wealthy kids had CD's and rich kids had mini-disks. Then the 2000's trashed all that with MP3s, MP4s, Itunes and Napster. Now the cool kids are buying records again.
Also, as I am applying for my first analyst job, I see this same type of tool overload disfunction every day. Each job board wants to process my resume slightly differently. Then the application might happen through the job board itself, or redirect to the company. Sometimes the company processes it internally, sometimes it is processed by a 3rd party where I have to set up an account and password. Sometimes the company I am applying to is a staffing company, a recruiting company or a contract company and they use a 3rd party processor to vet for the 4th? party. Sometimes the posting will be a cross posting from another job board, that is a recruiter posting for a 3rd party that uses a 2nd 3rd party to process the application. Maybe applications would be a good application for "The Mesh"?