MAQL translates to SQL at query time based on the context. In your example about NPS, I can have a simple query "SELECT customer, nps" where nps is a metric you define once and it translates to a SQL query with "GROUP BY customer". If you call "SELECT region, nps", it translates to SQL with "GROUP BY region". It's pretty much as if you pass function definition as an argument.
Apr 22, 2022·edited Apr 22, 2022Liked by Benn Stancil
Benn, your Substack and DBT Slack contributions have convinced me that an open source metrics layer on top of dbt feeding headless bi is the future of the stack. It makes a ton of sense and rounds out the ecosystem. Thanks for your thoughtfulness and willingness to share.
Very Sweet Article, Benn. Talking about the real problems and focussing on the Metrics / KPIs, making it easy for Business to be on track.
I'm working on a platform - Alphaa AI - which can enable business to ask questions around Metric / KPI and the platform converts it into SQL Query and gets the data.
Would be great, if we can collaborate on the topic.
What would be the difference between a Metrics layer and the old concept of Materialized Views? In my view, SQL queries plus and understandable view schema is enough to solve this issue if you are in the relational world. We also implemented a similar idea with Elastic + Logstash to create a "table view" of the KPIs. Then, the business guy imports such table view into Power BI or Qlick View, for playing around data.
Nice! Quite unaware of the specificness of the problem we were moving to solve this in Data Virtualization, by offering both the parts (week definition =...) and the wholes (production by week=.... (based on week definition).
Whatever happened to Supergrain? Their website now positions them as a marketing campaign creation tool and doesn't mention metrics or headless BI. A metrics layer is a sensible idea but I wonder if it is easy to get traction in the market and convince customers with established BI implementations to insert a new layer and re-write all their dashboards.
Hi! Benn. Very inspiring article! I have read about it several times.
Coincidentally, before I read your article and learned about Airbnb's work on metrics platforms, my company, Kyligence, helped several companies build their metric platforms. Kyligence serves as the metrics computation engine in our user's cases.
Hi Benn. Great post and I couldn't agree with you more. As CTO and co-founder of AtScale, we've been working on building what we call a "semantic layer" for several years now. Semantic layers have been at the heart of BI platforms for years (Business Objects Universe, anyone?) so the concept is not new. However, like you, I believe the semantic layer should be separate from the consumption (i.e. BI, ML) tool so that it can be truly "universal" and provide consistency across a wide range of consumers. My co-founder wrote this blog that describes this approach: https://www.atscale.com/universal-semantic-layer/what-is-a-semantic-layer-why-would-i-want-one.
we have solved some of this problem (https://www.thoughtspot.com/trial) where our answers query which is written in natural language and converts query into SQL and renders data in dashboards .. the architecture is similar to what is described as solution here
Hi Benn, this was a good read. Have you considered storing all granular data in a database like Druid with Turnilo/Imply on top? Done this way, you wouldn't have to give up detail (like with roll-ups). Druid is performant as well, especially given we're dealing with granular data.
Hi Benn! Thanks for the nice article. Do you think tools like Atscale and Kyligence are worth considering for solving the universal semantic/metric layer problem ? If yes, then how successful are they ?
I work in microsoft data stack, in MS world(power bi, analysis service tabular model etc) there is DAX(data expression) language layer which is certainly similar to metrics layer proposed here, if we could come up a universal open source language similar to DAX but accessible to all BI tools, that would be wonderful! DAX is pretty solid language.
Hi Benn. Great Article and equivalently interesting comments!! I keep on coming back to the problems of ineffective self service solutions and missing metrics layer. Any thoughts of how these two are similar or try to solve same problem? For eg experience around discovering, creating, augmenting and monitoring actionable business metrics? (from end users perspective)
Benn,
I work for GoodData and I believe we do exactly what you describe. Here are examples of metric definitions in MAQL: https://github.com/zsvoboda/gooddata-jdbc/blob/main/scripts/orders.maql
MAQL translates to SQL at query time based on the context. In your example about NPS, I can have a simple query "SELECT customer, nps" where nps is a metric you define once and it translates to a SQL query with "GROUP BY customer". If you call "SELECT region, nps", it translates to SQL with "GROUP BY region". It's pretty much as if you pass function definition as an argument.
All that sits on top of your data warehouse and you can access it via APIs and SDKs, or connect it to any visualization library. More about it here: https://medium.com/gooddata-developers/gooddata-cn-modern-headless-bi-is-now-available-for-free-f4390ad4ca29
Benn, your Substack and DBT Slack contributions have convinced me that an open source metrics layer on top of dbt feeding headless bi is the future of the stack. It makes a ton of sense and rounds out the ecosystem. Thanks for your thoughtfulness and willingness to share.
Very Sweet Article, Benn. Talking about the real problems and focussing on the Metrics / KPIs, making it easy for Business to be on track.
I'm working on a platform - Alphaa AI - which can enable business to ask questions around Metric / KPI and the platform converts it into SQL Query and gets the data.
Would be great, if we can collaborate on the topic.
Here is a little demo:
https://www.youtube.com/watch?v=IHODWM9UOn0
PS: happy to do a POC around this to make this concept come alive.
What would be the difference between a Metrics layer and the old concept of Materialized Views? In my view, SQL queries plus and understandable view schema is enough to solve this issue if you are in the relational world. We also implemented a similar idea with Elastic + Logstash to create a "table view" of the KPIs. Then, the business guy imports such table view into Power BI or Qlick View, for playing around data.
Nice! Quite unaware of the specificness of the problem we were moving to solve this in Data Virtualization, by offering both the parts (week definition =...) and the wholes (production by week=.... (based on week definition).
Whatever happened to Supergrain? Their website now positions them as a marketing campaign creation tool and doesn't mention metrics or headless BI. A metrics layer is a sensible idea but I wonder if it is easy to get traction in the market and convince customers with established BI implementations to insert a new layer and re-write all their dashboards.
Hi! Benn. Very inspiring article! I have read about it several times.
Coincidentally, before I read your article and learned about Airbnb's work on metrics platforms, my company, Kyligence, helped several companies build their metric platforms. Kyligence serves as the metrics computation engine in our user's cases.
Inspired by your piece, I wrote an article about my understanding the metrics layer. I have also included an example of a metrics platform from our user's experience. https://medium.com/kyligence/understanding-the-metrics-store-c213341e4c25
Good post Benn, totally agree with your POV. I am currently investigating a new open source solution that aims to fill this gap: https://metriql.com/
Worthwhile to take a look at it / update your post based on your findings?
Hi Benn. Great post and I couldn't agree with you more. As CTO and co-founder of AtScale, we've been working on building what we call a "semantic layer" for several years now. Semantic layers have been at the heart of BI platforms for years (Business Objects Universe, anyone?) so the concept is not new. However, like you, I believe the semantic layer should be separate from the consumption (i.e. BI, ML) tool so that it can be truly "universal" and provide consistency across a wide range of consumers. My co-founder wrote this blog that describes this approach: https://www.atscale.com/universal-semantic-layer/what-is-a-semantic-layer-why-would-i-want-one.
I have been looking for this information for a long time, I was very surprised when I found it here.
we have solved some of this problem (https://www.thoughtspot.com/trial) where our answers query which is written in natural language and converts query into SQL and renders data in dashboards .. the architecture is similar to what is described as solution here
Hi Benn. Maybe this article shows how Airbnb are solving the problem you describe? https://medium.com/airbnb-engineering/how-airbnb-achieved-metric-consistency-at-scale-f23cc53dea70
Hi Benn, this was a good read. Have you considered storing all granular data in a database like Druid with Turnilo/Imply on top? Done this way, you wouldn't have to give up detail (like with roll-ups). Druid is performant as well, especially given we're dealing with granular data.
Hi Benn! Thanks for the nice article. Do you think tools like Atscale and Kyligence are worth considering for solving the universal semantic/metric layer problem ? If yes, then how successful are they ?
I work in microsoft data stack, in MS world(power bi, analysis service tabular model etc) there is DAX(data expression) language layer which is certainly similar to metrics layer proposed here, if we could come up a universal open source language similar to DAX but accessible to all BI tools, that would be wonderful! DAX is pretty solid language.
Hi Benn. Great Article and equivalently interesting comments!! I keep on coming back to the problems of ineffective self service solutions and missing metrics layer. Any thoughts of how these two are similar or try to solve same problem? For eg experience around discovering, creating, augmenting and monitoring actionable business metrics? (from end users perspective)