The "disjointed" Universal Semantic Layer section was 100% JOINTED for me. I'm an in-the-trenches business owner of "the data" at a NYSE company in an industry that's two decades back in a time warp. I got handed "the data" because I've been helping connect all the systems for the last 15 years and know the whole thing end-to-end. This section was 100% spot-on...and I really appreciate your contrarian perspective...
Thanks! And yeah, the "we acquired 10 companies and 20 different ERP systems and they all need to talk to each other even though none of them have been updated in a decade" is a whole other world of pain.
I'm so far along in my career, several C-suiters tell me I'm the only one (out of 50,000+ employees) who understands it end-to-end...I should quit and let them re-hire me as a consultant, but at 25 years in, I don't have a viable action plan if they call my bluff!
Ahh, I was looking for this. I remembered reading somewhere that people typically go through two iterations, but I couldn't remember where that came from.
Interesting take on the analytics engineer role evolution, Benn
Given this reframing around SQL and ETL tools, do you see this becoming more accessible to fractional/part-time arrangements?
Particularly curious if you're seeing patterns in which types of companies (size, stage, industry) might be well-suited for fractional analytics engineering vs. needing full-time roles.
Ehh, not really? Or like, I don't think that really works.
Maybe something to expand on later, but I think the biggest value of analytics engineering is (or should be) actually architecture. It's designing the flow charts of how data moves around different tools, and designing the DAGs for how you get from messy data to data that's useful. That's a very dynamic system on both ends - both the inputs and the needed outputs constantly change - and it's one that someone needs to be paying attention to. If they don't, you end up with a bunch of redundancies, runaway costs, a knot of models, etc etc etc.
In theory, both analysts and data engineers probably have the skills to do it. But analysts are too worried about what happens with the outputs (and are too incentivized to just tack on some new shortcut when they need something) and engineers typically want to be engineers writing code, not people drawing flow charts about how SQL models work together (nor do they typically know the outputs well enough to design that intermediate system). Analytics engineering is a useful role to fill that gap to me.
And consultants probably can't fill it either, because it changes too much. You can't build it out and walk away; as soon as you do, the tidy design will get overrun by the entropy around it. It's an ongoing maintenance job. Which, I suppose you could do with part-time hours - maybe it's only 10 hours a week, and not 40 or whatever - but it's 10 hours a week in perpetuity, not 40 hours up front. And most consulting engagements seem to prefer the latter structure over the former.
The distance is the biggest problem I have seen and experienced. Asking a data engineer to design outputs is like asking them to do analysis for you, they aren't close enough to understand it and, to be brutally honest, most (not all) don't care to understand the outputs. That leads to a quick, get it off my plate and stop looking at me, response.
Interestingly, the dynamic I saw in a large group was that Data Engineering group wanted to build the most narrow solution possible, the BI group wanted to created a complete unified model of everything, and I was leading the Data Science and Data Analyst group being held hijack by both of them. The particularly frustrating part was that my group knew the subset of the data we absolutely needed, but no one would listen to us.
Yeah, that distance was one of the big points in the original arguments for building something like dbt. If analysts are going to use the models, they should create the models.
Which makes sense, but I'm not sure analysts will ever really have time to think that way. It's like asking the engineers who are on call and constantly reacting to new problems to also be the architects. You're doing too much fire fighting to also always be thinking about how to design and maintain a tidy system. Which is why I think the analytics engineer thing works, if it's framed right. They need to be aware of the business problems, but not responsible for the fires, so that they can just think about the bigger picture stuff.
Your point about dbt addressing the 'distance' problem is fascinating. Makes me wonder:
(1) what other tool gaps might exist that could bridge similar role-based disconnects in the data stack?
(2) With this framing of understanding outputs being crucial, do you see a future where we have more hybrid roles - perhaps data analysts who transition to architecture-focused engineering roles, bringing their output-understanding advantage with them? Or is the analytics engineer role already evolving to fill this space?
1) Any of the mid-code tools seem like they'd work? To me, the main job of an analyst is to sit in between a bunch of technical systems and a bunch of business systems. They have data that's generated by machines and by application backends; they have to answer questions that are about money and customers and business performance. Someone's gotta be the translation between those things, and they probably need tools that are similarly shaped.
2) Ehhh, I'm sort of mixed on the hybrid role. I think the problem with those sorts of roles is you can't do all the tasks at once. To me, it's not a skill issue, where an analyst who learns data engineering is what we all need; it's a problem with focus. If you're worried about the analyst stuff, there's just no way to also be the architect (unless things are so small that you have lots of time). That's the thing i do think that's good about analytics engineering - it's a role that intentionally designed to fill the gap that analysts don't have the time to do. (Whether or not we're good at it is a different question.)
We need to ask more of our data engineers, and I say that as one myself. A data engineer that doesn’t care about the outputs is going to have limited impact. Ditto for doing analysis, at least at an intermediate level, since it’s hard to design the right outputs to data consumers if you don’t know how they are used. I’ve been fortunate to work on teams that had a high standard for the role and expected those behaviors (among others). It allowed us to collaborate effectively with data consumers of all types and better anticipate needs.
I agree with your sentiment, I see way too many teams with the behaviors you described. Many of the issues Benn highlights in this newsletter would be alleviated with a more rigorous data engineering discipline. But we have a long way to go…
yeah, I think that's fair. I'm always somewhat skeptical of solutions to problems that are essentially "be better," though part of the issue with these roles is that they sometimes have confusing dividing lines, and people can get away with not paying attention to some stuff that it'd probably be better if they did. So while I'm not sure we can just teach people to be better analysts or whatever, I suspect folks could redraw the lines for what analysts or engineers are responsible for, and change things that way.
Really appreciate this perspective on analytics engineering as ongoing architecture maintenance rather than just initial setup.
Your point about entropy is particularly compelling - it reminds me of Olga Berezovsky's recent post https://open.substack.com/pub/dataanalysis/p/my-consulting-journey-0-to-685k about her consulting journey where she noted how analytics projects kept expanding because once you see data issues, it's hard to walk away.
Makes me wonder if there's a middle ground where companies too small for full-time AEs could share a dedicated long-term architect (not short-term consultant) who builds deep context over time, even if part-time. But perhaps that's still fighting against the natural gravity of the role.
I think my view on that would be at that size, you just gotta choose. Do you want to build a spiraling system that you know you'll probably tear down one day, but be able to answer stuff fairly quickly in the meantime? Or do you want to move slowly on the question answering stuff, and try to be intentional about the architecture? Either acknowledge it's temporary and absorb the debt (with the intent of declaring bankruptcy on it) or slow down and build it well. I think what's bad if when people try to straddle between the two.
The "disjointed" Universal Semantic Layer section was 100% JOINTED for me. I'm an in-the-trenches business owner of "the data" at a NYSE company in an industry that's two decades back in a time warp. I got handed "the data" because I've been helping connect all the systems for the last 15 years and know the whole thing end-to-end. This section was 100% spot-on...and I really appreciate your contrarian perspective...
Thanks! And yeah, the "we acquired 10 companies and 20 different ERP systems and they all need to talk to each other even though none of them have been updated in a decade" is a whole other world of pain.
I'm so far along in my career, several C-suiters tell me I'm the only one (out of 50,000+ employees) who understands it end-to-end...I should quit and let them re-hire me as a consultant, but at 25 years in, I don't have a viable action plan if they call my bluff!
Nice data setup you got there, shame if something happened to it
My life...
I love me a good game theory puzzle. There's a surprising amount of lore behind the game to "Guess 2/3 of the average" https://en.wikipedia.org/wiki/Guess_2/3_of_the_average
Ahh, I was looking for this. I remembered reading somewhere that people typically go through two iterations, but I couldn't remember where that came from.
Interesting take on the analytics engineer role evolution, Benn
Given this reframing around SQL and ETL tools, do you see this becoming more accessible to fractional/part-time arrangements?
Particularly curious if you're seeing patterns in which types of companies (size, stage, industry) might be well-suited for fractional analytics engineering vs. needing full-time roles.
Ps: followed you on bsky
Ehh, not really? Or like, I don't think that really works.
Maybe something to expand on later, but I think the biggest value of analytics engineering is (or should be) actually architecture. It's designing the flow charts of how data moves around different tools, and designing the DAGs for how you get from messy data to data that's useful. That's a very dynamic system on both ends - both the inputs and the needed outputs constantly change - and it's one that someone needs to be paying attention to. If they don't, you end up with a bunch of redundancies, runaway costs, a knot of models, etc etc etc.
In theory, both analysts and data engineers probably have the skills to do it. But analysts are too worried about what happens with the outputs (and are too incentivized to just tack on some new shortcut when they need something) and engineers typically want to be engineers writing code, not people drawing flow charts about how SQL models work together (nor do they typically know the outputs well enough to design that intermediate system). Analytics engineering is a useful role to fill that gap to me.
And consultants probably can't fill it either, because it changes too much. You can't build it out and walk away; as soon as you do, the tidy design will get overrun by the entropy around it. It's an ongoing maintenance job. Which, I suppose you could do with part-time hours - maybe it's only 10 hours a week, and not 40 or whatever - but it's 10 hours a week in perpetuity, not 40 hours up front. And most consulting engagements seem to prefer the latter structure over the former.
The distance is the biggest problem I have seen and experienced. Asking a data engineer to design outputs is like asking them to do analysis for you, they aren't close enough to understand it and, to be brutally honest, most (not all) don't care to understand the outputs. That leads to a quick, get it off my plate and stop looking at me, response.
Interestingly, the dynamic I saw in a large group was that Data Engineering group wanted to build the most narrow solution possible, the BI group wanted to created a complete unified model of everything, and I was leading the Data Science and Data Analyst group being held hijack by both of them. The particularly frustrating part was that my group knew the subset of the data we absolutely needed, but no one would listen to us.
Yeah, that distance was one of the big points in the original arguments for building something like dbt. If analysts are going to use the models, they should create the models.
Which makes sense, but I'm not sure analysts will ever really have time to think that way. It's like asking the engineers who are on call and constantly reacting to new problems to also be the architects. You're doing too much fire fighting to also always be thinking about how to design and maintain a tidy system. Which is why I think the analytics engineer thing works, if it's framed right. They need to be aware of the business problems, but not responsible for the fires, so that they can just think about the bigger picture stuff.
Your point about dbt addressing the 'distance' problem is fascinating. Makes me wonder:
(1) what other tool gaps might exist that could bridge similar role-based disconnects in the data stack?
(2) With this framing of understanding outputs being crucial, do you see a future where we have more hybrid roles - perhaps data analysts who transition to architecture-focused engineering roles, bringing their output-understanding advantage with them? Or is the analytics engineer role already evolving to fill this space?
1) Any of the mid-code tools seem like they'd work? To me, the main job of an analyst is to sit in between a bunch of technical systems and a bunch of business systems. They have data that's generated by machines and by application backends; they have to answer questions that are about money and customers and business performance. Someone's gotta be the translation between those things, and they probably need tools that are similarly shaped.
2) Ehhh, I'm sort of mixed on the hybrid role. I think the problem with those sorts of roles is you can't do all the tasks at once. To me, it's not a skill issue, where an analyst who learns data engineering is what we all need; it's a problem with focus. If you're worried about the analyst stuff, there's just no way to also be the architect (unless things are so small that you have lots of time). That's the thing i do think that's good about analytics engineering - it's a role that intentionally designed to fill the gap that analysts don't have the time to do. (Whether or not we're good at it is a different question.)
We need to ask more of our data engineers, and I say that as one myself. A data engineer that doesn’t care about the outputs is going to have limited impact. Ditto for doing analysis, at least at an intermediate level, since it’s hard to design the right outputs to data consumers if you don’t know how they are used. I’ve been fortunate to work on teams that had a high standard for the role and expected those behaviors (among others). It allowed us to collaborate effectively with data consumers of all types and better anticipate needs.
I agree with your sentiment, I see way too many teams with the behaviors you described. Many of the issues Benn highlights in this newsletter would be alleviated with a more rigorous data engineering discipline. But we have a long way to go…
yeah, I think that's fair. I'm always somewhat skeptical of solutions to problems that are essentially "be better," though part of the issue with these roles is that they sometimes have confusing dividing lines, and people can get away with not paying attention to some stuff that it'd probably be better if they did. So while I'm not sure we can just teach people to be better analysts or whatever, I suspect folks could redraw the lines for what analysts or engineers are responsible for, and change things that way.
Really appreciate this perspective on analytics engineering as ongoing architecture maintenance rather than just initial setup.
Your point about entropy is particularly compelling - it reminds me of Olga Berezovsky's recent post https://open.substack.com/pub/dataanalysis/p/my-consulting-journey-0-to-685k about her consulting journey where she noted how analytics projects kept expanding because once you see data issues, it's hard to walk away.
Makes me wonder if there's a middle ground where companies too small for full-time AEs could share a dedicated long-term architect (not short-term consultant) who builds deep context over time, even if part-time. But perhaps that's still fighting against the natural gravity of the role.
I think my view on that would be at that size, you just gotta choose. Do you want to build a spiraling system that you know you'll probably tear down one day, but be able to answer stuff fairly quickly in the meantime? Or do you want to move slowly on the question answering stuff, and try to be intentional about the architecture? Either acknowledge it's temporary and absorb the debt (with the intent of declaring bankruptcy on it) or slow down and build it well. I think what's bad if when people try to straddle between the two.