I personally made the shift from data engineering to analytics engineering, rather than from being a data analyst to an analytics engineer. I needed more creative freedom and wanted to blaze my own path. With data engineering I always felt there was a stigma that things had to be done a certain way. With a new field like analytics engineering, there are no "experts". We are all trying to figure out what works and sharing that with others. I also get to create and implement my own best practices rather than following a textbook example.
The entire paragraph about why "a lot of us got into data" resonates so much with me I'm seriously considering adding it to my LinkedIn profile. This feels spot on Benn!
Stumbled upon your blog and literally devouring one blog post one after the other. I find them all so refreshing and insightful! Loved the metaphore: This is my wildly speculative and loosely supported theory about what’s happening: A lot of us got into data because we were problem solvers who liked puzzles and weren’t afraid of numbers. We liked thinking creatively, but not like a capital-C Creative; instead, we liked finding interesting paths through structured problems. Don’t give us blank canvas or Word doc; give us a board game, a Lego, a brain teaser, or Wordle.
Jul 1, 2022·edited Jul 2, 2022Liked by Benn Stancil
To me, there are 3 additional major components: Clarity + Experience + Compensation
Clarity:
DS got muddied over the past few years as companies blended analysts, data science, and ML/AI into a single "DS" bucket. When it came down to things, the majority of DS end up being Analysts and that is the least glorious of the DS jobs. They weren't excited about working in spreadsheets and creating line charts, they were excited about figuring out if you were pregnant before your closest friends/family even knew. Additionally, the work actually isn't bad. Cleaning data and modeling the warehouse is its own type of fun. The problem was that no one was rewarded for that work. Impact is over indexed during perf review and no one was pointing to the tables to show impact, they pointed to the analysis. The tables were just a means to the end. With the new model, there is a specific role to cover it and the impact of the table is actually captured.
Experience:
DS have had to do a lot of cleaning over the years, they built up expertise and tooling as they did so. Switching away from that and going pure DS would be a big hit to their day-to-day and comfort. They are also the obvious choice to staff this spot since they have been doing it a long time and are the go-to for doing this work.
Compensation:
There are two parts to this. (1) AE is a new role, there is a supply issue so you can demand a higher comp. (2) Engineers generally have a higher comp than DS. Analytics Engineers are often allocated as Engineers and therefore get a higher comp while still often getting a customized interview experience that is lighter on the leetcode and more focused on leetsql.
Another insightful piece, Benn. As someone who has made the journey from analyst to engineer, I feel everything you wrote in my bones. Analysis is open-ended and, for better or worse, judged on nebulous outcomes and not on effort.
If you spend two weeks digging into a data set only to realize there's nothing actionable in it, nobody's going to be happy, even if that discovery would be impossible without the work. On the other hand, if your analysis yields results that drive a win, you're always sharing that glory with your stakeholders. When you succeed, you succeed with the team. When you fail, you fail alone. That kind of sucks.
One additional point I'd make about the surge in analysts who want to try out engineering is this: how many analysts started out as analysts, and how many started out elsewhere in the business? I think it's important to remember that a lot of analysts have always been "hidden technologists"; they've had an interest in this stuff before they could even articulate it. I don't think it's been super uncommon for analysts to become DBAs, database engineers, data engineers, and even SDEs.
Now that there's an adjacent role with all of the cool software engineering tooling and best practices, I don't think it's any wonder that a lot of people are hopping to try it out.
Hi Ben, love reading/listening to you in podcasts as well.
Few questions:
1) Who did the cleaning and modelling prior to analytics engineer became a thing?
2) Does an analytic engineer now replaces this roll?
3) If the answer to question #2 is a no, would you say analytics engineer is simply a natural progress in a response to the creation of an enablement environment of tools and technologies(DBT and cloud data warehouse) which simply promoted self-service Semantic layer buildup inside the warehouse as opposed to in the bi tool itself?.
4) Do you see/already know if DBT is used elsewhere - not just serving analyst but more traditional roles like BIE’s/BI Dev’s and the likes?.
Dashboards(unless the role is pure order-taking) have an analysis element.
Even creating a new dataset usually implies the analysis.
I am stuck as I really do enjoy finding insights, but I don't always like customers! :p Also don't always like the need for ad-hoc assumptions unless particularly elegant. (However, I really really don't want to be a data grunt to blame when things go wrong in prod!!!)
That being said a painpoint is that all analysis is constrained by the politics of the org and the barriers of communication.
Even a brilliant analysis can be misunderstood. Arguably the MORE BRILLIANT the analysis the HARDER TO CONVEY.
If everything was just nerds talking to nerds, I would not be surprised if there is more balance.
Consultants (who arguably share the field if analysis) have their own networking systems though, and less need/desire to show how the sausage is made. A lot of the practical reality of sharing analysis is like consulting.
I personally made the shift from data engineering to analytics engineering, rather than from being a data analyst to an analytics engineer. I needed more creative freedom and wanted to blaze my own path. With data engineering I always felt there was a stigma that things had to be done a certain way. With a new field like analytics engineering, there are no "experts". We are all trying to figure out what works and sharing that with others. I also get to create and implement my own best practices rather than following a textbook example.
The entire paragraph about why "a lot of us got into data" resonates so much with me I'm seriously considering adding it to my LinkedIn profile. This feels spot on Benn!
Stumbled upon your blog and literally devouring one blog post one after the other. I find them all so refreshing and insightful! Loved the metaphore: This is my wildly speculative and loosely supported theory about what’s happening: A lot of us got into data because we were problem solvers who liked puzzles and weren’t afraid of numbers. We liked thinking creatively, but not like a capital-C Creative; instead, we liked finding interesting paths through structured problems. Don’t give us blank canvas or Word doc; give us a board game, a Lego, a brain teaser, or Wordle.
To me, there are 3 additional major components: Clarity + Experience + Compensation
Clarity:
DS got muddied over the past few years as companies blended analysts, data science, and ML/AI into a single "DS" bucket. When it came down to things, the majority of DS end up being Analysts and that is the least glorious of the DS jobs. They weren't excited about working in spreadsheets and creating line charts, they were excited about figuring out if you were pregnant before your closest friends/family even knew. Additionally, the work actually isn't bad. Cleaning data and modeling the warehouse is its own type of fun. The problem was that no one was rewarded for that work. Impact is over indexed during perf review and no one was pointing to the tables to show impact, they pointed to the analysis. The tables were just a means to the end. With the new model, there is a specific role to cover it and the impact of the table is actually captured.
Experience:
DS have had to do a lot of cleaning over the years, they built up expertise and tooling as they did so. Switching away from that and going pure DS would be a big hit to their day-to-day and comfort. They are also the obvious choice to staff this spot since they have been doing it a long time and are the go-to for doing this work.
Compensation:
There are two parts to this. (1) AE is a new role, there is a supply issue so you can demand a higher comp. (2) Engineers generally have a higher comp than DS. Analytics Engineers are often allocated as Engineers and therefore get a higher comp while still often getting a customized interview experience that is lighter on the leetcode and more focused on leetsql.
Another insightful piece, Benn. As someone who has made the journey from analyst to engineer, I feel everything you wrote in my bones. Analysis is open-ended and, for better or worse, judged on nebulous outcomes and not on effort.
If you spend two weeks digging into a data set only to realize there's nothing actionable in it, nobody's going to be happy, even if that discovery would be impossible without the work. On the other hand, if your analysis yields results that drive a win, you're always sharing that glory with your stakeholders. When you succeed, you succeed with the team. When you fail, you fail alone. That kind of sucks.
One additional point I'd make about the surge in analysts who want to try out engineering is this: how many analysts started out as analysts, and how many started out elsewhere in the business? I think it's important to remember that a lot of analysts have always been "hidden technologists"; they've had an interest in this stuff before they could even articulate it. I don't think it's been super uncommon for analysts to become DBAs, database engineers, data engineers, and even SDEs.
Now that there's an adjacent role with all of the cool software engineering tooling and best practices, I don't think it's any wonder that a lot of people are hopping to try it out.
Imagine if modern plumbing went this route...
Hi Ben, love reading/listening to you in podcasts as well.
Few questions:
1) Who did the cleaning and modelling prior to analytics engineer became a thing?
2) Does an analytic engineer now replaces this roll?
3) If the answer to question #2 is a no, would you say analytics engineer is simply a natural progress in a response to the creation of an enablement environment of tools and technologies(DBT and cloud data warehouse) which simply promoted self-service Semantic layer buildup inside the warehouse as opposed to in the bi tool itself?.
4) Do you see/already know if DBT is used elsewhere - not just serving analyst but more traditional roles like BIE’s/BI Dev’s and the likes?.
Thanks!
"Real analysis" is tricky to define.
Dashboards(unless the role is pure order-taking) have an analysis element.
Even creating a new dataset usually implies the analysis.
I am stuck as I really do enjoy finding insights, but I don't always like customers! :p Also don't always like the need for ad-hoc assumptions unless particularly elegant. (However, I really really don't want to be a data grunt to blame when things go wrong in prod!!!)
That being said a painpoint is that all analysis is constrained by the politics of the org and the barriers of communication.
Even a brilliant analysis can be misunderstood. Arguably the MORE BRILLIANT the analysis the HARDER TO CONVEY.
If everything was just nerds talking to nerds, I would not be surprised if there is more balance.
Consultants (who arguably share the field if analysis) have their own networking systems though, and less need/desire to show how the sausage is made. A lot of the practical reality of sharing analysis is like consulting.