20 Comments

I personally made the shift from data engineering to analytics engineering, rather than from being a data analyst to an analytics engineer. I needed more creative freedom and wanted to blaze my own path. With data engineering I always felt there was a stigma that things had to be done a certain way. With a new field like analytics engineering, there are no "experts". We are all trying to figure out what works and sharing that with others. I also get to create and implement my own best practices rather than following a textbook example.

Expand full comment
author

That's actually a really interesting angle I hadn't thought about before. It's not very often that you get to a chance to be pioneer a new field, where most of what's in front of you in unexplored territory. For a certain type of person, I suspect that's a very compelling draw.

Expand full comment
Jul 1, 2022Liked by Benn Stancil

The entire paragraph about why "a lot of us got into data" resonates so much with me I'm seriously considering adding it to my LinkedIn profile. This feels spot on Benn!

Expand full comment
Jan 8, 2023Liked by Benn Stancil

Stumbled upon your blog and literally devouring one blog post one after the other. I find them all so refreshing and insightful! Loved the metaphore: This is my wildly speculative and loosely supported theory about what’s happening: A lot of us got into data because we were problem solvers who liked puzzles and weren’t afraid of numbers. We liked thinking creatively, but not like a capital-C Creative; instead, we liked finding interesting paths through structured problems. Don’t give us blank canvas or Word doc; give us a board game, a Lego, a brain teaser, or Wordle.

Expand full comment
author

Thanks! I really appreciate that, and glad you like it!

Expand full comment
Jul 1, 2022·edited Jul 2, 2022Liked by Benn Stancil

To me, there are 3 additional major components: Clarity + Experience + Compensation

Clarity:

DS got muddied over the past few years as companies blended analysts, data science, and ML/AI into a single "DS" bucket. When it came down to things, the majority of DS end up being Analysts and that is the least glorious of the DS jobs. They weren't excited about working in spreadsheets and creating line charts, they were excited about figuring out if you were pregnant before your closest friends/family even knew. Additionally, the work actually isn't bad. Cleaning data and modeling the warehouse is its own type of fun. The problem was that no one was rewarded for that work. Impact is over indexed during perf review and no one was pointing to the tables to show impact, they pointed to the analysis. The tables were just a means to the end. With the new model, there is a specific role to cover it and the impact of the table is actually captured.

Experience:

DS have had to do a lot of cleaning over the years, they built up expertise and tooling as they did so. Switching away from that and going pure DS would be a big hit to their day-to-day and comfort. They are also the obvious choice to staff this spot since they have been doing it a long time and are the go-to for doing this work.

Compensation:

There are two parts to this. (1) AE is a new role, there is a supply issue so you can demand a higher comp. (2) Engineers generally have a higher comp than DS. Analytics Engineers are often allocated as Engineers and therefore get a higher comp while still often getting a customized interview experience that is lighter on the leetcode and more focused on leetsql.

Expand full comment
author

I agree on most of what you're saying about clarity, though I'd disagree that the analysts were necessarily the least glorious data job. That work - the "searching for insights in data" thing - is what so many people said they wanted to do. While it wasn't celebrated the say DS work was, it was still the work that people *said* they wanted to do. Which is the thing that's kind of ironic to me.

And on comp, agreed, I think that's part of the draw. I don't think it _should_ be, but I think it is. https://benn.substack.com/p/the-technical-pay-gap

Expand full comment
Jul 5, 2022·edited Jul 5, 2022Liked by Benn Stancil

agreed. definitely agree on the comp gap and i actually also agree with the clarification you put on analyst role, though i view being an analyst very similar to work like woodworking. meaning that the idea of doing it is much nicer than what it is like when you actually have to do it for your job (even when you remove what is now isolated for AEs). you do a little bit for a side project and you think "hey, getting paid for this would be cool".

there is certainly a subset of the group that really enjoys it and could happily do it for years and years. However, for many, they think they want to do it but then reality hits and they realize they only enjoyed doing analysis to find fun + cool quirks in the data rather than actually driving business value, maintaining dashboards, and pulling numbers for execs and PMs. their morale then gets hit when they realize they are being paid less than their peers, their work isn't nearly as celebrated, and most of what they do needs to be rebuilt/thrown-away every 6-18months.

analyst has a low barrier to entry and a common name so it is easier for it to show up as the thing they "said" they wanted to do

Expand full comment
author

The question all that raises for me is, then what's next? Analysts have this career path problem where it's not clear what roles they advance into, and that becomes even more pronounced if they also burn themselves out on being an analyst. So if you're that person, 18 months in, interested in the job in the abstract but disillusioned by its reality, where do you go next?

Expand full comment
Jul 6, 2022Liked by Benn Stancil

I think they transition really well into product management. The experience as analysts gives them a deep knowledge of the product and stakeholders and quickly ramps them up on how the company measures and defines success. they see all the good+bad and then as a PM they are in a place where they can drive change. fwiw, I think this is true for all of DS but is a harder sell.

the bad part is that there is a supply/demand issue. too many analysts, too few PM roles. I think some of this gets resolved in the coming years as we build more metric/semantic layers with metric focused UIs. this means we require less analysts due to effort deduplication, productivity improvement, and more self service.

the key to getting there is ensuring the metric layers extend into the T of the ETL layer and can surface context to consumers. tools like dbt are table aware but not data aware. they know things are happening but don't understand what is happening

Expand full comment
author

Yeah, I think that's a pretty narrow filter, to go from such a general role to a relatively specific one, especially since I'd imagine a lot of data people don't want to be PMs anyway.

The point about there being more data products - and therefore more pull for PMs to be data people - is interesting though. If we end up rebuilding lots of things on a data layer, it probably changes who needs to build those things, or at least the skills they need to have.

Expand full comment
Jul 1, 2022Liked by Benn Stancil

Another insightful piece, Benn. As someone who has made the journey from analyst to engineer, I feel everything you wrote in my bones. Analysis is open-ended and, for better or worse, judged on nebulous outcomes and not on effort.

If you spend two weeks digging into a data set only to realize there's nothing actionable in it, nobody's going to be happy, even if that discovery would be impossible without the work. On the other hand, if your analysis yields results that drive a win, you're always sharing that glory with your stakeholders. When you succeed, you succeed with the team. When you fail, you fail alone. That kind of sucks.

One additional point I'd make about the surge in analysts who want to try out engineering is this: how many analysts started out as analysts, and how many started out elsewhere in the business? I think it's important to remember that a lot of analysts have always been "hidden technologists"; they've had an interest in this stuff before they could even articulate it. I don't think it's been super uncommon for analysts to become DBAs, database engineers, data engineers, and even SDEs.

Now that there's an adjacent role with all of the cool software engineering tooling and best practices, I don't think it's any wonder that a lot of people are hopping to try it out.

Expand full comment
author

Yeah, I think that's a big thing that makes analytics hard - you can have big wins, but a lot of time, you come away with nothing. For analytics engineering (and, I'd guess, engineering more generally), there's a slow march of progress. You don't have many weeks that feel like a total loss (even though wandering is part of the process https://stkbailey.substack.com/p/wander-well). I hadn't thought much about that being something that people do alone, vs succeeding together, though that's an interesting angle.

On the pull of other roles, that's also a good point. I know a number of analysts who've moved to other roles (from PMs to operational leaders to data engineers, etc). So maybe the analysts are just a long-time disgruntled bunch..

Expand full comment

Imagine if modern plumbing went this route...

Expand full comment
author

Do plumbers want to be plumbers? (that's a genuine question)

Expand full comment

I don't think they do but then again I've never asked one. I have heard a few talk about their trade and mostly the message was that it was reliable work ( i.e. its a dirty job but someone has to do it).

Expand full comment

Hi Ben, love reading/listening to you in podcasts as well.

Few questions:

1) Who did the cleaning and modelling prior to analytics engineer became a thing?

2) Does an analytic engineer now replaces this roll?

3) If the answer to question #2 is a no, would you say analytics engineer is simply a natural progress in a response to the creation of an enablement environment of tools and technologies(DBT and cloud data warehouse) which simply promoted self-service Semantic layer buildup inside the warehouse as opposed to in the bi tool itself?.

4) Do you see/already know if DBT is used elsewhere - not just serving analyst but more traditional roles like BIE’s/BI Dev’s and the likes?.

Thanks!

Expand full comment
author

Thanks! My take on your questions...

1) It was a mix of analysts and data engineers, though in my experience, it was mostly analysts.

2) No, it augments / splits it. That part is now more analytics engineer, where analysts are (ideally) doing the analysis, answering questions, etc. Though that split isn't always that clean, and different companies do it a bit differently.

3) For the most part, yes. Though it's a bit of an uncomfortable split, because analytics engineers encroach a bit on the "analysis" work in some orgs (eg, is creating a report analysis or data prep?).

4) I think this very much happens. It's a broader architectural shift, where BI tools that used to fully own semantic models now share that with dbt, so dbt is used to solve a lot problems that BI devs would solve (even if dbt isn't _in_ the BI tool).

Expand full comment

"Real analysis" is tricky to define.

Dashboards(unless the role is pure order-taking) have an analysis element.

Even creating a new dataset usually implies the analysis.

I am stuck as I really do enjoy finding insights, but I don't always like customers! :p Also don't always like the need for ad-hoc assumptions unless particularly elegant. (However, I really really don't want to be a data grunt to blame when things go wrong in prod!!!)

That being said a painpoint is that all analysis is constrained by the politics of the org and the barriers of communication.

Even a brilliant analysis can be misunderstood. Arguably the MORE BRILLIANT the analysis the HARDER TO CONVEY.

If everything was just nerds talking to nerds, I would not be surprised if there is more balance.

Consultants (who arguably share the field if analysis) have their own networking systems though, and less need/desire to show how the sausage is made. A lot of the practical reality of sharing analysis is like consulting.

Expand full comment
author

I think that's the crux of the issue - the customers, communication, and politics *are* the analysis. It's like saying I want to be a shortstop so that I can field grounders, but not throw them to first. If you only do one, the job's not done. I think a lot of people get into the field thinking they can do the former and not the latter, which is what leads to the disillusionment down the road.

Expand full comment