The Road Ahead for Analytics Engineering Practitioners
So you’re an analytics engineer. Things are good. Lots of openings for people with your skills. Market is booming.
Tools are popping left and right to make your life easy. Communities are gathering. VCs are cannon-balling money. Ecosystems are being built. New practitioners are flooding in. Your practice is more robust, more professional.
A golden age! 👑
But there’s that nagging feeling. Will it last? Forever and ever?
I’ve argued in the Analytics App Store post that, as was web development, analytics engineering is currently a magnetic role which attracts a lot of creative forces that will eventually commoditise its practice. And that will leave us with a choice of which path to follow from that point on.
I remember very well being you my young analytics engineering jedi., but in that previous era of web development bonanza.
Skys were blue, contracts were pouring in, web development was an adventure and FrontPage was the laughing stock of our community (have you ever seen the html output from FrontPage? 🤢)
But then came in DreamWeaver, a bunch of Content Management Systems (Drupal, Joomla, DotNetNuke, etc.) and eventually… Wordpress.
You had a few options from there: get on board the Wordpress express 🚂 and become that expert; move on to bigger and better things with web app development and the advent of frameworks such as Ruby on Rails; or build your own product by following the Lean Startup framework.
Analytics engineers will have to face a similar choice in the future.
As ecosystems of analytics tools mature and building a data infrastructure will be just as simple as spinning up a few instances and widgets, where shall the analytics engineer go from there?
Honestly, I don’t see it being any different than the paths we had as web developers, but just with a different flavor:
- Go all in with one ecosystem, be its evangelist and superstar practitioner.
- Move on to developing data apps, by leveraging the newly commoditised analytics stack and wrapping those into new ways of consuming that data.
- Or build data products from the ground up, harvesting new data and packaging it up to serve new markets.
Let’s go over those 3 paths in more depth.
The first path is to go all in into an analytics ecosystem and become its expert.
What ecosystems am I talking about? Well let’s see. The market is starting to consolidate (acquisitions are increasing and with VC money backing our beloved tools of choice, it’s not a far fetched argument to say that the classic exit strategy is to sell to a bigger fish) and we can already see one ecosystem somewhat fully formed: it’s the big vast world of Salesforce Analytics.
Even if limited in its scope, there’s a world of analytics that can be done with Salesforce services at the heart of your stack. With data harvesting feeding its flagship CRM, you can now perform ETL tasks with Salesforce-owned Tableau Prep, build BI dashboard and ad-hoc analysis with Tableau, and automatically share your insights with Slack (also owned by Salesforce).
And at the current pace of acquisitions, we’re not done having more tools that will complement that analytics skeleton.
Another extension to Salesforce analytics’ strategy is that it’s not trying to be a completely hermetic walled garden. The strategy is that the core of your analytics stack is Salesforce-centric, but you could extend it with services outside of Salesforce.
For example, Tableau struck an agreement lately with Looker to push your ETL output directly to Looker. Meaning that you could replace Looker’s LookML with Tableau’s data connections and use that as a source for your Looker dashboards.
As you can see, there’s a lot of potential to being an ecosystem expert. As long as the ecosystem thrives and clients are sticking around and are willing to invest in their platform, there’s a lot of value in honing in to that one ecosystem.
Ask anyone that is currently a Shopify developer. They will most certainly vouch for that path.
The second path is to focus on the development of data apps.
This one’s less about data infrastructures, although it relies on building upon a mature one. Your work focuses on the end users and how they’ll consume that data.
There is a panoply of tools that makes data discoverable. But it’s more than that. It’s about the modern data experience. I love how Benn Stancil describes that:
To most people — pleasant, social people, the kind who can make it through a party without arguing about SQL formatting — the modern data stack isn’t an architecture diagram or a gratuitous think piece on Substack or a fight on Twitter. It’s an experience — and often, it’s not a great one. It’s trying to figure out why growth is slowing before tomorrow’s board meeting; it’s getting everyone to agree to the quarterly revenue numbers when different tools and dashboards say different things; it’s sharing product usage data with a customer and them telling you their active user list somehow includes people who left the company six months ago; it’s an angry Slack message from the CEO saying their daily progress report is broken again.
End users don’t really care about the architecture, they want the right data, in the right format, in front of their eyeballs and at the right moment.
And here enters operational analytics.
I had never really heard of this until I met Boris Jabes who demoed Census. But the whole idea is that data should be consumable outside of BI tools, Jupyter notebooks and SQL interfaces. It’s about moving the data in front of the user, where he is and where it’s needed.
Need to know the cost of acquisition of a user and to which channel you can attribute that conversion? Bam 💥, it’s right there in Intercom for you to look at.
This is the realm of what was once considered a business analyst (hear me out!). It’s about bridging the tech team to end users. Convert business requirements into technical requirements. (right?)
It’s really about squeezing the most value out of your data for your end users.
And on to the third path, for the entrepreneurial / intrapreneurial folks, where we build data products.
Now this goes beyond architecting data infrastructures and serving it efficiently. Just like the web2.0 revolution saw the advent of web businesses (that are the mammoths of today), building data products is to harvest, package and serve a specific market with a dataset that will be of value to them.
Having been a consultant in analytics engineering for the past 5+ years now, I’ve seen my fair share of data infrastructure and business requirements. With time, an analytics stack pretty much always has the same attributes and you have those goto tools that are used to build them up.
It’s not that those stacks are a commodity yet, but I / we could almost start putting a stack together wearing a blindfold — ok, maybe not, but you get the point. And it’s just a matter of months (don’t quote me on this in the future please 🙏) before a service appears and makes spinning up such a stack a commodity.
Even though there’s always going to be innovation in the periphery of that stack’s core, this path is more about acknowledging that a stable, flexible data infrastructure can be put together quickly. And the question now becomes what to do with those superpowers?
I started walking down that path with discursus, which is an open source data platform that mines, shapes and exposes the digital artifacts of protests, their discourses and the actors that influence social reforms.
In a recent post which covers its v0.0.2 release, I shared an up-to-date representation of its architecture. It doesn’t go into all of the details, but I’m sure that by looking at it and browsing the repo, you’ll find the whole thing familiar.
That journey is far from over as building the data stack is just one component of choosing that path. You are developing a market and therefore have to provide ways to consume that data and build a business around it. I won’t go into those details as this could be the topic for a future post.
So my young analytics engineering jedi, which path will you choose?
The ecosystem expert path, where you shall know all of its obscure little details and evangelise its usage?
The data apps path, where you will work closely with end users to better serve their needs and extract as much value of that data as possible?
Or the data product path, where you will create new markets and inject new data sources in domains other than finance and e-commerce.
