The Analytics App Store

I grew up with the advent of personal computers and the Internet. At around 15, a friend of mine introduced me to HTML and we created personal interest websites with animated gifs of disco balls.

I got into Javascript and CSS, exchanging with small communities of similar youthful aficionados of web development. We signed our pages with “Contact webmaster here” with links to our ISP-attributed emails.

Thing were new, fresh, exciting and innovative.

I ended up studying computer science, with C++ and Java as my main languages, but I still preferred doing web development. So when I graduated in 2000, I worked as a web developer, changing the index.html file of my employer’s website directly on our Unix server. It was still the wild west, where I would implement a landing page change directly in prod and test it there.

It had been a fun ride so far and many others joined the party. Php was getting popular, and so was ASP. But for some reason the excitement worn off and I went back to school, looking for a subject that was as remote as possible to computer science as I could find — I ended up studying political philosophy…

So what do you do when you graduate with a degree in politics? You either pursue your studies all the way towards teaching, or you move on to something else. I moved on.

Or actually, it was more that I moved back to web development. Things had evolved, frameworks were springing out of everywhere and it was still a bit of a jungle out there.

A friend and I created our first business as a side project, which was called Thothle (long story, don’t ask). The app sourced from vendor APIs for textbook prices and referred searchers to the best deals out there. Referral programs were gaining momentum and we were getting paid a commission for books that ended up being sold because of our referral.

I built out the app in Ruby on Rails, and it was a breeze to do it. I remember going through a book that had all kinds of RoR recipes to build apps effectively. Much of the code was auto-generated by the framework and filling the gaps was so simple.

I got into Agile, Extreme Programming (is that even a thing anymore?), Scrum, etc. There were some great tools to do test-driven development, automated user testing and CI/CD. Git was slowly building momentum, although I was using Subversion (I think… seems like decades ago).

CMS like DotNetNuke (loved it!) and eventually WordPress made web development really accessible. Thothle eventually failed, as competition was increasing and we just couldn’t keep up. That experience thought me that web development had become a commodity and that just building out whatever idea you had wasn’t enough, you actually had to care about end users and what they wanted themselves.

That’s how I got more interested in the why than in the how. What to build should be driven by why it mattered, and not just because we could. And I guess the market was as well. The dot-net bubble burst had ripple effects.

Business analysts became a thing at that moment, at least where I was working. The role was essentially to be a bridge between “stupid” business people and “lazy” IT ones. At least, that’s how they both perceived one another.

They were living in a never-ending “Lost In Translation” movie. Nobody understood the other’s perspectives, and nobody seemed to willing to. That seemed like a great opportunity to be a bridge between biz and tech, and learn what to build and why.

I spent a few years doing that. I got to understand the business perspective of things, something I had never done previously. I got to understand business strategy, implementation and the operations. And how technology supports it.

I now had an external perspective on how web development was a commodity. We were coming up with web app’s requirements that were anchored in our own understanding of end users, and there just wasn’t any reason not to see it become reality: quickly, efficiently and cheaply.

Being that bridge between IT and business, I started to become the goto person for business units who wanted to get quantitative insights about the performance of their teams. I somehow had picked up Tableau along the way and was getting very excited about R.

It felt like the beginning of web development again.

It was rudimentary, it was a wild west, no method, just a few tools, but some real excitement. I remember big teams were slowly and painfully putting together their Cognos Cubes and I was just flying, answering business questions with R and Tableau within the same day.

There were small communities of practitioners, people who were technical enough, but also quite interested in the business questions that drove those analysis. I felt right at home.

I read The Lean Startup by Eric Ries when it came out and that ignited my brain. A framework to build businesses, using agile methodologies, and data. So cool! And then Lean Analytics by Alistair Croll and Benjamin Yoskovitz came out, I devoured it, my brain melted and I might still have scars from it.

I found an intersection of business, analytics and agile methodologies that was so exciting that I just wanted in. I had 3 small kids at that moment, I left my cushy job and became a freelancer in that domain (sleep deprivation makes you less wary of risk I guess).

I’ll spare you the details, but I got to work with a Canadian-based marijuana company, a Wall Street alternative data shop, a European influencer market and I was hooked. I had some really bad gigs as well, but I concentrated on the good ones and got the daily dose of dopamine dump I was looking for.

The future looked good! And it got even better when I stumbled upon dbt.

As companies relied more heavily on analytics, just pulling data manually and doing analysis on top of it didn’t work out anymore. So even though I spent 80% of my time in R and Tableau doing data analysis, I had to start automating the production of data.

Enter Python and my ETL nightmares. I honestly had no clue what I was doing. I was writing scripts that were pulling data from multiple sources, transforming it and pushing it to Mysql. I had cron jobs set up and very crude notifications of when errors were popping off.

And they were popping off. CONS-TANT-LY!

Hence the nightmares. Whenever my mobile phone vibrated, I tensed up as I was expecting to see problems with my data pipelines that might require me to take care of it. Now!

Ah, the joys that dbt brought to me. Stitch was equally miraculous as well. And Redshift. The entire modern BI stack for that matter.

All that to say that the fine folks at Fishtown Analytics (who created dbt and hosts their awesome community of practitioners) had felt that itch way before I did and came up with such an elegant solution that it not only solved that data transformation issue, it re-introduced all of the devops concepts I had loved in the past and made them available again. Forever grateful!

Version control, testing, continuous integration and deployment. The works!

A process was emerging again. Tools and methodologies. I got to better understand what data warehousing was all about. Dimensional modeling became a thing for me. It all, made, sense!

And slowly, really gradually, it started to feel just as how web development felt to me at some point — organized, coherent and well tooled. It wasn’t the wild west anymore. There was a structured approach to it and anybody could jump on that wagon.

And they did and still do. You just have to look at the size of that dbt Slack community to convince yourself. Thousands of people are now part of it!

Modern BI stack practitioners are joining in the fun and they’re pushing the envelope, coming up with great ideas, elegant execution of those ideas. Shit is evolving!

It’s not that there aren’t any challenges left in the modern BI landscape. Just like there still are challenges with web development. It’s just that there are less barriers to building up high-quality BI infrastructures quickly and cheaply.

So my question is: are we entering the commoditization of the modern BI architectures?

My gut feeling is that we are. Now what are the implications of that? Just like iPhones made mobile phones a commodity, what is going to be the “App Store” equivalent for modern analytics?

When web development became a commodity, end users expected more than just animated disco balls to look at. You had to figure out what users were craving for, even anticipating it. All those bells and whistles about the technologies you used became unimportant. User value became what was important.

I think it’s starting to be the same with modern analytics. Now that businesses can so easily own their data and store it in a well-structured and expandable format, what’s the next layer that will evolve on top of that architecture?

The best recent example for me has been this recipe that Fishtown Analytics wrote up on doing conversion attribution in your data warehouse. That’s a common business problem that can now easily be solve by owning your data in a cloud data warehouse.

How about other business questions? Just in the product space, there are many that could be packaged. How about recipes for user lifetime value, churn prediction, feature engagement, experiment analysis, etc? That’s what end users are interested in after all.

Can we go further than that? Could modern BI stacks fuel a 3rd wave of product analytics (after Google Analytics, then SaaS apps like Amplitude and Mixpanel)? I think so. You can replicate what those tools are doing, but in your own BI architecture. And you can go further than what their features allows.

And what about other functions, such as marketing, financials, operations, HR, client support, etc?

To me the commoditization of the modern BI architectures is exciting. Just like with web development and iPhones, it allowed for an explosion of a new economy. It could be argued that the analytics App Store is just around the corner.