Foreseeable developments in data

Dear Data-Traveller, please note that this is a Linkedin-Remix.

I posted this content already on Linkedin in April 2022, but I want to make sure it doesn´t get lost in the social network abyss.

For your accessibility-experience and also for our own content backup, we repost the original text here.

Have a look, leave a like if you like it, and join the conversation in the comments if this sparks a thought!

Link to my original post

Screenshot with comments:

Plain Text:

It’s great to have all the evangelist and happy talks about the modern data stack. It creates awareness and potential more adaption. And more adoption more experiments and this more knowledge. That’s the theory.

But it’s even more important to have reasonable and critical voices. The ones that turn the lights to the shady areas. Who give context to the storyline that shows risks of new things. Because as we know everything is an opportunity and risk at the same time.

Please read this piece by Lauren Balik (someone who should be on your follow list) about the potential situation in data in the next months and maybe years. It gives you an alternative narrative to the modern data stack and all the unquestionable love for dbt.

Down rounds are coming. Compression of capital availability will force M&A. Many will go out of business.

*cue Game of Thrones Dothraki voice*

“It is known.”

—

When retail compressed ~10 years ago due to the proliferation of ecommerce, the most exposed businesses were in the middle of the market. Macy’s. JCPenney. Sears.

To call them second-movers would be giving them too much credit. They were third, fourth, fifth, etc. movers with capital intensive businesses holding tons of inventory, stuck in capital intensive distribution models tied up in real estate and poor logistics networks.

Today, Macy’s survives, though much of the rest of the middle has slowly demised.

It takes a long time for a whale carcass to decay, but a dead whale is a dead whale. Along the way, all kinds of scavengers take their cut.

—

What does this mean for the data world?

-The tolerance for supporting larger-than-necessary data teams will disappear. Teams 50% of the size will suffice. Human capital is far more expensive and a much greater liability than tooling and services.

When you have a team of 4-5 people and need to cover $1-1.5m in employee costs and the business doesn’t even know last month’s MAUs, that’s an obvious area for cuts.

-Human capital intensive tooling will be at risk. Right now, there are some people at venture-backed companies who are 100% “Airflow guys” or teams of 2-3 “analytics engineers” who spend 90%+ of their time noodling around in dbt.

It’s simply absurd to think that SMBs and MMs will be able to cover this overhead here.

The cuts will be concentrated first in highly leveraged businesses with limited revenues, on data teams where human capital is considered (S)G&A.

-There will be a proliferation of solutioning around quality and cost control. Right now, the observability, reliability, quality categories are nascent. Cost control is almost unheard of and remains highly services based.

There simply will not be tolerance for cloud bills that could be halved for the same output, and nowhere in cloud bills is there more cruft and low hanging fruit than in data egress and in these bats— insane resource-intensive data models that rely on expensive human capital to build and maintain.

—

Where are the opportunities?

-Consolidated platforms, especially around commoditized functions (eg. data integration)
-Universal APIs (eg. Rutter, Plaid) and things like universal data models/ semantic layers
-Services, and managed services
-Cost saving, efficiency producing solutioning. Wider distribution of FinOps. The demand will occur. Many buyers of these solutions will be purchasing this type of solution for the first time. There will be education needed here.

Packaging FinOps, quality control, and adult behavior is *chef’s kiss* revenue, baby.

The days of throwing on a cowboy hat 🤠 and firing up the cloud warehouse to “do data stuff” are ending.

I’m excited.

Lauren Balik on LinkedIn