Is 2022 the break-through year for streaming in the modern data stack? I think yes.

Dear Data-Traveller, please note that this is a Linkedin-Remix.

I posted this content already on Linkedin in January 2022, but I want to make sure it doesn´t get lost in the social network abyss.

For your accessibility-experience and also for our own content backup, we repost the original text here.

Have a look, leave a like if you like it, and join the conversation in the comments if this sparks a thought!

Original Post:

Plain Text:

Is 2022 the break-through year for streaming in the modern data stack?

I think yes – read further to learn why it took so long and what is still missing.

Much wiser people like Kai Waehner or Robert Sahlin will smile when reading this. They know about the power and potential of streaming for years.

But I added modern data stack in my initial sentence on purpose.

In application setups I see more often Kafka as a foundation. And there has been so much investment to make Kafka easier to operate and so available even to smaller teams (see Confluent).

In the modern data stack everything is still based on batch. Hourly or daily loads.

But why?

Some reasons:

– there is no hard need for real-time at the moment. Most of the things where data ends up (90% in dashboards) are fine when they are updated regularly

– streaming is a whole new world to learn and operate (Kafka, Flink)

I think it’s a chicken & egg problem. When there are easier ways to handle data as streams people will use it more often and we will develop the use cases that 100% based on streaming data.

And you can see the glimpses already:

– Robert Sahlin is building StreamProcesser – a lightweight streaming processor for analytical purpose

– BigQuery supports stream inserts for some time, but invests more into it

– Jitsu can stream your event data into BigQuery already (much faster than Google Analytics 4)

– Rockset and Materialize offer production ready real-time databases

2022 will be the year the modern data stack starts to adopt streaming. Can’t wait to see it.