Architecture Weekly #155 - 27th November 2023

Nov 27, 2023

Welcome to the new week!

Event-driven systems are famous for immutability, and GDPR is famous for data removal. Does that mean that event-driven systems cannot be compliant with privacy rules?

Data governance practices are, unfortunately, not discussed enough in event-driven systems. People treat events more like freehand drawing, sometimes as mesh, but ending too often with a mess instead.

In my latest article, I explained both techniques specific to event-driven systems and tools like Kafka, Marten, and EventStoreDB and put them into broader strategic thinking about privacy.

How to deal with privacy and GDPR in Event-Driven systems

It’s a follow-up to the guide to GDPR for busy developers I published two weeks ago.

A reminder about tomorrow’s webinar about Leveraging BPMN for Seamless Team Collaboration in Software Development with Mário Bittencourt. We’ll learn how to use it to describe and document our business process to get a common understanding.

It’ll happen on 28th November at 6 PM. See more details.

Speaking about collaboration and modelling, I have two more resources for you:

They nicely explain the variety of tools we can use and their importance in building a collaboration culture in our teams. That’s critical to gain insights and perspectives from different stakeholders. Of course, it’s tricky, and we should also consider mental fatigue to make it sustainable.

I was mentally fatigued from last week’s OpenAI madness. Let me just put it here as it is, out of the chronicler’s duty…

The Verge - Sam Altman to return as CEO of OpenAI

Yeah… Let’s move on.

Moving the data on is the key thing in Event Streaming solutions. Having data is one thing, but processing it efficiently, passing it to others, analysing it, and correlating it is a growing challenge. We already have tools that provide the possibility to do it inflight (like Apache Flink), but the number of data to process is growing even faster.

Check what Confluent CEO Jay Kreps thinks about the future of it and how Stipe is doing that on a bigger scale:

Have a look also at the insights from Claire Caroll on how to get better skills in being a data analyst:

Claire Carroll - How to be a more Impactful Data Analyst

Automation is a critical part of continuous integration and deployment. But how do you fluently switch automation systems? If we’re running our project long enough, our tooling will need to change. That includes CI/CD, and there being fast and supportive of the process is extremely important.

For big projects such changes are too often postponed, as we don’t want to have our delivery process disturbed. Check two great resources from Spotify.

CoRecursive Podcast - Platform Takes The Pain with Pia Nilsson

Spotify - Switching Build Systems, Seamlessly

The first one is super intriguing as Pia Nilsson explains how they transitioned from full team autonomy, which too often meant an isolated island, to a more centralised platform. It’s a great story on transitioning the platform team mentality from playing with CI/CD tools and experimenting to becoming an enabling team for others. I love the term Platform Takes The Pain, it’s a great motto, as too often I saw that platform teams believe they’re the foundational and elitist team. At the same time, they’re there to help others deliver business value.

The seconds show how they switched their build systems for iOS applications to Bazel. It’s a great, not a long, but multilayered case study showing how to tackle such migration and what to consider while doing it. A nice operation on trying to cut the time you’ll have to build systems to the bare minimum. As always, the hybrid approach costs a lot.

Maintaining multiple build systems during a migration can be costly. In our case, we saw differences between the two build systems running in parallel which, at times, caused interesting compilation failures in one of the two build systems. For example, developers would report builds passing locally but failing in CI. Even though we had greatly reduced the time developers spent waiting for their changes to be tested in CI, our platform engineers now spent more time helping feature engineers debug build system issues.(…) We realized that the only way to completely remove this maintenance cost was to move to a single build system as soon as possible.

If you’re into Platform Teams, check this resource from CNCF:

CNCF - Platforms White Paper

They released their guidance explaining:

Why platforms?
What is a platform
Attributes of successful platforms
Attributes of successful platform teams
Challenges when implementing platforms
How to measure the success of platforms
Capabilities of platforms

And also remember, as Charity Mayors said in:

Yan Cui - Charity Majors on DevOps culture and team dynamic

It’s not about whether you deploy on Fridays or not; it’s about “owning your own shit”. So, be responsible for what you’re running and make yourself accountable for how you run and what you run on prod.

Mistakes can happen and will happen, but we need to be sure that we have the tools, skills and confidence that we’ll be able to handle that.

A nice inspiration for dealing with failures can be research from other industries, especially with more mature engineering practices. For instance:

Christine Shea, Chris. W. Johnson - Understanding the Contribution of Degraded Modes of Operation as a Cause of Incidents and Accidents in Air Traffic Management

Check also other links!

Cheers!
Oskar

p.s. I invite you to join the paid version of Architecture Weekly. It already contains the exclusive Discord channel for subscribers (and my GitHub sponsors), monthly webinars, etc. It is a vibrant space for knowledge sharing. Don’t wait to be a part of it!

p.s.2. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, and putting pressure on your local government or companies. You can also support Ukraine by donating, e.g. to the Ukraine humanitarian organisation, Ambulances for Ukraine or Red Cross.

Architecture

DevOps

API

Databases

Frontend

Testing

Microsoft - Consumer-driven Contract Testing

Java

Viktor Klang - Teaching old Streams new tricks

.NET

Industry

Security

Trivia

Christine Shea, Chris. W. Johnson - Understanding the Contribution of Degraded Modes of Operation as a Cause of Incidents and Accidents in Air Traffic Management

Architecture Weekly