Architecture Weekly #142 - 28st August 2023

Aug 28, 2023

Welcome to the new week!

I think that prototyping is an underestimated design skill. We immediately fall to whiteboard diagrams, sticky notes, or endless discussions about the design.

But there's another way. I showed that in my last article using the following ingredients:
- Type-Driven Design,
- Decider pattern,
- Event Sourcing
- and code that with C# and Marten.

Thanks to the Type-Driven Design, we're getting expressiveness and simplicity. From Event Sourcing, we're sprinkling it with a focus on business. Decider helps to compose that, and Marten makes it real.

Oskar Dudycz - Prototyping, an underestimated design skill

To prototype our design quickly, we must use techniques that reduce cognitive load and favour simplicity over complexity. I’m a huge fan of slicing architecture by our behaviour, and using Event Sourcing or CQRS is not that it’s just hippy. It’s that I believe that those tools can make a difference.

That’s also why I enjoyed the .NET Rocks podcast episode, where Jeremy D. Miller discussed various perspectives on how to achieve Minimal Architecture and why it’s important:

.NET Rocks - Minimal Architecture with Jeremy Miller

The underestimated aspect impacting cognitive load is build time. We stopped caring about that optimisation and tried to solve it by buying newer, beefy computers. Funnily, since 2005, we know that Free Lunch is Over for hardware enhancements.

Google published the whitepaper explaining why optimising the local and CI/CD build pipeline is still important. Not only technically but also for us humans to be more productive. Read more:

Google - Developer Productivity for Humans, Part 4: Build Latency, Predictability, and Developer Productivity

Martin Kleppmann is one of my favourite authors in the distributed systems space. His book “Designing Data-Intensive Applications” is one of my best programming books. He’s a person involved in building solutions like Kafka and a scientist. In the last few years, he decided to bring the control back from cloud-first to local-first. He and his collaborators wrote the Local-first manifesto with Seven ideals for local-first software.

1. No spinners: your work at your fingertips
2. Your work is not trapped on one device
3. The network is optional
4. Seamless collaboration with your colleagues
5. The Long Now
6. Security and privacy by default
7. You retain ultimate ownership and control

So again, collaboration, ownership and productivity. If you want to learn more about how they’re progressing in the research to make it a real thing, not just an idea, read a coverage made by Wired:

Wired - The Cloud Is a Prison. Can the Local-First Software Movement Set Us Free?

Some call Cloud a prison, some a castle that’s under siege. Ok, Troy Hunt didn’t use such words but explained in detail the fight with API Bots with CloudFlare Turnstile:

Troy Hunt - Fighting API Bots with Cloudflare's Invisible Turnstile

As always, it’s great coverage, presenting issues and a solution for unexpected cases when you hit a bigger scale. Even if that might not touch you, it’s worth knowing that there are solutions to deal with that other than Captcha’s.

A month ago, I linked a recap of how AWS S3 was built; today, I want to show you the next solution built on top of it.

WarpStream - A Kafka compatible data streaming platform built directly on top of S3

Intriguingly, authors claim that it’s not tiered storage:

The Agent does not require local disks at all. Data streams directly from the Agent to object storage instead of being replicated via extremely expensive cross-AZ networking at an effective cost of $0.05/GB in AWS at retail prices. That's the same cost as storing data in S3 for over 2 months! Instead, the WarpStream Agent leverages the free networking between EC2 and S3 in AWS, which just so happens to durably replicate your data along the way.
(…)
WarpStream Agents make a few files per second, but each file contains records from multiple topics and partitions. In the background the pool of Agents compacts those small files into larger files to make reprocessing historical data for both single partitions and whole topics both cost effective and high throughput.

Check also more in their architecture overview.

I think such a creative way of using cheap blob storage can be more popular. Even AWS made a similar idea with AuroraDB.

Check also other links!
Oskar

p.s. I invite you to join the paid version of Architecture Weekly. It already contains the exclusive Discord channel for subscribers (and my GitHub sponsors), monthly webinars, etc. It is a vibrant space for knowledge sharing. Don’t wait to be a part of it!

p.s.2. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, and putting pressure on your local government or companies. You can also support Ukraine by donating, e.g. to the Ukraine humanitarian organisation, Ambulances for Ukraine or Red Cross.