Fork your dependencies, trim them to only your use case, never update unless it breaks for your users. I’ve been vocal about this for 10+ years. I’ve always said that updating is way riskier than latent bugs (which can be tracked and CVEs monitored).

If you are updating a dependency, it’s on you to analyze every single commit in the full transitive set of dependencies. If you dont see anything compelling, dont update!

I remember at HashiCorp once in awhile an engineer would try to update a dep or replace a DIY lib with an external one and id always ask “show me the commit we need.” Dont update for the sake of it.

Feeling pretty swell about this mentality with all the supply chain attacks happening.

That's from Mitchell Hashimoto. A friend sent it to me, and the first word he reached for was bold. Mine too. I mostly agree with him, honestly. But the second I read it, my head jumped to a caveat, and the caveat turned out to be the thing I actually wanted to write about.

You can fork a small library and trim it to what you use. I’ve done it, it’s fine. But could you fork React and maintain it? I couldn’t. I don’t think most teams could either. So “fork your dependencies” is wonderful advice right up to the point where the dependency is too big to hold in your hands, and then it quietly stops being advice at all.

And that caveat says something about the advice itself. I think that it was never really about forking. It’s about knowing exactly what you’ve taken on and being willing to own it, and that’s the bit most of us skip. We don’t decide to take a dependency. We install one. And almost everything that goes wrong with dependencies:

• the supply chain attacks,

• the licence dramas,

• the half-finished SBOMs,

• the things you find out about only after they break

comes back to that one move: we took dependence on someone else's product without ever deciding to. Everything else here is a variation on it.

To fork or not?

I was reminded of this not long ago. I was doing a live Q&A about my TypeScript port of some code, and a whole chunk of it was one question asked five different ways:

Why did you write your own deepEquals instead of just pulling a package?

I tried to explain that it wasn't laziness, and it wasn't not-invented-here pride either. I explained that it’s the code I write once and forget. Of course, it’ll take some time, but not that long as you might think. The reception was, let's say, mixed. I get it, it’s much easier to just install a package and outsource maintenance. Especially when you're in a hurry, you don't decide anything. You reach. And this reach can be a decent interim solution, but then you should reflect on yourself on the next step.

Why do I reach for my own code on the small stuff? Look at which packages actually get hit in all these supply chain attacks we keep reading about. It’s almost always the little ones. The one-liners. Remember the Left-pad Incident? It’s usually some tiny or low-level helper nobody thinks about. They’re everywhere, they’re trivial, and precisely because they’re trivial, nobody is watching them. Which is also exactly why they’re so easy to write yourself. So for me, handwriting a small helper isn’t paranoia. It’s the calmer option, and it’s one of the few cases where I know exactly what’s in my own system.

That's also why, in Emmett, my Event Sourcing library, I'm almost stubborn about keeping the core package free of dependencies and limiting whatever I inject elsewhere. Probably too stubborn, if I'm honest. But I'd rather err that way, because when it goes wrong on the user’s side, it goes wrong far worse than a bit of code I maintain myself. It’s about responsibility for the outcome of your actions.

If I had to compress what I believe into one line, it wouldn’t be “fork everything”, and it wouldn’t be “write everything yourself”. It would be something duller than both:

The number itself isn’t the point. It just tells you how much you’re agreeing to own without ever seeing it.

Because your direct dependencies were never the worst part. You chose those. Most of the time, you can read them, and you can follow what they’re doing. The scary part is the dependencies of your dependencies, and theirs, all the way down, the part you didn’t choose, can’t see, and have no say over.

And I want to be careful here, because it’s easy to let this slide into another round of JavaScript-bashing, and that’s not what I mean. Every ecosystem has this. JS and TypeScript just sit at one far end of it, where there’s a package for absolutely everything. Which is good, in general, you’re not rebuilding the wheel every other week, the way you are in some other places. But it’s also how you end up with a node_modules you couldn’t fully explain if someone put a gun to your head.

At the other extreme, there’s Microsoft and .NET, where the instinct runs so hard the opposite way that it tips into Not Invented Here Syndrome. Neither end is the “right” one. They’re both defaults people drift into without ever making a call.

So for me, it’s not about reaching zero dependencies. But having dependencies that we cautiously agreed upon.

Which takes me to the part that, in my experience, almost nobody does. You can’t make a call on what you can’t see, and if you don’t even have the basic knowledge (e.g. some list) of what you depend on, then every conversation about supply chain risk is a bit of theatre.

Dependency Inventory

In most of the environments, there are tools to generate the Software Bill of Materials - the inventory of your dependency tree. In some, they’re even built in. It’s easy to dunk on NPM, but it’d be better to do due diligence before doing so. Not many people seem to know this, but recent versions of npm ship an npm sbom. So the tooling exists, even in NPM. That isn’t the problem.

The problem is that most organisations have never generated one in their life. No SBOM, no inventory, nothing written down anywhere. So the day the next Log4Shell lands, and there will be a next one, they can’t answer the very first question anyone will ask them: do we run this, and if so, where?

On the other hand, tools often don't help here, even those built to do so. NPM audit mostly does the opposite. I honestly can’t remember the last time I installed something, and the audit didn’t immediately tell me to bump a stack of packages. Most of it is false positives, with no real attempt to say how dangerous any of it is. And that lands you in the oldest trap going: if it’s always red, you stop looking at red. So the one signal that was supposed to make you stop and decide ends up training everyone to decide nothing.

Bus factor and rug pulls

There’s a related thing I can’t quite leave alone, so let me wander into it. I think a lot of teams spend their energy on the symptom and never once look at the source.

Watch what happens in the .NET world whenever a popular package changes the deal. Fluent Assertions went commercial. Moq shipped a thing that quietly hashed your git email and phoned it home. MassTransit and AutoMapper announced commercial licenses within the same stretch. And nearly every time, the reaction across .NET shops is identical. It’s a mixture of

• let’s rip it out and write their own,

• search for a free alternative

• Cry to Microsoft to buy the lib or provide a replacement,

Essentially: a throw-the-baby-out-with-the-bathwater strategy.

And for me, that’s solving the wrong thing entirely. The source isn’t that the package started charging money, or pulled a rug. The source is that we took on a critical dependency without ever admitting to ourselves that it was critical, and never once thought about what we’d do if the terms changed. We didn’t consider the bus factor, and we didn’t do due diligence to ensure the work on it was sustainable and could continue. Pulling it out and hand-rolling a replacement fixes none of that. It just resets the same trap, this time with only the code we maintain.

The IdentityServer episode was the clearest version of it I’ve seen. People were upset that they had to pay suddenly. Then, in the next sentence, calling it a critical security component. Then, in the sentence after that, asking what the free alternatives were. A critical security component that you want for free and are ready to swap out overnight is, to my mind, asking for a security incident.

And there’s a bit of maths that quietly settles most of these arguments, if anyone bothers to do it. Take what the licence costs you per year. Then take into account what it would cost to have an engineer build and maintain your own version. Put the two side by side.

Almost every time, paying the maintainer comes out cheaper, and on top of that, you’ve lowered the bus factor on something you already lean on, which is its own kind of supply chain security. “We’d write it ourselves, but then we’d have to maintain it” is true. I just read it as the argument for paying the person who already does, not against it. If you depend on something, its survival is your problem too. That’s part of owning the decision.

I know this case too well.

LLM as a fork

Getting back to Mitchell’s thought. The part I find most interesting is because of the moment we’re in. I keep hearing that LLMs change all of this, that writing your own small things is suddenly trivial, so the whole dependency question softens. I don’t buy it. It’s never that easy. Writing the small thing was never the hard part anyway. Owning it, understanding it, maintaining it, being the one on the hook when it breaks at 2 am, that’s the hard part, and no model takes that off your plate.

I don’t see how LLMs can change the cost of owning code. They can (maybe) change the cost of producing it. That doesn’t fix the “install without deciding”. The old move was install and move on. The new move is “vibe it” and move on. Same missing decision, new flavour. The same lack of responsibility and ownership.

This trend isn’t new. It’s a classic Shadow IT. If you haven’t been around long enough to run into the term, Shadow IT refers to the tools and systems people build or adopt within a company without going through whoever is officially meant to approve them. The spreadsheet that quietly runs a whole department. The little script someone wrote on a Friday that half the team now depends on. The integration nobody in the platform group has ever heard of. It has always existed because people route around slow governance to get their job done, and most of the time, nobody notices until it breaks.

With LLMs, it’s more tempting than it has ever been. Someone in sales promises a customer a feature the team supposedly needs. The team has no time, so they cobble a tool together from the API and ship it. It doesn’t work. The customer says they’re not paying for this. It escalates. The thing was unowned from the moment it was conceived; nobody decided to take it on, it just appeared, and the blame game is starting.

And here’s where I think it all settles, because the corporate steamroller flattens everything in the end. Companies will dictate the allowed list, the way they always have. The cautious majority will stick to what’s known and popular: React, TypeScript, Python, Spring Boot. That’s what they did last time, and the time before. And the people who want to move faster will do it off the books, with an LLM, as Shadow IT. The declarative, standards-based frameworks that hide their complexity will do well in that world, because that style suits how these tools work, but it’s the same ceiling as before. You bet on React. You don’t own it. The small stuff you can hold; the big stuff stays as bet.

What to do then?

We cannot fix the entire software industry, but we can fix how our own engineering teams operate. Instead of waiting for automated audits to scream at us, or ripping out packages in an emotional panic, I suggest a simple, regular exercise for your organisation.

Sit down and explicitly define your dependency posture:

1. Inventory: List the dependencies you use (even without peer dependencies). Use tools npm-sbom to actually see what you are pulling in.

2. Criticality: Identify which of these packages are absolutely critical to your system.

3. Lifecycle: Define a clear strategy for upgrading and versioning them. Are you updating just for the sake of it, or are you looking for specific commits like Mitchell suggests?

4. The Bus Factor: Ask yourself: what happens if the author of a critical package gets hit by a bus, burns out, or the tool becomes paid?

5. Mitigation: Decide on a concrete backup plan for that exact scenario. Do you fork it? Do you pay the license fee? Maybe pay earlier for support or help in another way to maintain it.

6. Response Time: Estimate how quickly you can upgrade and deploy the application if a major security breach occurs in a dependency. Also, if the strategy is to use replacement, then how fast will you be able to replace this dependency?

Building reliable software requires intent. We don't have to write everything from scratch, but we must be precise about what we bring into our software. Architecture is not just about writing code; it is about choosing which liabilities you are willing to own.

Cheers,

Oskar

p.s.2. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

How soon is now? In PostgreSQL, it’s not always as soon as you’d think. I learned that the hard way recently, so you don’t have to.

It took me hours and wasn’t easy to reproduce, even though the fix is one line. I found it in a Cybertec post, as I quite often do when I’m staring at something odd in PostgreSQL. I’m supposed to know my way around the database, but I missed it, which is another reason I want to write this down.

I was working on distributed locking in Emmett. When you scale a service horizontally, you can easily end up with two instances of the same message processor running at once. That’s bad. Both instances would pull the same events, both would write to the same projection storage, and we’d get duplicated side effects, overwritten state and broken checkpoints. So we need to guarantee that exactly one instance of each processor is active at any time. Emmett does that using two things working hand in glove: PostgreSQL advisory locks and a row in the emt_processors table. The row keeps the durable side of ownership: which instance currently holds the processor (processor_instance_id), when it last checked in (last_updated), and what state it’s in (status). I described the full design in Rebuilding Event-Driven Read Models in a safe and resilient way, so I won’t bore you with the whole picture here.

For this story, the part that matters is what happens when an instance crashes. The crashed processor’s connection is gone, so its advisory lock has already been released. A new instance can grab the advisory lock without resistance. But the row in emt_processors still says status = 'running' and still points to the previous owner, because the crash didn’t give anyone a chance to clean it up.

From the outside, we can’t tell whether the previous owner has crashed or is just between heartbeats. So we wait. If the row’s last_updated is older than a configurable timeout, the new instance is allowed to claim ownership anyway. Anyone quiet for that long is treated as gone. To make this graceful, the lock acquisition runs inside a retry policy. A fresh instance starting just after a crash doesn’t fail straight away; it retries until the timeout window expires.

The bug

The takeover decision lives in the upsert against emt_processors. In the real function, that upsert sits inside a Common Table Expression (CTE) alongside a pg_try_advisory_xact_lock call.

For the record: the snippets below skip that wrapping (and trim a couple of unused parameters) to keep the focus on the upsert, where the bug lives. The full version is in the source.

CREATE OR REPLACE FUNCTION emt_try_acquire_processor_lock(
    p_processor_id           TEXT,
    p_processor_instance_id  TEXT,
    p_lock_timeout_seconds   INT
)
RETURNS BOOLEAN
LANGUAGE plpgsql
AS $$
BEGIN
  INSERT INTO emt_processors (processor_id, processor_instance_id, status, last_updated)
  VALUES (p_processor_id, p_processor_instance_id, 'running', now())
  ON CONFLICT (processor_id) DO UPDATE
  SET processor_instance_id = p_processor_instance_id,
      status                = 'running',
      last_updated          = now()
  WHERE   
     -- same instance reconnecting
     emt_processors.processor_instance_id = p_processor_instance_id                      
     -- previous owner stopped cleanly
     OR emt_processors.status = 'stopped'     
     -- previous owner timed out    
     OR emt_processors.last_updated
        < now() - (p_lock_timeout_seconds || ' seconds')::interval;
  RETURN FOUND;
END;
$$;

The last branch is the takeover. It reads naturally: if the previous owner hasn’t checked in for longer than the timeout, the new instance can replace them. All tests were green. Stop me if you think you’ve heard this one before. The problem surfaced through user feedback (thanks, Martin!), and it took me a long time to reproduce; none of the existing tests covered the scenario that triggered it. Once I had a new end-to-end test that pinpointed the symptom, the rest was the usual, long, boring, debugging loop.

To see why, open psql and run this:

BEGIN;
SELECT now() AS tx_now, clock_timestamp() AS wall_clock;
SELECT pg_sleep(2);
SELECT now() AS tx_now, clock_timestamp() AS wall_clock;
COMMIT;

You’ll get something like:

            tx_now             |          wall_clock
-------------------------------+-------------------------------
 2026-05-25 10:00:00.123456+00 | 2026-05-25 10:00:00.124012+00

            tx_now             |          wall_clock
-------------------------------+-------------------------------
 2026-05-25 10:00:00.123456+00 | 2026-05-25 10:00:02.131845+00

The first column is the same in both rows. The second one is two seconds apart. As it turns out, now() is a synonym for transaction_timestamp(): it returns the time the transaction began, and keeps returning that value for every statement inside the same transaction. A light that never goes out, in other words. clock_timestamp() reads the wall clock each time it’s called, so it advances as time does. Cybertec wrote a good walkthrough of the whole family of timestamp functions if you want the full picture.

What difference does it make? For a column like last_updated, the constancy of now() is usually what you want: every row touched in the same transaction shares a single timestamp, which keeps audit logs and write batches coherent. For asking “has enough time passed?” inside the same transaction, the same constancy works against us.

Now back to the retry. The consumer that calls tryAcquire looks roughly like this:

pool.withTransaction((tx) =>
  asyncRetry(
    () => tryAcquireProcessorLock(tx.execute, options),
    { retries: 10, minTimeout: 200, maxTimeout: 1000 },
  ),
);

pool.withTransaction opens a database transaction and passes its executor to the body. asyncRetry then repeatedly calls the stored procedure on that same executor, with a backoff between attempts. So even though the retries are spread out in real time, every call runs inside the same database transaction:

withTransaction        (transaction starts at T)
  └── asyncRetry
       ├── call lock function    → now() = T
       ├── call lock function    → now() = T   (200 ms later)
       ├── call lock function    → now() = T   (400 ms later)
       └── ...

last_updated < now() - timeout evaluates the same way every iteration. The predicate is effectively constant for the lifetime of that transaction. From the database’s perspective, no time was passing between attempts, even though the retries were spread across real seconds. (of course, the valid question is whether retries should happen inside a transaction, but let’s say that this is out of scope of today’s article, deal?).

So what was the fix? Change the time source inside the function. PL/pgSQL lets you declare local variables, so I added one at the top, initialised from clock_timestamp(), and used it everywhere the function previously called now():

CREATE OR REPLACE FUNCTION emt_try_acquire_processor_lock(
    p_processor_id           TEXT,
    p_processor_instance_id  TEXT,
    p_lock_timeout_seconds   INT
)
RETURNS BOOLEAN
LANGUAGE plpgsql
AS $$
DECLARE
  v_current_time TIMESTAMPTZ := clock_timestamp();
BEGIN
  INSERT INTO emt_processors (processor_id, processor_instance_id, status, last_updated)
  VALUES (p_processor_id, p_processor_instance_id, 'running', v_current_time)
  ON CONFLICT (processor_id) DO UPDATE
  SET processor_instance_id = p_processor_instance_id,
      status                = 'running',
      last_updated          = v_current_time
  WHERE emt_processors.processor_instance_id = p_processor_instance_id
     OR emt_processors.status = 'stopped'
     OR emt_processors.last_updated
        < v_current_time - (p_lock_timeout_seconds || ' seconds')::interval;
  RETURN FOUND;
END;
$$;

clock_timestamp() ignores transaction boundaries. Every call to the stored procedure reads the wall clock fresh, so each retry sees a slightly later value than the last. After enough retries inside the wrapping transaction, the takeover predicate flips, and the new instance wins.

The function uses the timestamp in two places: when setting last_updated on the new owner, and when comparing the previous owner’s last_updated to the timeout. I switched both to v_current_time, so the write and the check read from the same wall clock. Mixing clock_timestamp() on one side and now() on the other would leave a subtler version of the same bug.

A cleaner option for the future is to move the retry one layer up, so each attempt opens its own transaction. That would remove the trap entirely and let the function go back to plain now(). For now, the local variable does the job.

Why my tests didn’t catch it

Now to the testing side. I had a careful suite of integration tests for the stored procedure. Two instances racing for the lock. The same instance reconnects after a crash. Takeover after a custom timeout. They all passed, and they could not have caught this bug. Here’s the shape of a typical one:

await pool.withTransaction((connection) =>
  lock.tryAcquire({ execute: connection.execute }),
);

That’s the shape: set up some state, call tryAcquire once, check the result. A single call to the stored procedure works fine with now() in the WHERE clause. There is only one timestamp involved per call, and the predicate evaluates correctly against it. The bug only shows up when several calls share one transaction, which happens when the consumer’s withTransaction wraps a retry policy. The stored-procedure tests never put those two together.

The end-to-end consumer tests do go through the consumer’s withTransaction wrapper, but they covered the happy paths: clean start, clean stop, two consumers competing, and an instance reclaiming its own stale lock. None of them combined the three conditions that together expose the bug:

1. a previous owner whose row still says status = 'running' (a crash, not a graceful stop),

2. a new instance arriving with a different instance ID,

3. a retry acquisition policy with a timeout short enough for the retries to outlast it inside the test’s deadline.

Any one of those missing and the takeover predicate either succeeded on the first attempt (so the retry never fired), or the test finished before the retry’s failure to make progress was visible.

So why did this slip through both layers? The stored-procedure tests never combined a retry policy with the stale-row state, so they never produced multiple calls inside one transaction. The end-to-end tests did exercise the retry path, but none of them happened to combine all three conditions above at once. Both layers had blind spots, and the bug lived exactly where they overlapped.

The Pull Request that fixes the bug also adds a new family of tests that mount tryAcquire under the same transactional wrapper the real consumer uses, with the crash + new-instance + retry combination wired up on purpose. That’s the kind of test I should have had from the start.

What I’m taking away

Two things about now():

• now() is the right tool when you want every row touched in the same transaction to share a timestamp. created_at, last_updated, audit columns. That stability is a feature.

• now() is the wrong tool when you want to ask “has time moved on?” from inside a transaction. Use clock_timestamp() when you genuinely mean the wall clock.

And one harder thing about tests. Inner tests for stored procedures give me a tight feedback loop and pinpoint failures, but only inside the scaffolding I build for them. End-to-end tests run the real wiring, but I can’t enumerate every combination of timeouts, instance IDs and crash states without the suite collapsing under its own weight. The combination where this bug lived wasn’t reachable from either side by accident; it needed a deliberate setup.

I don’t have a clean rule for where to draw that line, and honestly, I don’t think there is one. My takeaway is to look at the seam: the spot where the inner test invokes the code differently from how the production caller invokes it. Here, it was a single-call test against a retry loop sharing one transaction. Wherever that gap sits, write a test there. Not at the unit level, not at the full end-to-end level, but in a setup that mirrors how the real caller actually drives the code and exercises the path you care about.

TLDR

now() returns the start of the current transaction, not the current moment. Inside a transaction it doesn’t change between statements. If you wrap a retry loop around a function that uses now() in a WHERE clause, and the retry loop runs inside one transaction, the predicate is frozen and the retries do nothing. Use clock_timestamp() when you mean “right now”. And pay extra attention to the seam between inner and end-to-end tests, because that’s where mismatches between how tests drive the code and how production drives it tend to hide.

The fix and the new tests live in Emmett Pull Request #339.

Uff. That bug was nasty.

Read also:

• Rebuilding Event-Driven Read Models in a safe and resilient way, with the locking design this bug lives inside,

• Distributed Locking: A Practical Guide,

• Consumers, projectors, reactors and all that messaging jazz in Emmett,

• Checkpointing the message processing,

• Cybertec: PostgreSQL now() vs now()::timestamp vs clock_timestamp(),

• Is keeping dates in UTC really the best solution?.

Or my other articles about PostgreSQL:

• Postgres Superpowers in Practice,

• PostgreSQL partitioning, logical replication and other Q&A

• PostgreSQL JSONB - Powerful Storage for Semi-Structured Data

• Push-based Outbox Pattern with Postgres Logical Replication

• How to get all messages through Postgres logical replication

• The Write-Ahead Log: The underrated Reliability Foundation for Databases and Distributed systems

• How Postgres sequences issues can impact your messaging guarantees

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

For instance, I believe there’s a strong synergy between C4 Model, Context Maps, and EventStorming. They all allow us to look at the system from a different angle and act as simulations, providing different feedback on whether our model will fly.

Look below for the diagram I prepared for my upcoming workshop.

It’s a C4 diagram showing containers in a hospitality system (so-called Property Management, where property means hotel).

A container gives quick feedback on how many pieces I’ll potentially need to manage and deploy, and it also shows me boundaries.

Context Maps help in understanding how they relate to each other, which module or team has bigger leverage, and can force more on others. It can also show me the information flow and simulate which module will expose the API.

I can then take some business process and have a look at how the message flow will look. This can happen as an early simulation or after EventStorming sessions.

Then C4 allow me to also look inside the container, show components if I need to understand more details. I can group containers if my bounded context has more of them.

How to start with it? Let me show you an example.

The starting point could be understanding the system's overall view; integrations with external systems usually introduce the most complexity. C4 System Diagram works well for that.

We see actors representing our system users and interactions with external systems. We can decide what integration is out of scope for now, and may come as the second step, but not for now. We can also show the current state as is, then in the second diagram show our vision of what we want it to be. Be creative, this is not a relational database, we don’t need to normalise it.

Then we may decide to dive into what we need to build to facilitate that, make a bet based on our current knowledge, zoom in and check the C4 Container view.

As mentioned, this can already be used both for static documentation and for discussions during a workshop, simulating the potential complexity and boundaries. Important to note: until it’s settled, it’s more than fine to show different versions, discuss them, and add notes and questions.

We may have concluded that C4 is fine, but it doesn’t capture all the important reasons we modelled it this way. Or during the workshop, it may not be clear how to discuss precisely how to break it down. Context maps can help in that.

They can give us information on which module is generic (Open Host), which module (or team) is more important (upstream), and which module or team has less leverage (Downstream). Which provides api or data (Supplier) and which one is using it (Consumer).

We could have a dedicated model only focusing, on that, but we could also capture it in the same one. It could look as follows:

We can even use colours, e.g., upstream as green and downstream as red. This can already give us feedback, e.g., that Cashiering is an important module for financial processing and that it is downstream of multiple modules. Maybe we can change it and redefine boundaries, or maybe live with it, but take some corrective actions.

We can also put more data and use a specific example of a process and have a look at the specific process, as shown in the original diagram:

This can also pinpoint some issues, e.g., a generic module, which is more likely to expose commands (see more on why in my article on Passive-Aggressive Events). We can see if we have all the needed ACLs defined, or which module acts as a coordinator.

Of course, keeping it all on the same diagram is not perfect; it’d be great if we had a tool that would enable zoom-in/zoom-out, link and enable/disable some of the details and notation. Then we can even better play with what we have. Maybe I should vibe such?

Nevertheless, I encourage you to try different techniques, experiment, and combine them. For instance, Example Mapping would, imho, play great here as an extension. I know that some use Wardley Maps or Domain Storytelling.

Start small, use existing techniques and tools, and try to see how you could benefit from mixing them. Show it to your friends and try to collaborate and have fun together.

I even learned from Richard Gross that this has an even name: Model Storming. So, crunching your design with different modelling techniques and coming up with some useful variations.

I have even more mashups like that. Tell me if you’d like me to expand on it or this part in the follow-up articles!

Also, if you did such experiments, please share them with me and others. Happy to see what you came up with.

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

I was toying with LLM-based domain research. It reminded me of the common mistake we make when we try to practice DDD: overreliance on what domain experts are telling us.

I wanted to remind myself of the domain I used to know: hospitality management, to adjust my upcoming workshop. I selected LLM (Claude Opus) as a sparing partner. And boy, I got a loooot of details. I was swamped by them. The LLM modelled everything: marketing consent, loyalty timing, inventory management, revenue posting, regulatory submissions, data retention policies, all at the same level of detail, which buried the core checkout flow under noise, the thing I asked about.

That’s not much different from initial work with domain experts. When we’re starting discovery, people tend to start by explaining everything they do and feel is important. Quite often, that means you’ll get more domain-expert pet peeves in the surroundings than in the process descriptions.

You also get a lot of jargon, words that sound familiar but mean something different.

That’s also what I got from LLM, I asked it to review reference sources like:

• popular tools documentation,

• open specifications from the hospitality organisations,

• some laws like GDPR related stuff,

I researched how different tools handle the guest checkout process. I knew it, and implemented it in past; it’s a surprisingly complex process, as you need to:

• verify if the stay is fully paid and close the financial account,

• generate invoice

• mark guest’s stay as completed,

• schedule full room cleanup,

• mark in the inventory that the room will be available soon (unless the maid finds that it’s broken),

• cleanup GDPR-related data that’s not needed to keep,

• adjust loyalty points and update CRM,

• etc.

I remembered how it works, but not all the details, and I would like to double-check whether anything in the industry has changed.

I wanted to get the overall vision first, then gradually dive deeper. As mentioned, I was overwhelmed with naming like:

• Folio,

• Drain Pending postings,

• Property,

• Account Receivables,

etc.

Don’t get me wrong, those were valid names in these domains. Sounds plausible if you’ve never worked in hospitality. Sounds plausible if you have, too. So what’s wrong with it?

They’re valid terms, as people actually use them, but weird, as they don’t tell “how the process works” but “how people do stuff,” and that’s not the same thing.

For instance, Property is a hotel, kind of makes sense. But Folio?

This name comes from the Oracle Opera. Yes, Oracle has a system in this domain, a dominating one. At some point, authors decided to name things this way, and now 30 years later, that vocabulary is baked into how thousands of hoteliers talk about their work. “Drain pending postings” is a phrase real cashiers say.

“Settle the folio” is a thing real cashiers do.

The problem: none of it explains how our system should work. It only tells what current systems do.

Also, when I asked about the business rules and policies, I got stuff like:

The system immediately posts room and tax charges for day-use reservations upon opening the Billing screen.

Why are transactions such as night stay charges or taxes added when we open the billing screen? That sounds counterintuitive, but maybe it was a fair tradeoff 30 or 20 years ago for an on-premises system installed in the specific hotel (ekhm “property|). Yes, Opera had (and maybe still had) consultants, similar to SAP. They’d go to the hotel, go to the back office, log in to the server and hack stored procedures in the Oracle database to fine-tune Opera behaviour.

Also, is there really a “Draining Pending Postings” option before doing checkout nowadays? Maybe it is, but in modern systems, we shouldn’t manually check all bills, etc., and explicitly pull them from integrated payment solutions. Nowadays, all the accommodation charges should already be recorded on the bill. The financial module continuously collects charges from payment gateways throughout the stay. There’s no separate moment of “draining”. The LLM invented a coordination command to match a phrase (“drain the interfaces”) that real cashiers use as shorthand for “let me check that nothing’s outstanding.”

As I asked, LLMs researched systems in this space: OPERA, Mews, Apaleo, and Cloudbeds. Each has its own vocabulary and mechanics, and the training data heavily favours OPERA because its documentation is everywhere. When I pushed back on terminology, the model would just swap in different OPERA jargon instead of actually thinking about what’s modern. The blending happened invisibly, mixing vocabulary from one system with mechanics from another, all delivered with equal confidence.

To be fair, the same happens quite often when talking to domain experts. They explain how it works now and what people do. Quite often, they bring us solutions instead of problems. Usually, solutions are based on their experience and how they see the updated version. This can be fine as a brain dump, but it’s not enough to translate it directly into the software design.

In Domain-Driven Design, finding and understanding the ubiquitous language is considered the most important aspect. Yet, Ubiquitous Language is not the source of truth. It’s the way to keep our heads from exploding due to the constant split-brain situation. It’s a tool to reduce cognitive load and the need for additional translation. That’s why we separate domain contexts and bind them to specific departments, people, and the language they use.

It’s fine to start with the current state of the art. Understanding how people do their job. We need to understand, though, that what we get is a mixture of habits (both good and bad), tribal knowledge, jargon, etc. If you ask different tribes, each will tell you something different.

Domain language is a cognitive tool, not gospel. LLMs compound this problem by reiterating competitor vocabulary without understanding the reasoning behind those systems, and they tend to align with whatever the prompter already believes.

Domain experts also struggle to define what they want and how it should work. That’s also why they hire us. It’s our job to help them and to transfer those sometimes contradicting visions into working software. That’s what we’re learning and what we do when modelling and step-by-step shaping the working software. That’s our work as engineers. We should work together to have a proper outcome. Collaborate.

And I’m not making the bold statement here that we’re smarter or we know better. We’re not; we have different expertise and different roles in the software development process.

Contrary to what many people believe in the DDD community, in my opinion, we’re not here to become domain experts; we’re here to build software.

The model we built doesn’t have to reflect the whole universe; it’s a way to take part of the business, understand, and automate it in the form of a software system. A software system is a tool for the business make more money. So software needs to be useful in a certain, defined way. Not all possible ways.

We need to learn how to communicate with business people, for instance:

• Don’t use jargon or acronyms or assume someone should know something.

• Understand that what someone wants to tell us might not be what they hear.

• Do not take others’ behaviour personally.

• Be assertive, critical and sceptical. Also, to our own judgments.

• Be curious about the business domain. Don’t assume too much.

• Don’t use “business won’t let me” as an easy excuse.

For some people, that’s too much. That’s probably why they try to ask LLM instead of Domain Experts, hoping that we won’t need to learn that, and we’ll get a solution for free.

We’re sometimes annoyed by being pushed hard by business people, I get that, but if we want to build something useful, that’s also where LLM will fail, as they will just agree eventually to what we believe. And that’ll probably be even worse than the skewed reality shown by the domain expert, since this will be Artificial Reality.

I keep hearing blank statements that “LLMs are great for research”. Blank because they usually are not followed up with what the author means by “great”. LLMs have many inherent limitations here. We’re anthropomorphising them too much. They don’t reason; they’re statistical machines. We should constantly remind ourselves of that.

I think people who claim that “LLMs are great at research” are just conflating their own skills and projecting them onto LLMs.

Hence, in my opinion, that’s why we’re getting those hot takes. Might be that they just have the skills to drill down, organise research, evaluate it, and model system design.

And I think this narrative is dangerous.

Because I, too, could sit down and say that I iteratively arrived at a solution thanks to LLMs. Write this article in a much different narrative, praising LLMs. I could say that if someone couldn’t get the same result, then that’s a skill issue.

But that wouldn’t be true, because someone without my experience in the domain and in modelling probably wouldn’t pick up on it.

Furthermore, it’s not necessarily true that I did it well; that would just be my perspective.

In practice, in this context, “LLM does it well” means “LLM does it the way I wanted it to.” And if the outcome is right, it is more likely that the “operator” had the necessary skill and used an LLM as a tool to speed it up.

The research I did looked at what our competition does and how they name things, but without internal knowledge of why those systems got to where they are. If our goal is to provide additional value to users, then just doing a blind copy won’t take us far. Instead of doing Lift and Shift, building something that has the same features but does better, we’ll get Lift and Shift with silent f.

Yes, LLMS can gather and compile multiple sources into a summary. That’s actually impressive and can spare us a lot of time. They’re also taught in the public documents, so some of their knowledge is built in. They look impressive on the surface, but not so if you look deeper. In the mentioned research, I got disconnected pieces of information, the techniques were randomly assembled with muddled technical and business language, and the whole thing didn’t tell a coherent story. Because of the randomness, you never know where the knowledge came from, what was omitted and what was skewed. With domain experts, you can at least understand the origin of their biases.

LLM also has limitations. The biggest is the context size, and being a yes-man. If we ask for the big picture, usually, we’ll end up with a swamp of information instead. If we don’t know how to sort this knowledge, what we’re looking for, and do a proper drill-down, then we won’t be able to untangle it. If we don’t have domain experts and don’t know the domain, how will we challenge issues like the ones I gave above?

I also keep hearing that LLMs are really useful in modelling. Sure, I tried that. I also asked LLM to try to model, and well, even after numerous iterations and using modelling tools, the results were mediocre. Mostly cliches and bad modelling practices.

The output muddied what we had before. The context was lost, and the process was oversimplified. When asked to apply specific tools like the C4 model, EventStorming, and Context Maps, it was forcing DDD patterns instead of describing actual processes. Even with precise instructions, it was building models with broken notation where rules floated outside the command-event chains, leaning on clichéd solutions, losing context between iterations. The domain was also presented as a big ball of mud. Concepts from room reservation bleed into the cashiering module. It was a recurring pattern across different versions of the same mistakes.

If you give too much context, they’ll mix and blend multiple conflicting vocabularies from different operational tribes trying to satisfy you. If you give it too little, it will come with simplistic hallucinations.

Is it hopeless then? Not at all, we can get help from LLM, but when we use it as a tool, not a replacement. We should not outsource thinking.

LLMs are useful for grasping the big picture and identifying known unknowns that we may miss as we get into the domain. We can learn about the language specifics, as mentioned in the Folio, Postings, Account Receivables, etc. LLMs can help us drill down into interesting aspects. We can get a brief understanding of domains we don’t know at all. But then we need to dive deeper and collaborate with our domain experts, real stakeholders. We need to understand what we want to build, organise our findings and focus on a certain context.

If we want to model our system, then LLMs can help us do boring work. It can organise our findings and create rapid text-based transitions with a defined format (including modelling practices). This can work. Yet it’s still our job to do domain discovery, refine the knowledge, evaluate and test it. We still need to focus on the feedback loop with real world.

We won’t hide from gathering modelling skills, we won’t skip the process of translating the domain into technical design, and we’ll still need to drive how and when to drill down. Those are engineering skills that were needed and will still be needed. And that’s fine, as if they wouldn’t, why would someone want to hire us?

I see that (for unknown reasons) collaboration is not discussed anymore. Working collaboratively with fellow humans has become something people dread rather than embrace. Maybe some people hope that they won’t need to talk to “domain experts” and “fight them”, because they “have everything here in LLM”.

And yes, they have all.

All the same issues they would have with domain experts.

As much as we shouldn’t trust domain experts blindly, but work with them, we shouldn’t blindly trust LLMs. We shouldn’t drop our engineering and design skills and outsource them.

If we want our software products to be better than the competition, simply blending their experience isn’t enough. We should focus on finding our special sauce, and I don’t see any other way to make it right than to work together and collaborate. We can use LLMs to help us do it faster, but as tools, not as solutions or replacements.

Check also:

• Bring me problems, not solutions!,

• A few words on communication,

• How to design software architecture pragmatically,

• Business Won’t Let Me and other lies we tell to ourselves

• Vibing, Harness and OODA loop,

• Interactive Rubber Ducking with GenAI,

• The End of Coding? Wrong Question,

• Requiem for a 10x Engineer Dream,

• Tech Debt doesn’t exist, but trade-offs do

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

Borys had the best dribbling, Cybor bent it like Beckham, there were a few little kids who could pull off 200 keepie-uppies. My class 5B always gave 5A a hiding after a tough match. For a while now, we’d only been playing by the new rules, no back-passing to the keeper. I didn’t do hundreds of tricks, I’d cap my keepie-upiess at 20–30. It didn’t entertain me. I didn’t need excessive ornaments. I thought that, at the end of the day, fundamentals are what matter most. Kicking the ball where you want it to go. What’s the difference if it slams into the top corner, curves in like a Brazilian wonder-goal, or barely rolls over the line? What matters is that it ends up in the net. That’s what counts in the end. Art for art’s sake? Not for me.

In FIFA, I never picked Brazil, not even in ‘98 did I play as France. That would’ve been too easy. Better to go against the grain. What’s the challenge in playing a guitar that’s already in tune? Better to lose to better players than to win against worse ones. Better to be a benchwarmer at Manchester United than the captain at Miedź Legnica.

I always read the instructions. You’ve got to prepare. You’ve got to assemble it the right way.

I don’t buy the first thing I see; I check price-comparison sites, read forums, and set up auction snipers on auction platforms.

Yesterday, after two years, I finally fixed the broken blinds.

• “What’s your biggest weakness?”

• “I’m a perfectionist.”

Sometimes I call myself the master of unfinished ideas. Like most of us, I dream of having my own idea, my own product, my own startup. I regularly throw myself at the next brilliant (in intent) idea, and I regularly fail to finish it. Every attempt ends the same way - “the same place, different girlfriend.”

Of course, every project of mine has to have solid foundations. Architecture first, then the framework. Everything has to be top-notch, the latest tech, best practices, and design patterns. I’ve finally got a free hand, after all, no nagging customer to worry about. Honestly, our industry is often a lot like Polish State Railways: everything would run brilliantly if it weren’t for the customers. We constantly laugh at customers: what do they know, idiots, don’t teach me how to do my job. Who cares about colours, a button shifted three pixels to the left, the wrong shade? What matters is that it’s SOLID. The most important thing is good architecture and beautifully functioning internals. Our projects are like icebergs, massive underneath, but somehow not much visible to show for it.

Last week, a book landed in my hands, well, an ebook: “Just Fucking Ship” by Amy Hoy. It hit my mood of brooding over what I’m doing wrong perfectly. Why is Borys the schoolyard footballer everyone remembers, and not me? Why does my dad say I’m all gas, no follow-through, that I burn hot and burn out fast? Why are Xamarin and SignalR thriving, while my unfinished framework with similar ambitions sits at the back of my hard drive, untouched for years? You’d think I’m doing everything right. Reason. Solid foundations. Knowledge. Experience. Mr. Prim and Proper.

Maybe it’s exactly because I make the same mistakes everyone (well, almost everyone) makes:

• I focus on chasing the rabbit, not catching it. I start a project from the framework, a website from the layout, and before any of that, from picking the domain name. So I burn through my peak motivation on bullshit that’s least important from a product perspective.

• I don’t take notes, I don’t write down plans, “why would I? I don’t need to, I’m exercising my memory.”

• I don’t focus on the goal, I don’t set deadlines.

• I don’t follow the “start small, grow big” principle.

Amy’s book isn’t groundbreaking; it doesn’t hand you a golden recipe — it doesn’t even aspire to. What it does give you is a breath of fresh air in your head, a few mental hatches popping open, and a handful of valuable, concrete, real-life advice. The most important: set yourself a goal, a deadline, and just fucking do it. First, the goal and the time, then the scope and the methods. Of course, we’ll all say, “yeah, but it doesn’t always work that way”. But look at it differently: when we want to invite friends over for dinner, do we plan, “well, maybe in two weeks instead, because I won’t have time to make a three-course meal, a cake, and homemade liqueur”? We don’t do that. We adjust the menu or buy something pre-made. Why can’t we apply the same approach to our projects?

Amy notices that we focus too much on the project as a whole. We don’t set ourselves small goals — the kind that, when achieved, would give us satisfaction and motivation to keep going. We get so fixated on the grand vision that we end up abandoning the idea without doing anything at all. We’re afraid it isn’t cool enough, it isn’t innovative enough. Why don’t we let other people judge that? We stay frozen at the starting line instead of just running and seeing how far we get. We forget that to get to the upper floor, you take the stairs one step at a time. Standing on the ground floor, wondering if we can make it up, gets us nowhere. We can even take the elevator, as long as we just fucking do it.

After this read, I’m not going to turn my life upside down, I’m not suddenly going to become a different person. I’m just going to try, this time, to finally fucking deliver on my plans.

A week ago, at almost 30 years old, I picked up my first football medal. There’s still hope.

Cheers!

Oskar

Hey, have a look at what I made during the weekend. I had some time, grabbed a beer, turned on the computer and tried to code this feature. If I could do so much during the weekend, how much could you and your team do with it in 2 weeks?

It’s almost a 1:1 quote of what I heard from the startup founder I worked with over 10 years ago. I’m sure that you’ve heard similar phrases from people you worked with. We all know the annoying type of person who doesn’t code anymore but thinks, “I still got it!”. Then they threw a piece of stuff at you to “just fine-tune it a bit and do final touches”. Then they’re the first ones to ask “Why so long?“.

Nowadays, the Internet is full of such people. They shout about what they did with Claude or how much progress LLM tools have made. Some even predict the end of coding. I already wrote that this is wrong perspective. I won’t repeat that, but I want to say that…

Vibing isn’t new and isn’t always an issue.

I’m saying that LLM tools are an appraisal for ignorance. The more ignorant we are of the topic we’re working with, the better we see the outcomes. And that, by itself, is not always bad, as there’s power in ignorance if we focus on getting it done with the simplest tools we have.

Still, this can be terrible if we fall in love too much with what we’ve vibed.

To understand why that “weekend beer” energy is both a superpower and a liability, we need to look at the OODA Loop.

Disclaimer, it’s not a competition for Ralph Wiggum Loop. It’s much older and generic.

Military strategist John Boyd developed the OODA loop (Observe, Orient, Decide, Act) for fighter pilots. In a dogfight, the pilot who cycles through these four stages the fastest and most accurately survives.

In software, the “dogfight” is the gap between your intent and the production-ready feature.

OODA loop is built from four steps:

1. Observe - This is the intake of raw, unfiltered information. In our world, this means looking at the state of the system.

2. Orient - This is the most critical and difficult stage. It’s where you filter your observations through your experience, culture, and technical knowledge.

3. Decide - Based on your orientation, you formulate a hypothesis.

4. Act - You execute.

Getting back to my favourite founder and LLM-based tools.

The reason founder could build a PoC in a weekend while the team needed more than two weeks is that he bypassed the Observe and Orient phases. He went straight from a vague idea to Act.

If we skip or brush past the observation step, it feels like lightning speed. If the fancy UI grid is there and it does something we wanted, we move on. We’ve outsourced Orientation to our own ego. It’s too easy to assume that because we wrote it, it works.

Observation is the intake of raw data. In a professional environment, our eyes aren’t enough. We need a Harness. If we don’t have automations, tests, integration tests, and pristine traces, we aren’t observing the system; we’re just looking at it. If the inputs are messy, our observation is clouded.

But real engineering, the kind that takes those “two weeks”, is about closing the loop properly. That’s also where we need different perspectives and knowledge sharing.

Orientation is where you process those observations. This is the part where LLMs make us feel smarter than we are. If we don’t understand how a database handles concurrent connections, our “orientation” of a generated script will be shallow. We’ll see code that “looks” right, decide it’s fine, and act by deploying it.

The “I still got it” crowd loves the Decide and Act phases because that’s where the visible progress happens. LLM tools have made these phases nearly instantaneous. We can decide to build a feature and have the code for it in ten seconds.

The problem is that the faster we Act, the faster we need to Observe. If our “Act” phase takes seconds but our “Observe” phase requires a manual weekend of clicking around and drinking beer, our OODA loop is broken. We’re just generating a pile of stuff that we haven’t actually verified.

That’s why the team usually needs more than an imaginary “two weeks”. They are not “fine-tuning” the single-brilliant-dude masterpiece. They are building the infrastructure required to make the OODA loop sustainable.

And to make that possible, they need to run the full loop: Observe, Orient, Decide, Act. And do it multiple times. That takes time, but it’s required to assess the direction, automate what needs to be automated, and ensure they can iterate further and run this loop sustainably. That’s critical for delivering the outcome at the expected pace.

Of course, there’s a danger here, overfocusing on the Orient and Decide can lead to overengineering, building stuff we don’t need. That’s where ignorance can be blissful, especially when we connect it with humility. Being humble about what we don’t know and trying things the easiest way, then learning and making enhancements. Still, humility fails under deadline pressure. The harness doesn’t.

Let me give you…

The example

I’m adding proper Observability and Open Telemetry to Emmett right now. I spent some time working on it and instrumented the first component: Command Handling.

Of course, I had tests to prove it works, but I don’t trust them enough, and I wanted to try it on a real sample, since you never know until you run it. Even the best test suite won’t tell you all.

So I decided to plug it into the sample. See if it works, how ergonomic the API is and how it fits conventions in this area.

To do it, I decided to use Grafana stack and set it up with Docker Compose. So, stable, boring stack. Not going to lie, I vibed the config. Not that there are no docs, but I intentionally wanted to see the typical config people use.

If someone says LLM-based tools are great at proof of concepts, they don’t run the stuff they vibed. If I made the observation based on the initial config, then an oriented decision would be that it won’t work. Of course, then I did the typical back-and-forth, with the LLM tool doing some Linux command Voodoo to make it work. Once. Then, if you try to repeat it, you won’t know how to do it without doing Voodoo again.

Again, that’s not much different from the other stuff we do. I’m sure that you had multiple cases, when someone didn’t use Continuous Deployment tools, but clicked through Azure, AWS, GCP portal, deployed the stack, and then there was no trace on how to set it up again (e.g. to have a different environment for testing or demos for customers).

So, we need a harness, we need a leash to keep our process on track.

How to do the harness? My advice is to start simple. We may ask LLMs to give us shell scripts, and we may ask them to run them multiple times. We also need experience and knowledge of what we want to achieve and the tools we use. It’s fine not to remember all the YAML config to set up the Grafana stack, but it’s not fine not to understand why you even use it, how it relates, and how to set it up.

Still, our first loop can close on the first working solution, even a manually vibed one. But that’s not even a PoC. We need to automate them.

I asked LLM to take notes on what issues it had, and it solved them. Then, based on that, I asked to research how to code it in TypeScript. And to use tools I know, used in past, validating if there are no new more modern ones. For instance, I was a big fan of Gulp.js and Bullseye in the past, but they’re mostly dead. I wanted to have something in the same spirit, using native, maintained tooling.

I ended up with the following tools:

• execa for running shell scripts,

• native fetch for calling http endpoints,

• native Node.js test tools for checking if the stack works as expected.

Then I asked it to create the script to automate the shell Voodoo they did to make Grafana stack and Docker Compose work.

Essentially, it should:

1. Run Docker Compose script starting up services (Grafana, Prometheus, Loki, Tempo, PostgreSQL, etc.).

2. Wait for them to check when they’re ready (it usually takes some time).

3. Start the application and make a request.

4. Check if the predefined dashboard with Emmett metrics appears, and shows expected traces and metrics.

Initial diagnostic tools looked like that

async function fetchWithDiag(label: string, url: string, init?: RequestInit) {
  const res = await fetch(url, init);
  if (!res.ok) {
    const body = await res.text().catch(() => '(could not read body)');
    console.error(`\n  ✗ ${label} → HTTP ${res.status}\n  body: ${body}\n`);
  }
  return res;
}

async function diagnoseCollector() {
  const text = await fetch(URLS.otelCollectorMetrics)
    .then((r) => r.text())
    .catch(() => 'unreachable');
  const emmett = text
    .split('\n')
    .filter((l) => l.startsWith('emmett_') && !l.startsWith('#'))
    .slice(0, 5);
  console.log(
    emmett.length
      ? `\n  collector /metrics (emmett lines):\n  ${emmett.join('\n  ')}`
      : '\n  collector /metrics: no emmett_* lines found',
  );
}

async function diagnosePrometheus() {
  const json = await fetch(
    `${URLS.prometheus}/api/v1/label/__name__/values`,
  )
    .then((r) => r.json() as Promise<{ data: string[] }>)
    .catch(() => ({ data: [] as string[] }));
  const emmett = json.data.filter((n) => n.startsWith('emmett_'));
  console.log(
    emmett.length
      ? `\n  Prometheus emmett_* metrics: ${emmett.join(', ')}`
      : '\n  Prometheus: no emmett_* metrics found yet',
  );
}

async function diagnoseLoki() {
  const labels = await fetch(`${URLS.loki}/loki/api/v1/labels`)
    .then((r) => r.json() as Promise<{ data?: string[] }>)
    .catch(() => ({ data: [] as string[] }));
  console.log(`\n  Loki labels: ${(labels.data ?? []).join(', ') || '(none)'}`);
}

async function diagnoseDockerLogs(service: string, lines = 10) {
  const { stdout } = await execa('docker', [
    ...COMPOSE,
    'logs',
    '--tail',
    String(lines),
    service,
  ]).catch(() => ({ stdout: '(could not get logs)' }));
  console.log(`\n  docker logs ${service} (last ${lines}):\n  ${stdout.split('\n').join('\n  ')}`);
}

Are they pretty? No. Can they be improved? Yes. Do they have to be improved at this specific moment? No.

The setup uses test infrastructure

const CLEANUP = process.env['CLEANUP'] === '1' || process.env['CLEANUP'] === 'true';
const CLEANUP_AFTER = process.env['CLEANUP_AFTER'] === '1' || process.env['CLEANUP_AFTER'] === 'true';
const NO_START = process.env['NO_START'] === '1' || process.env['NO_START'] === 'true';

// ─── configuration ───────────────────────────────────────────────────────────

const COMPOSE = ['compose', '-f', 'docker-compose.yml', '--profile', 'observability'];

const URLS = {
  app: 'http://localhost:3000',
  prometheus: 'http://localhost:9090',
  tempo: 'http://localhost:3200',
  loki: 'http://localhost:3100',
  grafana: 'http://localhost:3001',
  otelCollectorMetrics: 'http://localhost:8889/metrics',
};

// Fresh client per run — avoids stale cart state from previous runs.
const SERVICE_NAME = 'expressjs-with-postgresql';
const CLIENT_ID = randomUUID();
const CART_ENDPOINT = `${URLS.app}/clients/${CLIENT_ID}/shopping-carts/current/product-items`;
const CONFIRM_ENDPOINT = `${URLS.app}/clients/${CLIENT_ID}/shopping-carts/current/confirm`;

// Matches the .http file — unitPrice is resolved server-side.
const ADD_PRODUCT_BODY = JSON.stringify({ productId: randomUUID(), quantity: 10 });


before(async () => {
  console.log(`\n▶ client ID for this run: ${CLIENT_ID}\n`);

  if (NO_START) {
    console.log('▶ --no-start: skipping docker compose and app startup');
    return;
  }

  if (CLEANUP) {
    console.log('▶ --cleanup: killing port 3000 and tearing down stack (down -v)…');
    await execa('bash', ['-c', 'fuser -k 3000/tcp 2>/dev/null || true']).catch(() => {});
    await new Promise((r) => setTimeout(r, 500));
    await execa('docker', [...COMPOSE, 'down', '-v', '--remove-orphans'], {
      stdio: 'inherit',
    });
  }

  const stackReady = await fetch(`${URLS.prometheus}/-/ready`)
    .then((r) => r.ok)
    .catch(() => false);

  if (stackReady) {
    console.log('▶ observability stack already up — skipping docker compose up');
  } else {
    console.log('▶ starting observability stack…');
    await execa('docker', [...COMPOSE, 'up', '-d'], { stdio: 'inherit' });
  }

  console.log('▶ waiting for backends…');
  await waitFor(() => checkUrl('Prometheus', `${URLS.prometheus}/-/ready`), {
    timeout: 90_000, label: 'Prometheus',
  });
  await waitFor(() => checkUrl('Grafana', `${URLS.grafana}/api/health`), {
    timeout: 90_000, label: 'Grafana',
  });
  await waitFor(() => checkUrl('Tempo', `${URLS.tempo}/ready`), {
    timeout: 90_000, label: 'Tempo',
  });
  await waitFor(() => checkUrl('Loki', `${URLS.loki}/ready`), {
    timeout: 90_000, label: 'Loki',
  });

  // /health returns { status: 'ok', service: 'expressjs-with-postgresql' } —
  // checking service name lets us distinguish our app from other processes on :3000.
  const checkOurApp = () =>
    checkUrl('app /health', `${URLS.app}/health`, async (res) => {
      const json = (await res.json().catch(() => ({}))) as { service?: string };
      if (json.service !== SERVICE_NAME) {
        console.log(
          `    app /health: service="${json.service ?? '(missing)'}", expected="${SERVICE_NAME}"`,
        );
        return false;
      }
      return true;
    });

  const appIsOurs = stackReady && (await checkOurApp());

  if (appIsOurs) {
    console.log('▶ app already running and healthy — skipping npm start');
  } else {
    const portTaken = await fetch(URLS.app).then(() => true).catch(() => false);
    if (portTaken) {
      // Port is occupied but not by our app — stale process or unrelated service.
      console.error(
        '\n  ✗ Port 3000 is occupied by a process that is not this app.\n' +
          '  It may be a stale version of this app (connected to a wiped database)\n' +
          '  or a completely different service.\n' +
          '  Fix: run  npm run verify:observability:cleanup  to kill it and restart,\n' +
          '  or manually free port 3000.\n',
      );
      process.exit(1);
    }

    console.log('▶ starting app…');
    app = execa('npm', ['start'], { stdio: 'inherit' });

    await waitFor(checkOurApp, { timeout: 60_000, label: 'app /health' });
  }

  console.log('▶ setup complete\n');
});

As you see, nothing fancy, the cleanup is even simpler

after(async () => {
  if (app) {
    console.log('\n▶ stopping app…');
    app.kill('SIGTERM');
    await app.catch(() => {});
    console.log('▶ app stopped');
  }

  if (CLEANUP_AFTER) {
    console.log('▶ tearing down stack (down -v)…');
    await execa('docker', [...COMPOSE, 'down', '-v', '--remove-orphans'], {
      stdio: 'inherit',
    });
    console.log('▶ stack torn down');
  } else {
    console.log('▶ stack is still running');
    console.log('▶ to clean up: npm run verify:observability:cleanup');
  }
});

Having that we can run tests:

test('successful command returns x-trace-id header', async () => {
  const res = await fetchWithDiag('POST add product', CART_ENDPOINT, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: ADD_PRODUCT_BODY,
  });

  assert.equal(res.status, 204, `Expected 204 — body logged above`);

  const header = res.headers.get('x-trace-id');
  if (!header) {
    console.error(
      '  ✗ x-trace-id missing — verify the wrapper app in src/index.ts ' +
        'adds it via @opentelemetry/api before mounting the emmett app',
    );
  }
  assert.ok(header, 'x-trace-id header missing');
  assert.match(header, /^[0-9a-f]{32}$/, `"${header}" is not a 32-hex trace ID`);

  traceId = header;
  console.log(`  trace ID: ${traceId}`);
});

test('OTel collector exposes Emmett metrics on port 8889', async () => {
  // Send a few more requests so metrics are definitely recorded.
  for (let i = 0; i < 5; i++) {
    await fetch(CART_ENDPOINT, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: ADD_PRODUCT_BODY,
    });
  }

  try {
    await waitFor(
      async () => {
        let text: string;
        try {
          const res = await fetch(URLS.otelCollectorMetrics);
          text = await res.text();
        } catch {
          console.log('    collector :8889: connection refused');
          return false;
        }
        const emmettLines = text.split('\n').filter((l) => l.startsWith('emmett_') && !l.startsWith('#'));
        if (emmettLines.length === 0) {
          const allFamilies = [...new Set(text.split('\n').filter((l) => !l.startsWith('#') && l).map((l) => l.split('{')[0]))].slice(0, 5);
          console.log(`    collector :8889: no emmett_* metrics yet. Present: ${allFamilies.join(', ') || '(none)'}`);
          return false;
        }
        return true;
      },
      { timeout: 90_000, interval: 5_000, label: 'emmett metrics on collector :8889' },
    );
  } catch (err) {
    await diagnoseCollector();
    await diagnoseDockerLogs('otel-collector');
    throw err;
  }
});

I put it into a single file that can be run as a regular Node.js script.

It already showed me (and Claude) that what they initially did wasn’t working if you try to run it multiple times. It also showed that doing a full cleanup and rebuild, and making it reproducible, needs more work.

Is it done? Not yet; it takes too much time and resources to run it continuously throughout the pipeline. The code is a bit messy, so it needs to be organised. It’s segmented into blocks, includes basic automation and tests, and has already gone through some failures to get it done.

Could I do it better? Sure, and I will improve it, but that’s not the point. I wanted to show you my findings during weekend vibing (without beer tho), the real, not polished iteration, before I run the next one.

The main idea behind OODA loops is not to be perfect, but to iterate quickly, gather feedback as soon as possible, learn from it, develop another theory, and verify it through action.

It’s not about vibing, but it’s also not about analysis paralysis.

I hope you’re now better equipped to think about when vibing, with beer or without, with LLMs or without, actually helps, and when it doesn’t.

Vibe coding is just high-frequency steering. It only works if you have a Harness: a mechanical way to observe and orient, so you don’t steer the whole project into a wall.

Act takes seconds now. Observe takes as long as it always did. Without a harness, you’re not going faster; you’re just making more stuff you haven’t checked.

Harness is not magic, a new discipline, or the next buzzword; I hope I showed you that a bit in this article on what it may look like.

So iterate fast, but wisely remembering to do the full loop. It’s great that LLMs can help us make Acting faster, but we should not skip other steps. We should aim for a fast feedback loop to iterate in the right direction and achieve continuous improvement, to deliver proper value.

Just like Vibing isn’t new, we shouldn’t abandon “old” engineering practices. We should also not replace collaboration with solitary self-high fives.

Check also:

• Emmett Pull Request with mentioned changes

• Interactive Rubber Ducking with GenAI

• The End of Coding? Wrong Question

• A few tricks on how to set up related Docker images with docker-compose

• Docker Compose Profiles, one the most useful and underrated features

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

Try not. Do. Or do not. There is no try!

I’m calling this the Yoda Principle.

Master Yoda said that to Luke Skywalker a long time ago in a galaxy far, far away. He was teaching Luke how to name commands properly while trying to untangle some legacy enterprise mess.

I’m sure you’ve also seen a death star of weirdly-named stuff. Some of them have already tripped you hours of thinking whether someone named this thing badly, or there’s some hidden truth behind it.

Let’s discuss that by the example: E-Commerce order fulfilment.

The order is placed automatically once the customer confirms the items in the shopping cart. We’re not making any product reservations before the shopping cart is confirmed, as this would lock it for other customers, and, as you know, they tend to drop items from their carts.

Once we receive the event notification that the cart has been confirmed, we’ll start the order fulfilment process. It starts (as mentioned) with the order initiation, which acknowledges and initiates the multi-step fulfilment process.

The first step is checking product availability before confirming the order. We need to determine whether we can proceed with completing the shipment and initiating product payment. If the product is unavailable, we need to either ask the customer to wait until we have it again or cancel the order.

We have dedicated modules for order fulfilment and for inventory. Fulfilment is the orchestrator, and inventory is responsible for tracking the state in warehouses.

The ordering module would need to call the inventory module to verify product availability. We could send a command (through the messaging system or web api). How would we name it?

What about VerifyProductExists? We’d send the product id and quantity from the order information, and return true if we have enough products, false otherwise. Sounds fair, right?

Well, it may seem nice at first glance, but what happens if more than one order verifies the same product availability, and we’re running short?

Then we’re vulnerable to race conditions I described in Tell, don’t ask! Or, how to keep an eye on boiling milk. The information we get is only valid at the time of querying. If we don’t lock the product quantity, it can change before we get a response (think: Black Friday-like demand).

Naming our command like VerifyProductExists is a mistake.

VerifyProductExists is not even a command; it’s a query. Command is a request (intention) to run business logic. Query is a request to return data.

Of course, pragmatically, our command can return status information about the result of our operation. But the intention is different.

What’s our real intention here?

The real intention is that we’d like to reserve products so we can ship them and get payment for them, not to verify if they exist. It’d be better to name our command as ReserveProducts or LockProducts.

Why does it matter?

If we’re naming our commands with Verify/Validate/Check prefixes, we’re putting ourselves into the wrong mindset. We’re not focused on actions and integrations, but just brief checking. If we’re in such a mode, it’s easy to handwave the integration complexity.

Locking for shipment may require sending someone to double-check that the product is in the warehouse and hasn’t been stolen, damaged, or other steps. It may be an async process on its own. Still from the ordering module, we should not care, as we’re telling what our intention is and expect to get events informing us whether the reservation succeeded, failed, or timed out (we don’t want to lock products forever in case of order fulfilment issues, but only for some time).

Prefixes like Verify/Validate/Check, etc., are just synonyms for trying. And well, commands are always a form of trying. The handling module can reject the command, as its business rules and state are the source of truth.

We should always assume that the command processing can fail. We should not be discouraged by that, and we should double-check everything. We should not be intimidated by the potential failure, but prepared for it.

We should try not. Do or do not. There is no try.

What if we have both? So VerifyProductExists and LockProducts? It can work if the first one is a query used by the Shopping Cart module, without any guarantee that the data isn’t stale, on a best-effort basis.

If we’re always requiring VerifyProductExists from the handling module before LockProducts, we’re making our communication chatty. I described that in What does Mr Bean opening the car have to do with programming? that this is not only a bad developer experience, but also just redundancy. Locking should already verify whether the product exists, so why require someone to memorise those scenarios instead of checking it internally?

The same goes for cases like:

• verify payment has been made,

• check if the order wasn’t already fulfilled,

• validate if the shipment has been completed,

• etc.

All of them either hide a missing business concept or should be a business rule verified within the specific action (e.g., confirming an order).

I recognise this may seem nitpicky, but big things are built on small details.

If we don’t think about such things, we’ll not only end up with misnamed integrations but also fight race conditions and incorrect boundaries.

Then we’ll be doomed.

So better think twice and do or do not.

May the force be with you!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

Have you ever heard phrases like.

Just an update, the milk ran out. Someone finished it and put the empty carton back.

Or

So everyone is aware, the meeting started 15 minutes ago.

Or

Heads up: the coffee machine is empty again.

I’m sure you either heard or used such phrases.

We all know that there’s some hidden intention behind it.

The intention is not to inform, but to trigger a certain action.

Formally, we’re reporting on events to announce the facts, but in practice, we’re using passive-aggressive words. The real intention is to command someone.

We don’t want to inform that the trash bin is full, but we want someone to take it out. We don’t want to inform that the coffee machine requires coffee beans refill, but we want someone to do it.

Passive-aggressive tone is the worst. It’s toxic for both sides of the communication. Usually, it’s just better to ask someone to do it.

The same rule applies in event-driven modelling. We should avoid passive-aggressive communication at all costs.

We should watch out for Passive-Agressive Events. So events that should be commands.

I already warned you in past not to let Event-Driven Architecture buzzwords fool us.

Event-Driven Architecture is an integration architecture style. We’re trying to model our business processes to run smoothly. To achieve that, we prefer a non-blocking communication flow, with things happening in parallel at their own pace. The goal is to achieve autonomous components, reducing the time needed to understand them. That helps maintain, or even replace them as your business evolves.

And events are enablers for that. They notify of what has happened, allowing other components to interpret facts and take the next steps.

But… Let me show one more photo.

It’s parliament, per the official definition: a room full of angry, shouting people.

If we model our communication only in terms of events, our system will look just like that. We’d just announce new facts in a passive-aggressive style and not be interested in what happens next. Oh, wait, are we really not interested? Actually, we are. If someone won’t do what we expect with our information, we’ll be even angrier.

What’s the difference between a command and an event? Both are messages. They convey specific information: a command indicating intent to do something, an event describes what has happened. From the computer’s point of view, they are no different. Only the business logic and the interpretation of the message can distinguish between an event and a command.

And that’s the main difference: commands can be rejected by the command handler. Events can only be ignored.

If we publish an event, we expect one or more consumers to be interested in it. Yet, we don’t know which components will do it. We just broadcast information.

This can easily change into passive-aggressive:

I did my work, now it’s your turn.

And here’s the crucial part. If we always have a single consumer for an event that needs to run the specific logic and expect to get the particular event back, then it should be a command. It’s not an event, we don’t inform. We want some component to take the next specific step and let us know when it’s finished.

Aren’t we making our communication synchronous?

What does it even mean, synchronous or asynchronous?

That’s what Sam Newman discussed in his great talk. The main conclusion is that synchronous vs asynchronous discussion is actually about blocking or non-blocking processing. And that’s much broader topic than the technical solution (so whether we call something in-process via an HTTP endpoint or a messaging system).

It’s a common misconception that events are published asynchronously through a messaging system (e.g. Kafka, RabbitMQ, SQS, WhateverQueue) and commands are sent through synchronous WebAPI. That can be true for a specific solution, but not as the general rule. As said, both events and commands are messages; we can send them through a messaging system or via HTTP (e.g. events via webhooks).

This misleading split came out from our expectation about handling. We expected the command handler to give us the result. For event handler, we don’t expect a specific result. At least in theory.

If we publish a specific event to the messaging system and expect a specific critical path of follow-up events, then we’re not making our communication non-blocking. It’s still sequential. We cannot proceed until the expected sequence occurs.

Whether something is blocking or not is not established by the tools we use, but by how our business process looks.

Speaking about it.

Let’s get back to our favourite E-Commerce Order scenario (read more in Predictable Identifiers: Enabling True Module Autonomy in Distributed Systems).

We could model it so we just publish the OrderConfirmed event and passively-aggressively expect that others will take it from there. So:

• The payment module will initiate the payment.

• Inventory will start completing shipments.

• The notification module will send a confirmation e-mail.

• Fraud detection module will check if the order is not rigged.

Once we receive information about a successful shipment or payment registration, we can complete the order.

You may notice two paths for order processing:

1. Blocking - We need to wait for information about payments and shipments. This is our critical path.

2. Non-blocking- Order process shouldn’t stop if the notification wasn’t sent or the data warehouse wasn’t able to process events. We’d like that to happen, but it’s expected rather than critical.

Now, both payments may fail (if our customer doesn’t have enough money), and the shipment may not be completed (if it’s Black Friday, and multiple people are competing for the same product).

If that happens, ordering module needs to take action, for instance, do reimbursement if the shipment wasn’t completed, and eventually cancel the order.

If we don’t foresee that and stay in passive-aggressive mode, we tend to forget about “negative” scenarios. it’s too easy to stay in I-Alread-Did-My-Job mode. This will have severe consequences: blocked orders, missed communication, and dissatisfied customers.

We may lear too late that another module can actually say no:

1. Payment module can say: Man, that’s not going to happen, you’ve already run out of money.

2. Shipment module can say: Man, I’m sorry, but you weren’t fast enough and we’ve run out of product.

And both of those scenarios will block successful order completion.

How to find such cases? Doing Example Mapping during modelling can be a good option for that.

Most importantly, we need to embrace the fact that some processes require direct, blocking communication, and others don’t. Just like in real life, sometimes it’s just more effective to tell someone to do something. We should avoid micromanagement and aim for autonomy, but not end up with anarchy.

In our case, it’d be better to have a coordinator (workflow, saga, process manager, To-Do List etc.) that publishes the OrderConfirmed event for modules not on the critical path and sends commands like RecordPayment and InitiateShipment.

By that, we’re separating responsibilities and making explicit what should be explicit. This also helps in understanding the business process, as you have a central place to see the critical flow and get proper observability.

Lacking tracing and observability of the business process is one of the most common issues I see in my clients’ projects. As said, if we don’t want to end up with parliament instead of proper communication in our system, we need to be explicit about our intention.

Is that all? Not quite, there’s one more message type we model as events that should not be events.

Gregor Hohpe, in “Enterprise Integration Patterns”, besides Event and Command defines one more message type: Document.

What’s the Document? It’s a state. Or to be precise: self-contained data we have at a certain point in time. We can store it, but we can also publish information about its new value.

That’s probably why Martin Fowler frames it as Event-Carried State Transfer, and I don’t like that term. For me, it’s extremely misleading as, it doesn’t tell what has happened, but what has changed. It just gathers the new version of the state (or the diff).

In my opinion, it’s a variation of State Obsession anti-pattern. Many people fell into that and believe it’s fine to connect the messaging system to the database, use tools like Change Data Capture, and publish it automatically to others. They end up with passive-aggressive communication style:

You have all you need. The whole state is in the events, just interpret it.

How can you reason about what has happened if instead of OrderConfirmed you get OrderCreated, OrderUpdated, OrderDeleted? You’d need to do the diff, compare with previous values, and do the guess about the reason of the specific change.

You deal with Clickbait Events and have a leaking business abstraction. All consumers need to understand the internals of your processing to detect a specific type of change. I wrote about it in detail in Internal and external events, or how to design event-driven API.

Again, the loose coupling of the event-driven processing is only loose for producers; consumers need to adapt. This can lead to hidden coupling, where a change in the producer breaks consumer flows. And that’s the worst type of coupling you can get.

If we’re making commands explicit, we’re also making an explicit relationship between components. It’s no longer flattened to producer <=> consumer, where the producer always shapes the communication. Now, if the other component exposes a command, that’s the driving force behind its behaviour. This helps to shape autonomy. In our case, we could make the Payment Module a generic module with a stable public API for registering payments, and an ordering module that requests them, in accordance with the Shipment Module. Fraud Detection could continue subscribing to events, as it already does. Context Mapping can greatly help in finding those relationships.

TLDR

We tend to be all about events these days, but they’re not the only message types. In our systems, messages take various forms: Events, Commands, and Documents, each serving distinct purposes:

• Documents are all about state transitions, which are essential for syncing data across services but missing deeper business insights.

• Commands represent a clear intent to act, directed with an expectation of execution, and can be accepted or rejected.

• Events are immutable facts, announced without waiting for a response. They’re like broadcasting news, hoping it catches the right ears.

Event-Driven Architectures enable loose coupling, but only for producers. To make consumers loosely coupled, we need to take extra steps, embrace different message types, and have them participate in modelling business processes.

If we go too far with an event-all-the-things communication style, we’ll make our system a room full of shouting people, with a passive-aggressive communication style. Or just aggressive.

In consequence, we won’t know what’s happening in our system, will see only noise, and will have a hard time making it reliable, observable and predictable. We should treat our messages as a communication contract, API and model their flow in a way that shapes our regular communication.

So next time, ask yourself if your event shouldn’t be a command. If it has always had a single consumer and you expect a specific event back, then it’s probably so. It’s all about being clear about the intention, not lying to yourself and others.

I hope this article will equip you with the knowledge to fix that.

If you’re dealing with such issues, I’m happy to help you through consulting, training or mentoring. Contact me and we’ll find a way to unblock you!

See also more in series about event modelling anti-patterns:

• State Obsession,

• Property Sourcing,

• I’ll just add one more field.

• Clickbait event,

• Passive Aggressive Events,

• Should you record multiple events from business logic?,

• Stream ids, event types prefixes and other event data you might not want to slice off.

Check also more general considerations:

• Events should be as small as possible, right?,

• What’s the difference between a command and an event?,

• Internal and external events, or how to design event-driven API,

• Event Streaming is not Event Sourcing!,

• Don’t let Event-Driven Architecture buzzwords fool you,

• How to design software architecture pragmatically,

• How to deal with privacy and GDPR in Event-Driven systems.

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

One of the first indications of getting old(er) is when people stop getting your movie or music references. Of course, based on this rule, some people are always old. That happens.

Recently, I realised during my workshops that referencing Friends is not so cool anymore. It started to happen when I was explaining the Example Mapping technique.

It always starts with “The one where”.

Just like in Friends.

I started to notice a bit slower head nodding and a bit more awkward smiles from the attendees. I repeated

You know, like in titles of Friends episodes.

And head-nodding stopped; only totally awkwardly polite smiles remained. Definitely, it wasn’t The One Where Everybody Finds Out. So I finally asked:

You weren’t watching Friends, didn’t you?

Obviously, the answer was:

Emmm. No…

Okay, then, if you don’t know Friends or the Example Mapping technique, this will be The One Where You Find Out.

Let’s say that we’re working on the guest checkout feature for a hotel management system.

We could start by asking the business how it works. We could get an answer that:

The guest approaches the desk and requests checkout. The clerk inquires about the quality of the products and services, and after receiving an answer, requests the room key. After gathering, the key clerk checks whether the balance is settled. If it’s settled, then proceed with the checkout. Marking the stay as completed.

Sounds straightforward, but we should already have several questions popping up, e.g. what does it mean that “balance is settled”? We could get quick feedback that:

This means that the difference between the sums of all charges and payments is equal to zero.

Then we could try to come up with an example:

Ah, so for instance, when guests haven’t paid upfront for their stay, right?

Right.

Oh, then we need to charge them, right?

Right.

We could visualise what we discovered in the following way:

Now, this generated another flow for us. We have a new feature we weren’t aware of: the guest’s stay payment registration. Let’s try to start this time from the visualisation.

It’s the one Oskar pays for his stay, because he wants to check out but didn’t pay upfront. The payment is registered, and we can try checking out again. Sounds fine, but we should ask whether there are any rules for payments. It may appear that:

Yes, there are some, for instance:

• Only guests with a valid credit card can pay with it for their stay,

• Guests paying in cash need to hand it over before accepting the payment.

And hey, we just found out new business rules, let’s put them on the board and update our flow to be more precise and reflect our scenario by adding a note that this scenario represents a guest paying with a credit card.

Now, what if the payment fails? Can it fail? Let’s ask the business!

Payment may fail if the payment gateway is unavailable or the issuer rejects it.

What if our internet connection is down for the whole day?

The clerk should ask the guest for cash.

What if the guest doesn’t have cash? 🤔

Then the shift manager can authorise unsettled balance checkout and register a charge with a delayed due date.

Here’s the updated flow. The one where Oskar pays for his entire stay with a credit card, but the Internet is down, and he doesn’t have cash.

Now, we found out:

• A new outcome, failed payment,

• A new rule, that we need Internet access to authorise credit card payment,

• A new feature, the shift manager can authorise unsettled balance checkout and register a charge with a delayed due date.

How would the authorisation look? How should we register a delayed charge?

We should follow double-entry bookkeeping and register the authorisation (type of payment) with the unbalanced amount to settle the balance for today, and register an additional delayed charge for the same amount

The flow will look like:

And that’s precisely how the Example Mapping session looks like. It’s a structured conversation format created by Matt Wynne. You take a user story, gather a small group (usually a developer, tester, and someone from the business side), and spend around 25-30 minutes breaking it down together.

You don’t need a big setup, a huge ceremon, you don’t need sticky notes, you can just use plain text like:

Given: Example

When: We use specific features

Then: Based on business rules, we get a specific outcome.

Business people don’t need to give them to you in such form. You can use the interview as I showed above and note it on your own, while you’re discussing stuff. It’s also a nice way to collaborate and visualise your discussions.

You don’t even need to start with an interview; you can use the Example Mapping as a brainstorming tool to generate as many examples of your (part of the) system. Then, try to model it as you see fit and ask the business for clarifications in the preferred form. It can help facilitate discussion with your team, not only with business stakeholders.

It’s super helpful, as misunderstandings are expensive. When multiple people try to describe the same rule using real examples, you’ll quickly see where assumptions don’t match. Better to find that out in a 30-minute chat than after two weeks of coding the wrong thing.

It also works as a readiness check. Too many red cards? The story isn’t ready. Too many blue cards? The story is probably too big. If examples come easily and everyone nods along - you’re good to go.

There’s also more, as you can try to distil business rules based on the examples and outcomes the business describes.

It’s worth noting that I cheated you in colours. The original looks like this:

Why did I change them?

Example Mapping plays nicely with other collaboration techniques like Event Storming. And if you’re familiar with the Event Storming colour scheme, that’s also the reason why I used it. I aligned them. See:

I’m typically using it during modelling sessions to:

• brainstorm (read more in Start Alone, Then Together: Why Software Modelling Needs Solitary Brainstorming),

• challenging existing models with real-world examples,

• expanding the model with uncovered (through examples) use cases,

• finding business rules,

• helping with facilitation by looking at the model from a different perspective.

And hot spots and notes, as known from EventStorming, are super helpful here. Read also more in The Underestimated Power of Hot Spots and Notes in EventStorming.

What’s more, if you look at the Given/When/Then pattern, you may notice that it works nicely with Behaviour-Driven Design. I already wrote that Behaviour-Driven Design is more than tests. How to do it? Check here.

I’ll also expand on it in the next articles. I’m doing the extreme Example Mapping with events, so stay tuned, the more will come.

For now, check also those materials:

• Seb Rose - short, practical and actionable intro to Example Mapping

• Kenny Baas-Schwegler - showing how to use Example Mapping with EventStorming

• An introduction by Matt Wynne himself,

• Other quick intro by Gojko Adzic.

And most importantly, try it. Take one feature from your system, try to crunch it, or start your design session with this technique.

Most of the teams I’m working with are enjoying this technique, as it’s a fun way to get quick, actionable outcomes.

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

You may already know that I’m a GenAI sceptic. And a general sceptic.

Do you know that scepticism comes from the Greek σκέπτομαι (skeptomai), meaning ‘to search, to think about, or look for’? So my intention is not to say no to everything new, but more to think about it first, and understand before I say yes.

There’s a lot of stuff about GenAI that makes me smile, but I still understand that my way is my way, and I won’t stop the world. I won’t even try. Thus, I want to research and consider how those tools can help me. I already wrote that I don’t feel like 10x Dev, but I’m finding more ways to get help from it.

One of the ways that helped me is something I call “Interactive Rubber-Ducking”.

Initially, I called it just brainstorming, but that wouldn’t be precise, as I’m not using it to brainstorm ideas, more to challenge and clarify them.

Most of the code I write nowadays is done in my OSS projects. I’m grateful to have a great community with people actively contributing in different ways; still, the canonical design and code work is on my side. As I work in an event-driven niche, I’m often alone with my own thoughts. I try to use the RFC process and discuss it with other fellow humans, but they are not always available. Even if they do, to avoid wasting their time, I need to know what to tell or ask them. I need to give some proposals (with alternatives) to have an effective discussion. I may seem organised, but that’s not always the thing. Sitting in your own head is not a great place to be in general. If you’re a technical leader or an architect, I’m sure that you know that solitude too well.

GenAI tools are not great sparing partners. They’re Yes men. If they read this article, they’d for sure confirm it. They’d probably do it even without reading it. Of course, you can ask them not to be sycophant. You can ask numerous MUSTS with capital letters and bolded NEVER here and there, and it can help, but it won’t fully beat the way they were trained.

And talking to yourself is a similar experience: you’ll find numerous ways to justify your own decisions, and looking at the same place for too long will make you miss obvious blind spots.

Ok, so why would we take those two blind “people” and try to make them help each other?

That’s kinda what “Interactive Rubber-Ducking” is. It takes a blind human with an idea, and another blind not-so-human asking questions. It starts with such a prompt:

Ask me one question at a time so we can develop a thorough, step-by-step spec for this idea. Each question should build on my previous answers, and our end goal is to have a detailed specification I can hand off to a developer. Let’s do this iteratively and dig into every relevant detail. Remember, only one question at a time.

Once we are done, save the spec as spec.md

Before asking another question, store the previous one with the answer in qa.md. Write literally the question and answer, not just a summary.

Here’s the idea:

Don’t treat it as “one magical prompt that will change your life”. Most important is why we’re doing it, what happens next, and who’s actually doing the work. Spoiler alert: it’s not LLM.

I’m using it as a command in Claude Code, and, most importantly, with beefier models like Opus, which can better reason and ask better questions. Doing it with lower-level models always gave me much worse results.

I’m using it with Claude Code, not Claude Chat, because I want the model to scan my codebase. I can ask it to look in certain areas or to reference my answers. I can even ask to search the web or MCPs like Context 7 to check documentation and APIs for popular libraries. Then it’s getting more into brainstorming sometimes, than rubber-ducking, but that’s fine.

As a result, we’ll get two artefacts:

• qa.md - with the log of the back-and-forth discussion,

• spec.md - in theory spec built by LLM, but imho it’s more of a concise summary.

It may look like Specification-Driven Design, but it’s not.

My goal for this exercise is not to get an actionable specification.

The goal is to get our LLM-based Rubber Duck to ask us hard questions and make us think, not to make the LLM think for us. Find blind spots, and challenge our thinking.

But we’re drivers, we need to know what we want to do, we need to know all the WHYs, and we also need to know HOW. LLM is here to help, but not to do creative work for us. It just pulls it out from our heads.

It also helps to see how our design may be seen by others, especially such mediocre thinkers as LLMs.

I don’t expect the Agent to be able to start implementing the spec. I expect it to reflect all considerations and summarise findings. I’m always double-checking to make sure it includes all the important points. If not, I’ll keep doing Q&A until I’m satisfied.

Having both of those files will allow us to keep a full discussion without losing important details, and a shorter version. We can feed that to another model for review, or try to work on tasks and develop a more detailed plan. Sometimes I plan on my own; for simpler tasks, I may ask the LLM to do it fully. Usually, I’m driving the LLM step by step, passing just specific asks.

The example? Why not.

I recently did such an exercise, trying to narrow down how to introduce the Second-level cache to Pongo and Emmett. What’s Second-level cache? A second-level cache is a local store of data managed by the persistence provider to improve application performance.

Why do I want to introduce it? Because I got an issue from the user that rebuilding projections with a lot events can take too long. One of the reasons is that applying an event on a projection takes:

• loading the current state,

• updating it,

• storing the result.

Those are two operations per event. If we have a batch of 100 events, that’d mean 200 operations; for 1000 events, this would be 2000, so the classical N+1 problem. We could do it differently, and within a batch:

• group events by the target documents,

• load all of them in one operation by finding all documents within an array of ids (taken from events),

• caching them,

• applying events in memory,

• storing updated documents in one operation.

Then we’ll get at worst 102 operations for 100 events and 1002 for 1000 events, so a linear increase.

Still, I decided to add batching and introduce 2nd-level cache as a solution. I and my experience were a driving force behind that. I could, of course, rubber duck it, but I had that already planned, not just detailed. I also did some initial research before doing the session.

Let me show you the result below. Not too bad if you ask me. Of course, the task wasn’t very innovative; it’s a standard way for mature solutions. Still questions were surprisingly good and helped me.

I hope this log will show you (besides how bad I’m at typos) some insight into how I think, talk, and consider the design.

Tell me your thoughts, if that matches your way, if you haven’t done it yet, try it and tell me how it went!

Interactive Rubber Ducking example

My idea

I’d like to provide 2nd-level cache to Pongo. I want to either use keyv (https://www.npmjs.com/package/keyv) or lru-cache (https://www.npmjs.com/package/lru-cache) packages.

For now, I’d like to do it by id.

I probably need to either extend pongoCollection src/packages/pongo/src/core/collection/pongoCollection.ts with findeOneById and findManyByIds methods, or check if _id is passed in and then try to access the cache.

Cache should be set up either on the pongo collection, pongo db or pongoClient.

Optimistic Concurrency errors should invalidate cache (as that means someone else updated the record in db, e.g. through a different node).

InsertMany and insertOne should put it into the cache.

I’d also like to expand the handle method to allow passing either an id or an array of IDs to support batch loading of entities and improve performance, and to support batch update/insert/delete.

Thoughts?

Which defaults should I set? (Of course, allowing users to override it).

Check the related code I pasted. I want to keep it aligned and as non-invasive as possible.

Q1: Cache library choice — lru-cache, keyv, or custom interface?

Question: You mentioned both keyv and lru-cache. They pull in different directions. Which trade-off do you prefer?

• A) Start with lru-cache, optionally let users swap adapters later.

• B) Start with keyv so cross-node sharing works out of the box.

• C) Define a thin cache interface (get/set/delete/clear) and ship lru-cache as the default adapter.

Answer: Custom cache interface (option C). No dependency on external APIs — keep it pluggable. Start with an in-memory provider only. Considered writing a custom LRU with me, but wants to ship fast. Key point: don’t stick to any third-party API surface, allow pluggability.

Q2: Where should the cache live in Pongo’s hierarchy?

Question: Three attachment points: A) PongoClient level (shared across all), B) PongoDb level (per database), C) PongoCollection level (per collection). Or configurable at multiple levels?

Answer: Allow defining it at different levels — cascading configuration. Set a default higher up, override per collection if needed.

Q3: Inheritance/override behavior for cascading cache config?

Question: For the cascading config — what should the inheritance/override behavior be?

Two sub-questions:

1. Should a lower level be able to disable caching even if a parent enables it?

2. Should cache settings (like max size, TTL) also cascade, or only the cache provider instance itself?

Answer:

1. Yes. Cache should be an object with type, general params and specific for this type, or 'disabled'. That’s a proper devex.

2. Yes, as above. But if one selects a different cache type then general params should be inherited but type-specific should not be inherited and should fallback to defaults.

Claude’s thoughts: The CacheConfig | 'disabled' pattern is clean — discriminated unions play nicely with TypeScript. The inheritance rule makes sense: general params (TTL, max size) cascade, but type-specific params reset to defaults when you switch providers, avoiding leaking nonsensical config (e.g., a Redis connection string bleeding into an in-memory cache). Also worth distinguishing between “I didn’t specify anything” (inherit from parent via undefined) and “I explicitly want defaults” (reset) — undefined vs 'disabled' vs CacheConfig gives a clean three-state: inherit / off / explicit.

Q4: Cache interface shape — async, batch ops, scoping?

Question: Settling the cache interface details:

1. Async or sync? Making the interface Promise-based from the start keeps it pluggable. Sound right?

2. Batch methods (getMany, setMany, deleteMany) — should the cache interface support batch ops natively, or loop internally?

3. Should clear() be scoped per collection, or always clear the entire cache?

Answer:

1. Maybe we could return either promise or no promise, then if we await it, it would support both? Also ensure that settings are close to what lru-cache and keyv are having.

2. Having them (even as syntactic sugar) would be a good devex.

3. We could pass down not only settings, but also the cache provider instance, then if settings are the same we can reuse cache without recreating it. If set had ttl provided, then we could do it by overriding those TTLs on item level. ClearAll probably would be something we should expose only for advanced, typically user should not be aware of cache.

Claude’s thoughts: MaybePromise<T> (i.e. T | PromiseLike<T>) works — await on a non-Promise just resolves immediately. No overhead for sync providers, no breaking change for async ones. Passing a cache instance is smart: config becomes either settings (Pongo creates the cache) or a pre-built instance (user brings their own). Multiple collections can explicitly share one cache instance. clear() should be internal/advanced — scope by collection prefix internally so a collection-level clear doesn’t nuke another collection’s entries when sharing an instance. For aligning with lru-cache/keyv settings: both support max (max entries) and ttl (TTL in ms). Start with max and ttl as the general params.

Q5: Cache key strategy

Question: Since cache instances might be shared across collections, how to avoid key collisions?

• A) Pongo manages key prefixing internally ({dbName}:{collectionName}:{id}). The cache interface works with plain string keys, namespacing is Pongo’s concern.

• B) Key is just the document _id. Simpler but only safe with per-collection instances.

Answer: A, definitely.

Q6: Cache invalidation and consistency with optimistic concurrency

Question: How should cache interact with different operations?

1. Successful writes → update/evict cache. Clear.

2. Concurrency errors → evict stale entry. Clear.

3. Reads with _id filter → cache hit or miss-then-populate. Clear.

4. handle method — should the read inside handle use the cache (risking more concurrency errors on stale data) or always go to DB?

Answer: Definitely use cache. We could consider disabling cache through handle options.

Claude’s thoughts: Use cache by default — optimistic concurrency is the safety net. If cached version is stale, write fails, we evict, caller can retry. Adding skipCache?: boolean to HandleOptions is a clean escape hatch for high-contention cases.

Q7: Should skipCache be available on read methods too?

Question: Should skipCache also be available on findOne and other read methods for symmetry, or keep it handle-only?

Answer: Good point, having skipCache on find methods (also findMany etc.) would be good.

Claude’s thoughts: Consistent API surface. Add skipCache?: boolean on the shared options type (e.g. CollectionOperationOptions) so it’s available everywhere without repeating yourself.

Q8: Batch operations on handle — signature and handler shape

Question: You want handle to accept an id or array of ids. Two design choices:

1. Overloaded signature (handle(id: string | string[], ...)) vs separate method (handleMany)?

2. Handler shape for batch: A) still receives/returns one document at a time (Pongo batches DB ops internally) or B) receives/returns full array?

Also noted: since PongoDocuments always have _version, if someone provides expectedVersion we could compare it against the cached version and decide if it’s stale without hitting the DB.

Answer: Option A for handler shape. handleMany would be consistent with other APIs, but single id or array (string | string[]) is clean enough.

Claude’s thoughts: Version comparison in cache is a cheap short-circuit — if cached _version doesn’t match expectedVersion, we know it’s stale without a round trip. On signature: TypeScript overloads can narrow the return type (string → single result, string[] → array). handle is a higher-level orchestration method, not direct CRUD, so a union input feels natural. Go with handle(id: string | string[], handler, options?).

Q9: Defaults — max entries, TTL, enabled by default?

Question: What defaults to set? Proposed: max: 1000, ttl: 60_000 (1 min), and should cache be enabled by default or opt-in?

Answer: Stick to recommended lru-cache defaults for now. Enable cache by default.

Claude’s thoughts: Enabling by default works because: (1) optimistic concurrency is always the safety net, (2) it’s document-by-id only so invalidation is straightforward, (3) users can set 'disabled'. lru-cache requires max to be set (no default) and has no default TTL. Suggested: max: 1000, no TTL (rely on LRU eviction + invalidation on writes/concurrency errors) — most lru-cache-aligned.

Q10: TTL or no TTL as staleness backstop?

Question: No TTL means cached entries live until LRU-evicted or invalidated by a Pongo write. If something updates the DB outside this Pongo instance (another service, direct SQL), the cache won’t know. Options:

• A) No TTL, accept the trade-off — OC catches it on next write.

• B) Set a conservative default TTL (e.g., 5 minutes) as a staleness bound.

Answer: Fine with using TTL and sometimes hitting OC issue. OC assumes rare conflicts.

Q11: Where does cache integration hook into existing code?

Question: Two approaches for minimal invasiveness:

1. Wrap at the pongoCollection level — intercept operations inside the factory function, driver/SQL layers untouched.

2. Decorator/wrapper pattern — cachedPongoCollection wraps PongoCollection, adds cache logic around delegation.

Answer: Leaning more to option 1. Thinks it’ll be easier to make observability precise eventually.

Claude’s assessment: Agreed with option 1. Cache decisions depend on internal state (whether filter targets _id, document _version, write success vs concurrency error). A wrapper would need to re-inspect all of that from the outside. For observability, cache hit/miss logic next to the actual DB call enables precise metrics in context. Pongo’s factory functions already compose behavior internally (transactions, schema migration) — cache is another concern at the same level. A decorator would duplicate filter-inspection logic and add indirection that doesn’t carry its weight.

Q12: Cache interaction with transactions

Question: How should cache interact with transactions?

1. Should writes within a transaction update the cache immediately, or only after commit?

2. Should reads within a transaction check the cache?

Answer:

1. After commit.

2. We should use cache, but allow skipping it on transaction or session options. We can also allow setting cache settings per session.

Claude’s thoughts: Reads within a transaction benefiting from cache is more aggressive but logical — most transactions are short-lived. The narrow case: insertOne then findOne same _id within a transaction — cache won’t have the uncommitted insert, falls through to DB, gets the right answer anyway. No correctness issue, just no perf benefit for that case. Per-session cache settings fit the cascading pattern (e.g., startSession({ cache: 'disabled' }) for batch migrations). Proposed cascade priority: client → db → collection → session → per-operation skipCache.

Q13: Cascade priority — does session override collection?

Question: Does the cascade priority client → db → collection → session → per-operation skipCache feel right? Session overriding collection is the non-obvious part — the argument being that session is the most immediate runtime context, while collection is structural config. Or would you prefer collection always wins over session?

Answer: That’s fine. Session should precede collection, as normally you create session to e.g. share connection, and join the number of operations logically, so it’d make sense actually to also do it to e.g. override settings like caching.

Claude’s thoughts: Session as logical grouping of operations with shared context (connection, transaction) — overriding cache settings at that level is extending the same concept. Bulk import in a session? Set cache disabled once on the session instead of skipCache on every call. Final cascade: client → db → collection → session → per-operation.

Q14: Cache provider error handling

Question: If the cache provider throws (e.g., Redis connection drops), should Pongo:

• A) Treat it as a cache miss — fall through to DB silently, maybe log a warning. Cache failures never break the application.

• B) Propagate the error — let the caller deal with it.

Answer: A — yes, when we add observability to Pongo, we need to make it also a metric and trace event/attribute/log entry.

Claude’s thoughts: Cache is an optimization, not a correctness requirement. Silent degradation now, observable degradation when the observability layer lands.

Q15: Cache warming, events/hooks, and delete operations

Question: Three remaining areas:

1. Cache warming — should Pongo support pre-populating the cache on startup?

2. Cache events/hooks — should users be able to listen to cache hits/misses/evictions?

3. Delete operations — deleteOne/deleteMany should evict from cache. Anything non-obvious there?

Answer:

1. For now keep it as a future concept, or define pluggable strategy for users.

2. You can add the basic hook to the Pongo Cache wrapper. Shouldn’t be that hard to just call some function.

3. Yes, they should evict if they can, we can delete (for now) when deleteMany deletes by ids.

Claude’s thoughts: Cache warming deferred — can add a strategy hook later. Basic event callbacks (onHit, onMiss, onEvict, onError) on the cache wrapper are low-cost and useful for debugging even before full observability. Delete eviction scoped to id-based deletes for now — filter-based deleteMany doesn’t evict (no way to know which ids were affected without querying first).

Interactive Rubber Ducking result

Pongo 2nd Level Cache — Specification

Overview

Add a document-level (by _id) 2nd level cache to Pongo with a pluggable provider interface. Ships with an in-memory provider backed by lru-cache. Enabled by default. Designed to be minimally invasive to existing code.

Cache Interface

PongoCacheProvider<T>

A thin, Pongo-owned interface. No coupling to any third-party API.

type MaybePromise<T> = T | PromiseLike<T>;

interface PongoCacheProvider {
  get(key: string): MaybePromise<PongoDocument | null | undefined>;
  set(key: string, value: PongoDocument, options?: { ttl?: number }): MaybePromise<void>;
  delete(key: string): MaybePromise<void>;
  getMany(keys: string[]): MaybePromise<(PongoDocument | null | undefined)[]>;
  setMany(entries: { key: string; value: PongoDocument; ttl?: number }[]): MaybePromise<void>;
  deleteMany(keys: string[]): MaybePromise<void>;
  clear(): MaybePromise<void>;
}

• MaybePromise return types: sync providers (in-memory) return values directly, async providers (Redis) return Promises. await handles both transparently.

• Batch methods (getMany, setMany, deleteMany) are first-class. Default in-memory implementation may loop internally, but the interface allows optimized batch ops for external providers.

• clear() is internal/advanced — not exposed to typical users. When sharing a cache instance across collections, scoping is handled via key prefixing by Pongo, not by the provider.

Cache key strategy

Pongo manages key prefixing internally: {dbName}:{collectionName}:{documentId}.

The cache provider works with plain string keys — namespacing is Pongo’s concern, not the provider’s.

Event hooks

The Pongo cache wrapper supports basic callbacks:

• onHit?(key: string): void

• onMiss?(key: string): void

• onEvict?(key: string): void

• onError?(error: unknown, operation: string): void

These are optional and intended for debugging and future observability integration.

Configuration

CacheConfig

type CacheConfig = {
 type: string;               // e.g., 'in-memory', 'redis', etc.
 max?: number;               // max entries (general param, cascades)
 ttl?: number;               // TTL in ms (general param, cascades)
  // type-specific options live here too, keyed by type
 [key: string]: unknown;
} | 'disabled';

Three states:

• undefined — inherit from parent level

• 'disabled' — explicitly turn off caching at this level

• CacheConfig object — explicit configuration

Cascading configuration

Cache config can be set at multiple levels. Each level inherits from its parent unless explicitly overridden:

client → db → collection → session → per-operation

Inheritance rules:

• General params (max, ttl) cascade down.

• If a lower level switches type, type-specific params reset to defaults (not inherited from parent).

• Session overrides collection — session is a logical grouping of operations, natural place to override runtime behavior (e.g., disable cache for a bulk import).

• Per-operation skipCache?: boolean is the most granular escape hatch.

Passing a cache instance

Users can provide either:

• Settings — Pongo creates and manages the cache provider.

• A pre-built cache provider instance — Pongo uses it directly.

If settings are the same across multiple collections, Pongo can reuse the same provider instance internally. When a user passes an instance, multiple collections can explicitly share one cache.

Defaults

• Enabled by default

• max: follow lru-cache recommended defaults (1000)

• ttl: follow lru-cache recommended defaults

• Default provider: in-memory (lru-cache)

Integration points

Where: pongoCollection factory function

Cache logic is added directly inside pongoCollection, not as an external decorator/wrapper. This gives cache operations access to internal state (filter inspection, _version, write outcomes) and keeps observability precise.

Read operations

findOne:

• If the filter targets _id, check cache first.

• Cache hit → return cached document.

• Cache miss → query DB, populate cache, return.

• skipCache?: boolean option available.

findMany / other query methods:

• If the filter is a list of _id values, check cache for each.

• Return cached hits, query DB for misses, populate cache with DB results.

• Non-_id filters bypass cache entirely (cache is by-id only).

Write operations

insertOne:

• After successful insert, put the document into cache.

insertMany:

• After successful insert, put all documents into cache.

updateOne / updateMany / replaceOne:

• After successful write, update the cache entry with the new document state.

deleteOne:

• After successful delete, evict from cache.

deleteMany:

• If deleting by ids, evict those ids from cache.

• Filter-based deleteMany does not evict (no way to know affected ids without extra query). Future improvement possible.

Optimistic concurrency

• On concurrency error (version mismatch), evict the stale entry from cache. This is critical — a concurrency error means someone else updated the record (e.g., from another node), so the cached version is stale.

• If a caller provides expectedVersion and the cached document has _version, compare them in memory. If they don’t match, we know it’s stale without hitting the DB — cheap short-circuit.

handle method

• Accepts id: string | string[] (overloaded: string returns single result, string[] returns array).

• Read phase uses cache by default.

• Write phase updates/evicts cache based on outcome.

• skipCache?: boolean available in handle options.

• For batch: loads all ids from cache, fetches misses from DB in one query, processes handler per document, batch writes to DB, updates cache after commit.

Transactions

• Writes update cache only after commit. Not during the transaction — uncommitted data should not enter the cache.

• Reads within a transaction check cache by default (most transactions are short-lived, cached data is probably valid). If the transaction modified a document and then reads it, cache miss falls through to DB which returns the correct uncommitted state.

• Cache can be disabled per-session: startSession({ cache: 'disabled' }).

• Per-session cache settings follow the cascade (session overrides collection).

Error handling

Cache provider errors are swallowed — treated as cache misses. The application never fails because of a cache failure.

• Cache is an optimization, not a correctness requirement.

• Errors trigger onError hook for debugging.

• Future: errors will become metrics and trace events when Pongo’s observability layer lands.

Future considerations (out of scope for v1)

• Cache warming: pluggable strategy for pre-populating cache on startup. Interface hook defined but no default implementation shipped.

• Query-result caching: caching results of non-_id queries. Complex invalidation — deferred.

• Cross-node cache sharing: Redis/Memcached providers. The interface supports it; no provider shipped in v1.

• Filter-based deleteMany eviction: would require querying affected ids before delete.

• Observability integration: metrics (hit ratio, latency saved, eviction counts), trace attributes, log entries for cache operations.

Implementation approach

1. Define PongoCacheProvider interface and CacheConfig types.

2. Implement in-memory provider using lru-cache.

3. Build Pongo cache wrapper (key prefixing, event hooks, error swallowing).

4. Wire cascading config resolution (client → db → collection → session → per-op).

5. Integrate into pongoCollection: intercept _id-based reads, update cache on writes, evict on deletes and concurrency errors.

6. Extend handle to accept string | string[] with batch support.

7. Add skipCache to operation options.

8. Tests: unit tests for cache provider, integration tests for cache + collection operations, e2e tests for concurrency scenarios.

Cheers!

Oskar

p.s, and yes, Nano Banana also did pretty well generating the cover of this post!

p.s.2. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

Be careful what you wish for, because your wish may come true.

What LLMs revealed is how many people in our industry don't like to code.

It's intriguing that now they claim and showcase what they "built with Claude", whereas usually that means they generated a PoC.

It's funny, as people still focus on how they're building, so it's all about the code. And if that's the message sent outside, together with the thought that LLMs are already better than "average coder Joe", then the logical follow-up question is: why do we need those humans in the loop?

I think that most people look at the forest and see trees. The current way of working with LLMs is not scalable. It's transition phase. I can't imagine calling myself an engineer and doing ONLY stringly-typed development with chat or markdown.

What I can imagine is getting help from it, and using those tools as a help for research, for generating the OUTPUT, still keeping me responsible for the OUTCOME.

Why am I saying that we're in the transition phase? As prompting in our natural language is not precise, it's verbose, and adding a translation layer between our freeform prompts and programming languages is a waste of time (and tokens, which LLM vendors love).

I think that we'll still be coding, but with some other layer, as LLMs are good with structured input, like programming languages. So we might need other programming languages than we have atm. Might we need different tools to evaluate LLMs' output to make it deterministic? Might we need a different approach for engineering to make it scalable? Might we need more?

Still, I don't see those discussions.

I mostly see: noise and celebrities who don't code showing their beautiful PoCs. And doing a mic drop about the end of coding. Just like PoC represents the whole Software Development Life Cycle.

So, will we code or not?

If the answer is yes, let's talk about what's next, then let's discuss how and when.

If the answer is not, then let's talk about how our job will change, ask if it's still engineering, etc.

Let's try to make our discussions more precise, more focused on the essence, and avoid them from becoming just a bunch of anecdotal evidence.

Noone outside our industry cares how we code unless it changes the cost, quality, or delivery time.

Let's discuss the impact that matters, rather than just the amount of code we produce (or not).

Now, guess who I am quoting:

Imagine you’re a software application developer. Your programming language of choice (or the language that’s been foisted on you) is Java or Typescript . You’ve been at this for quite a while and your job doesn’t seem to be getting any easier.

These past few years you’ve seen the growth of multiple incompatible architectures. Now you’re supposed to cope with all this and make your applications work in a distributed client-server environment. The growth of the Internet, the World-Wide Web, and “electronic commerce” haveintroduced new dimensions of complexity into the development process.

The tools you use to develop applications don’t seem to help you much. You’re still coping with the same old problems; the fashionable new object-oriented techniques seem to have added new problems without solving the old ones.

You say to yourself and your friends, “There has to be a better way”!

The Better Way is Here Now

Now there is a better way—it’s our new model. Imagine, if you will, this development world…

• It’s still dead simple.

• Your development cycle is much faster.

• Your applications can be created across multiple platforms. Write your spec once, and you never need to port them—they will be recreated if you without your hand-rolled modification on multiple operating systems and hardware architectures.

• Your applications are adaptable to changing environments.

• Your end users can trust that your applications are secure, and you can use protection against viruses and tampering through security scans.

You don’t need to dream about these features. They’re here now.

And also this from another source:

When I started interviewing programmers in 2005, I would generally let them use any language or tool they wanted to solve the coding problems I gave them. 99% of the time, they chose Java.

Nowadays, they tend to choose LLMs.

Now, don’t get me wrong: there’s nothing wrong with LLM as an implementation tool.

Wait a minute, I want to modify that statement. I’m not claiming, in this particular article, that there’s anything wrong with LLM as an implementation tool. There are lots of things wrong with it but those will have to wait for a different article.

Instead what I’d like to claim is that LLM is not, generally, a hard enough programming tool that it can be used to discriminate between great programmers and mediocre programmers. It may be a fine tool to work in, but that’s not today’s topic. I would even go so far as to say that the fact that LLMs aere not hard enough is a feature, not a bug, but it does have this one problem.

Well, I cheated you, but only a bit. I changed “Java” to “LLM ” and cut some phrases.

The first one is from “The Java Language Environment” by Sun Microsystems, introducing Java in 1995.

The second one was from Joel Spolsky’s “The Perils of JavaSchools” article, written in 2001.

Let me be clear. I’m not trying to do grandpa talk on the old days and claim that it’s the same old thing.

What I’m trying to say is that we were continuously introducing new abstractions into our development cycle to scale it. By scale, I mean: getting more people to deliver more code. Even Java was invented for precisely this goal. Yes, the one that’s together with craftsmanship madness presented as “the enterprisy complex environment”. In the early days, it was just said that it’ll make us dumber.

The goal of abstraction is not to gatekeep but to allow us to reduce cognitive load. We invented new languages to help us, but then added more components like a distributed environment, multiregion, because of globalisation. For the same reasons, we use cloud-native tooling so we don't have to deal with it.

Some of the architecture and security tools were commoditised by the cloud. We don’t need to think about much stuff we had to do before.

Will Claude-native do the same?

We don’t need to learn Assembler, C++, Lisp anymore; we have lost a lot of mechanical sympathy. We deal with higher abstractions, but we still engineer solution, we still code, is it a different coding? It is. Is it better or worse? Well, it is how it is. As Gerald Weinberg said:

Things are the way they are because they got that way

And now the question is whether we’re fine with the way we’re doing stuff, and where we're getting to.

Personally, I don’t think that stringly-typed markdown or chat-based design will be the thing in the future.

Knowing how skilled we were always with:
- breaking down tasks into smaller chunks,
- writing precisely what we had in mind,
- thinking before doing,
- waging tradeoffs,

I'm optimistic about the Spec-Driven Design idea. It's going to be great.

Not.

If we want to call ourselves engineers, we need to put more structure and determinism.

I agree that reviewing all code generated by the GenAI is not sustainable.

But I also don’t think that generating tons of code is, in general, sustainable.

With the current state of the art we have, sure, that’s some solution to just generate based on the tools we have.

But let’s start to think what’s next.

Simon Wardley brought an interesting point:

That said, it is fine for an entire culture to decide that producing outputs matters more than understanding mechanisms. You only have to compare the practical engineering of the Roman Empire and the loss of inquiry from science in the Hellenistic age to see this. When the Roman Empire collapsed, the practical knowledge embedded in those institutions (how to maintain aqueducts, how to produce certain grades of concrete) was lost remarkably quickly. Not because people decided to forget it, but because the knowledge was procedural, embedded in chains of practice rather than recorded as transferable understanding. When the chains of practice broke, that embedded knowledge went with them. We had to rediscover the art of inquiry (i.e. Science) to bring them back.

So if we don’t code, how do we hone our skills? How will newcomers from LLMSchools (using Joel’s terms) be able to decide whether something is wrong or right? I don’t think that you can be good on something without doing it.

The paradox with code is that it’s not an asset; it’s a liability. Some say that code doesn’t matter, only proper design. But how do you define “proper design”? Yes, code style doesn’t matter as long as it works as expected. But…

But code eventually matters, as that’s the source of truth for what’s on production. As Alberto Brandolini said:

It's developers' (mis)understanding, not domain experts' knowledge, that gets released in production.

Now, it’s the developers’ and LLMs’ misunderstandings that are deployed to production, not the expert’s knowledge. Neither the markdown spec.

And coding is just one danger.

Outsourcing thinking is an even more dangerous path, as:

• If LLMs are doing everything, then again, what are humans for? Aren’t we cutting the branch on which we’re sitting?

• LLMs are statistical parrots. They repeat the most possible answer. Which means mediocre. This can still be fine enough for many cases, but for those we want to make a difference for? Definitely not.

• Just like we’re losing our coding skills by not doing them, we’re losing design skills by not practising them.

Of course, whatever happens, LLMs will stay with us. How and where it’s hard to say. Unless you have a Magic 8 Ball of 100% correct predictions. I don’t.

That’s why I’d like our industry to finally start mature discussions on the real impact. I would like us to stop acting like children, bragging about generating code and then claiming that code doesn’t matter.

I’d like to think about how to reshape our SDLC process and make it sustainable.

I’d like us to think about what tools we need, and how to change what we have.

If we won’t finally start to do it, then things will be the way they are because they got that way.

And it might not be what we wished for.

Cheers

Oskar

p.s. to kinda prove that I’m more sceptic and pragmatic than hater, I recently started playing with building an Agent with Emmett to better understand those tools. If you’d like to read about the findings and honest thoughts I have while doing it, please comment!

p.s.2. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

Last time, I shared with you how sneaky I was on transaction handling.. Today, the opposite: I’ll tell you how I fixed the issue when I tried to be too sneaky. I already told you that Sneaky Code Bites Back. The moral? Do as I tell, not how I do.

In some environments, we’re spoiled. We’re getting a lot from a Base Class Library or standard frameworks, so we stop thinking that those issues can exist. For instance, serialisation. Do you know how many data types JSON has? 6. Six. Sześć.

Exactly those:

• string,

• number,

• boolean,

• object,

• array,

• and (TADA!) null.

What about number precision and size? It is. That’s what I can tell you, but it’s not enough, e.g., to keep big int/long, etc. What about Dates? Also, there are none. I wrote about it longer here or how much fun that brings.

If you use statically typed languages and runtimes like C#, Java, etc., your serialiser can, in addition to parsing, perform mapping and, sometimes, validation. And it can also be tricky, as nicely Alexis King put in her “Parse, don’t validate”.

If you’re in a dynamic environment, like JavaScript, then you’re left with parsing and explicit mapping afterwards. What about TypeScript? Same case, types are only used during compilation, then erased and not visible at runtime. So, the place where we do parsing.

Because JSON was defined a long time ago, JavaScript moved on and now supports bigints (Big Integers) and Dates natively (what an achievement!), which created a gap I wanted to fill.

As you know from my previous articles (e.g. this one), big integers are quite important in distributed processing. You can represent the position in log with them. Since your log may be quite long, regular numbers aren’t enough. Or they’re long enough, until they overflow, then they’re not anymore.

I’m using those bigint types extensively in internals in Emmett and Pongo. And I store them in JSONs. I store them as alphanumeric strings, because strings don’t have a max length (or at least I don’t know such).

So, for instance, an event payload can look like:

{
  "type": "InvoiceIssued",
  "data": {
    "invoiceNumber": "123",
    "version": 1,
    "issuer": "John Doe",
    "issuedAt": "2026-02-23T14:07:20Z"
  },
  "metadata": {
    "streamPosition": "3",
    "globalPosition": "928391"
  }
}

As you can see in the metadata stream and global positions, the values are bigints (even if they’re smaller than the maximum value), and data can also use bigints if the user decides to (e.g., invoice number).

And encoding data is simple: you convert it to a string, call it a day. But how to get it back?

And here’s where my struggles started. How do you know that someone intentionally used bigint when they just wanted to store a number as a string?

There are several options. The first one is: encode value.

We could store it, for instance, as:

• prefixed value: “_bigint:928391”. But then you need to find a prefix that will be unique enough not to cause conflicts,

• nested object, e.g. { “_kind”: “bigint”, value: “928391” }.

Then, either based on the prefix or the object structure, we could automatically decode the value. Still, ther creates other issues, as the structure no longer matches the original value. If we’re just storing and retrieving, that shouldn’t be so bad, but… But remember that in Pongo I’m allowing the use of PostgreSQL and SQLite as document databases, supporting such queries:

const invoices = pongoDb.collection<Invoice>("invoices");

const invoiceNumber = 123n;
const invoice = await invoices.findOne({ invoiceNumber });

That gets translated into a fancy JSONB SQL query.

Of course, I could work around it by encoding the value, but… But I was lazy!

I decided to use a Get Out of Jail Free Card and just treat all strings with numbers as bigints. Sneaky. And it will get even sneakier.

In JavaScript, JSON.parse accepts a parameter that allows you to provide custom mapping logic. I decided to use it and check if the string is alphanumeric, and gulp, I’ve used a regular expression to parse it:

const bigIntReviver: JSONReviver = (_key, value) => {
  if (typeof value === 'string' && /^[+-]?\d+n?$/.test(value)) {
    return BigInt(value);
  }

  return value;
};

Yes, it’s either DNS or Regex. Or both.

I explained in another article that JavaScript runtime doesn’t like where you do CPU-heavy computations.

Small Regex isn’t CPU-heavy, but if you consider that ther will be done for each string in each document or event you try to deserialise, and multiply that by the number of concurrent requests? That can cause the JavaScript event loop to freeze.

What’s more, I plugged that automatically into node-postgres driver custom type handling, so each JSONB deserialization goes through it.

Again, not shit, Sherlock. I should have known it wasn’t the best choice, but I was busy trying to be sneaky at that moment.

Fortunately, a user, Dawid, benchmarked and noticed CPU freezes. It wasn’t catastrophic, but clearly needed a fix.

The Shift

And here I had several options. I could keep hacking on the same idea- maybe replace the Regex with a simpler string check, still globally. Or I could just ignore bigints during deserialisation entirely, let them stay strings, call it a day, move on. Or I could apply the encoding I mentioned earlier, prefixed values, and nested objects. All of those would fix the performance issue. And all of those would be the same kind of wrong choice I already made: trying to solve a schema problem without the schema.

Because that’s the actual mistake here, not the Regex. The pg driver has no idea what your schema looks like. It doesn’t know that “928391” is a bigint and “John Doe” is a name. It doesn’t know that “123” is an invoice number (bigint!) and “90210” is a zip code (string!). I asked it to guess, and it guessed wrong, because there is no right guess at that level.

Enough is enough. I had been planning to do ther properly for a while, and the performance issue gave me the push I needed.

Old rule says: “Make it work, make it right, make it pretty”. I had “make it work” covered for a long time. Now it was time for “make it right”.

And honestly? It wasn’t that hard. Maybe because “make it work” came first, I already understood the problem well enough to see the shape of the solution.

In Pongo, I dropped the automatic bigint parsing from the driver entirely. If you want bigint or date parsing, you say so at the client level:

const client = pongoClient({
  driver: databaseDriver,
  connectionString: postgresConnectionString,
  serialization: {
    options: {
      parseBigInts: true,
      parseDates: true,
    }
  },
});

By default, strings stay strings. You opt in. I didn’t want to break things for users who don’t care about bigint precision or don’t have performance-sensitive workloads. The serializer became an explicit parameter passed down to each query, each collection, each operation- instead of a global that silently changed everything.

That was the “make it right” part for Pongo. But disabling alone isn’t a solution, it’s a band-aid. Users who need bigints and dates still need a way to get them back after deserialisation. The question is: where does that conversion happen?

And that’s where upcasting comes in. Let me start with a simple example in Pongo, then build up.

Say you have a user document. In the database, dates are stored as ISO strings, and the version counter is a numeric string (because JSON). But in your application, you want proper Date objects and bigints:

type UserDocStored = {
  name: string;
  createdAt: string;
  lastLogin: string;
};

type UserDoc = {
  name: string;
  createdAt: Date;
  lastLogin: Date;
};

The upcast function does the conversion:

const upcast = (doc: UserDocStored): UserDoc => ({
  name: doc.name,
  createdAt: new Date(doc.createdAt),
  lastLogin: new Date(doc.lastLogin),
});

You wire it into the collection, and every read goes through it:

const collection = pongoDb.collection<UserDoc, UserDocStored>(
  'users',
  {
    schema: { versioning: { upcast } },
  },
);

What’s in the database:

{ "name": "Alice", "createdAt": "2024-01-15T10:30:00.000Z", ... }

What you get back:

{ name: 'Alice', createdAt: Date, ... }

That’s all. new Date(str) is cheap. Running a Regex against every string in the document is not. The CPU freeze Dawid spotted came from that check running millions of times at the driver level for every field on every concurrent request. With upcasting, the conversion runs only for the fields you declared, in a plain function, no Regex.

But ther is just type mapping - the simplest case. As I wrote about in my serialisation article, the explicit mapping pattern is useful for much more than just fixing types. It’s the same pattern you need for schema versioning. It defines the stored schema and the application schema separately together with function to transform one into the other.

Let’s say business requirements changed. You now need to group user data differently: a profile object for identity data, and a timestamps object for temporal data. The V1 documents are flat. The new V2 shape is nested:

type UserDocV1 = {
  name: string;
  createdAt: string;
  lastLogin: string;
};

type UserDocV2 = {
  profile: {
    name: string;
  };
  timestamps: {
    createdAt: Date;
    lastLogin: Date;
  };
};

Ther isn’t just a type change like string-to-Date anymore. The structure itself is different. Flat fields became nested objects, and field names moved into sub-objects. And you have thousands of V1 documents already stored. You can’t migrate them all at once (or don’t want to, because it’s risky, and some consumers might still expect V1). But your application now expects V2.

Compatibility FTW

Ther is where backward and forward compatibility come in.

Backward compatibility means: old data still works. V1 documents stored months ago need to be readable by the V2 code. The upcast handles ther. It reads the document in whatever shape it has and transforms it into V2.

Forward compatibility means: new data doesn’t break old consumers. If you have another service or an older deployment that still reads the V1 format, it needs to keep working. The downcast handles ther. When storing V2 documents, it writes the V1 fields alongside the V2 fields, so older readers can still find what they expect.

Together:

type StoredPayload = UserDocV1 & UserDocV2;

const upcast = (doc: StoredPayload): UserDocV2 => ({
  profile: doc.profile ?? { name: doc.name },
  timestamps: {
    createdAt: new Date(doc.timestamps?.createdAt ?? doc.createdAt),
    lastLogin: new Date(doc.timestamps?.lastLogin ?? doc.lastLogin),
  },
});

const downcast = (doc: UserDocV2): StoredPayload => ({
  name: doc.profile.name,
  createdAt: doc.timestamps.createdAt.toISOString(),
  lastLogin: doc.timestamps.lastLogin.toISOString(),
  profile: doc.profile,
  timestamps: doc.timestamps,
});

Look at the upcast: if the nested profile or timestamps fields exist (document was written by V2 code), it uses them. If they don’t exist (for the old V1 document), it falls back to the flat fields. One function handles both old and new documents: that’s backward compatibility.

And look at the downcast: it writes name, createdAt, lastLogin as flat string fields (V1 shape) alongside profile and timestamps (V2 shape). A service still reading V1 sees the flat fields and works fine. A service reading V2 sees the nested ones. That’s forward compatibility.

You wire both into the collection:

const collection = pongoDb.collection<UserDocV2, StoredPayload>(
  'users',
  {
    schema: { versioning: { upcast, downcast } },
  },
);

From here, your application code only deals with V2. The collection handles the translation in both directions:

const v2Doc: UserDocV2 = {
  profile: { name: 'Alice' },
  timestamps: {
    createdAt: new Date('2024-01-15T10:30:00.000Z'),
    lastLogin: new Date('2024-06-20T14:45:00.000Z'),
  },
};

await collection.insertOne(v2Doc);

What’s stored (downcasted, both shapes for compatibility):

{
  "name": "Alice", 
  "createdAt": "2024-01-15T10:30:00.000Z",
  "lastLogin": "2024-06-20T14:45:00.000Z",
  "profile": { 
    "name": "Alice" 
  },
  "timestamps": { 
    "createdAt": "2024-01-15T10:30:00.000Z",
    "lastLogin": "2024-06-20T14:45:00.000Z" 
  } 
}

Then you can read it back with:

const doc = await collection.findOne({ ... });

And get upcasted to V2 data in your application code:

{
  "profile": { 
    "name": "Alice" 
  },
  "timestamps": { 
    "createdAt": "2024-01-15T10:30:00.000Z",
    "lastLogin": "2024-06-20T14:45:00.000Z" 
  } 
}

Same collection, V1 and V2 documents coexisting. insertMany, replaceOne, findOne: all go through the upcast/downcast. No batch migration needed. You roll out the new code, and old documents are handled transparently.

There’s another thing the downcast gives you: querying remains backwards-compatible. Because the downcast writes the flat V1 fields alongside the nested V2 ones, a query like collection.findOne({ name: ‘Alice’ }) still works even though V2 code doesn’t use name directly anymore. The V1 field is there in the stored document. That matters if you have queries or indexes built against the old shape. They don’t break.

Now, for events, ther matters even more. In event sourcing, stored events are immutable- the log is append-only, and you don’t modify what was already written. I wrote about versioning patterns in more detail. The core idea is: your business evolves, your code evolves, your event schemas evolve, but the events in the store stay as they were. You can’t go back and rewrite them (well, you can, but you really shouldn’t). Upcasting is how you bridge the gap.

For Emmett, the same pattern works at the event store level. You define the stored shape (what JSON gives you from the database) and the application shape (what your code works with):

type ShoppingCartOpenedFromDB = Event<
  'ShoppingCartOpened',
  { openedAt: string; loyaltyPoints: string }
>;

type ShoppingCartOpened = Event<
  'ShoppingCartOpened',
  { openedAt: Date; loyaltyPoints: bigint }
>;

And an upcast that handles each event type:

const upcast = (event: Event): ShoppingCartEventWithDatesAndBigInt => {
  switch (event.type) {
    case 'ShoppingCartOpened': {
      const e = event as ShoppingCartOpenedFromDB;
      return {
        ...e,
        data: {
          openedAt: new Date(e.data.openedAt),
          loyaltyPoints: BigInt(e.data.loyaltyPoints),
        },
      };
    }
    case 'ShoppingCartConfirmed': {
      const e = event as ShoppingCartConfirmedFromDB;
      return {
        ...e,
        data: {
          confirmedAt: new Date(e.data.confirmedAt),
          totalCents: BigInt(e.data.totalCents),
        },
      };
    }
    default:
      return event as ShoppingCartEventWithDatesAndBigInt;
  }
};

You pass it when reading a stream:

const { state } = await eventStore.aggregateStream<
  ShoppingCartState,
  ShoppingCartEventWithDatesAndBigInt
>(shoppingCartId, {
  evolve: evolveState,
  initialState,
  read: { schema: { versioning: { upcast } } },
});

Or in a command handler:

const handle = CommandHandler<ShoppingCart, ShoppingCartEvent>({
  evolve,
  initialState: () => ({ ... }),
  schema: { versioning: { upcast: upcastDatesAndBigInt } },
});

The difference with events is that you can’t update them in place. For documents, you have both directions: upcast on read, downcast on write. For events, upcasting is the main tool because the event store is append-only. Old events stay as they were written. But downcasting has its place too.

Consider ther: you have a projection or a subscriber that was built months ago against the old event schema. Maybe it’s a read model that listens to ShoppingCartOpened and expects clientId as a flat string. But your current code evolved. Now ShoppingCartOpened carries a client object with id and name:

// What old subscribers expect
type ShoppingCartOpenedV1 = Event<
  'ShoppingCartOpened',
  { clientId: string; openedAt: string }
>;

// What current code produces
type ShoppingCartOpenedV2 = Event<
  'ShoppingCartOpened',
  { client: { id: string; name: string }; openedAt: Date }
>;

Upcasting enables the current code to read older events. The ones stored with just clientId. Downcasting helps old subscribers consume new events. It transforms the new client object back into the flat clientId they expect. Same principle as with documents, but especially important here because event subscribers often live in separate services or deployments that you can’t update all at once.

And the same upcast function that started as simple type mapping string → Date, string → bigint handles ther structural change too. You just add another case to the switch:

case 'ShoppingCartOpened': {
  const e = event as ShoppingCartOpenedV1;
  return {
    ...e,
    data: {
      client: { id: e.data.clientId, name: 'Unknown' },
      openedAt: new Date(e.data.openedAt),
    },
  };
}

Old events get the client object synthesised from the flat clientId. New events already have it. The evolve function only deals with the V2 shape.

And here’s where things started to click for me. I added upcasting to fix the bigint problem: explicit type mapping instead of a Regex. But the same mechanism, without any changes, also handles structural versioning. The simple string → Date mapping from the first example is the same code path as the clientId → client migration above. It’s one function, one place, one pattern for all of it: type coercion, field restructuring, schema migration.

Right decisions stack

Right decisions stack. The Regex hack was blocking the slot where upcasting should have been all along. Once I removed it, the performance got fixed, and I got schema versioning on top. One fix created room for the next one, which created room for the next. That doesn’t happen when you keep patching around the same bad decision.

Looking back, maybe the Regex wasn’t the wrong first move. The rule is “make it work, make it right, make it pretty”, in that order. The Regex made it work. It had performance problems, but it still let me ship and learn where the real problem was. If I had tried to design the upcast/downcast system from scratch, without having lived with the Regex for a while, I might have over-engineered it or missed the connection to schema versioning entirely. The understanding came from living with the shortcut.

Dawid raised a performance issue with Pongo projections, but the same Regex was running in Emmett as well. I could have fixed it in one place and called it a day. Instead, I used it as a push to do the thing I’d been planning anyway and applied it to both Pongo and Emmett to keep things consistent. Because I already understood the problem well enough, “make it right” turned out easier than I expected.

You can recover from shortcuts. You should. But you also shouldn’t be afraid to take them in the first place, as long as you come back and do it properly.

Full changes:

• Pongo PR #149,

• Emmett PR #292.

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

Not that it’s easy to teach tradeoffs, it’s a subtle art of explanation. You need to provide enough context and be precise so others don’t treat tradeoffs as a general best practice. Because they’re usually not such, they typically come from the muddy banks of the Wishkah River.

I think that a decent way is to tell the story. Not the fairy tale, but the specific tradeoff applied in practice. That’s what I’m going to do today: tell you how I cheated on database transactions.

Dumbo

Do you know Dumbo? Of course, flying elephant, such a sweetie. It’s also a codename of my Open Source project. I didn’t tell you about it so far, as it’s a shared dependency for Pongo and Emmett, responsible for:

• connection pooling,

• safe connection lifetime management,

• handling transactions,

• migrations,

• SQL execution.

Not that small a scope for a Dumb tool, aye? Still, the intention is to make usage dumb, hiding the weirdness of a specific SQL database, so I solve it once and don’t need to be constantly distracted thinking about it, but focus on:

• How to append and process events the best way in Emmett,

• How to make the best JSON handling and translation into specific SQL dialects in Pongo.

Dumbo usage is quite simple:

import { dumbo } from '@event-driven-io/dumbo';
import { pgDumboDriver } from '@event-driven-io/dumbo/pg';

const pool = dumbo({ connectionString, driver: pgDatabaseDriver });

You need to set up a specific database driver (e.g. pg for PostgreSQL or sqlite3 for SQLite), as Dumbo now supports multiple relational databases.

Having that, you can do stuff like:

import { SQL } from '@event-driven-io/dumbo';

await pool.execute.batchCommand([
  SQL`CREATE TABLE test_users (id SERIAL PRIMARY KEY, name TEXT)`,
  SQL`INSERT INTO test_users (name) VALUES ('Alice'), ('Bob')`,
]);

And also do queries:

const count = pool.execute.query<{count: number}>(
  SQL`SELECT COUNT(*) as count 
      FROM test_users 
      WHERE ${SQL.in('id', userIds)}`,
);

It’ll handle query parameterisation, data escaping, etc.

It can also handle transactions:

const users = await pool.withTransaction(async (tx) => {
  await tx.execute.command(
    SQL`INSERT INTO test_users (name) 
        VALUES (${firstUserName}), (${secondUserName})`,
  );

  return execute.query<User>(
    SQL`SELECT *
        FROM test_users 
        WHERE ${SQL.in('id', userIds)}`,
   );
});

As simple as this looks, you should already have the question about the first tradeoff I made in your head.

Why on earth I think that’s a sane move to write my own multi-database driver?!

Well, I agree, that’s not the best move at the first glance, but let me explain to you why in MY context this actually makes more sense:

1. Yes, there are tools like Knex, Kysely, Drizzle, etc. in Node.js land that handle similar stuff. They’re nice, I really like them, I really do, but… But they are all big and are bringing a lot of their conventions, and when I’m building storage tools like Emmett and Pongo, I need to have more control. I don’t want those tools and their limitations drive my architecture decisions. I also don’t want to be surprised when the creator decides to drop working on it, or when I become a victim of a supply chain attack. Still, when I want a new kitchen table, I don’t start by going to the forest with a saw. I’m still using existing database drivers, what’s more I’m allowing people to choose the one they prefer. I just don’t want to be driven (and also my users be driven) by some other higher abstraction tool.

2. Even with that, this decision can seem like a bold “how hard can it be?!” statement. And it is, but… But I’ve built, or was co-authoring, such tools in the past. Some were proprietary, some were Open Sourced (see Weasel). And they served me well.

3. And last but not least, the last point. Well, I only had those two above. OK, I can add that I didn’t plan to make it a general-usage tool, just a small one for my needs.

So I did.

And here we’re at the moment when I had to cheat on transactions because of those decisions.

Cloudflare D1

All relational databases seem similar, but only until you start using them extensively, or until you need to write your own storage library. Then you learn stuff that not necessarily the stuff you’d like to spend time on. For instance:

• What’s the difference between databases like PostgreSQL and single-threaded databases like SQLite or DuckDB, and how concurrent processing can be surprising

• or that sqlite3 only calls the first query, but the next silently ignores,

• etc.

Still, well, abstractions like Dumbo give the possibility to massage such cases behind the scenes.

And then Cloud Databases and SaaS Databases came into play, like Cloudlfare D1.

I was motivated by the generous sponsorship from Sam Hatoum to make supporting Emmett and Pongo a top priority. Thanks Sam, appreciate that! And I made it, but I had to cheat a bit.

Databases like Cloudflare D1 and Supabase expose databases as pay-as-you-go services. They optimise deployment, making it highly scalable, so you don’t need to care about it. It’s cost-effective for start-ups and scenarios when you’re not under a huge load. If you are, then you’ll need to pay more, but they also give you autoscaling at least.

They do it by exposing the database API through HTTP API, for instance PostgREST. This gives them easier management around throughput, security, etc., as each call is made as an HTTP request through the exposed API.

Yet, that kills some options like: ekhm, transactions. The challenge with transactions is that they don’t scale (that’s why MongoDB is WebScale). They don’t scale, as you’d need to open a transaction, do some freehand operations, and then commit or rollback. That means (typically) you need to keep a connection open during that time. If you’re building SaaS, it’s a no-go, because sneaky users would open it for a few hours, do crazy stuff, and kill your SaaS resources’ utilisation.

But, boy, aren’t transactions one of the selling points of relational databases? They do, so how to proceed?

Then you need to cheat, as I did.

Repeatable reads, batches, etc.

Cloudflare D1 doesn’t provide transaction data for the reasons I outlined, but it doesn’t leave us without other tools. Those two tools to let me do the smoke and mirrors trick are:

• sessions,

• SQL batches.

What are sessions? Per Cloudflare Docs:

A session encapsulates all the queries from one logical session for your application. For example, a session may correspond to all queries coming from a particular web browser session. All queries within a session read from a database instance which is as up-to-date as your query needs it to be. Sessions API ensures sequential consistency for all queries in a session.

Essentially, that means that we’re getting repeatable reads. So when we’re starting specific sessions, they will be handled sequentially.

And now, the second ingredient: Batches. Per Cloudflare docs

Sends multiple SQL statements inside a single call to the database. This can have a huge performance impact as it reduces latency from network round trips to D1. D1 operates in auto-commit. Our implementation guarantees that each statement in the list will execute and commit, sequentially, non-concurrently.

Batched statements are SQL transactions. If a statement in the sequence fails, then an error is returned for that specific statement, and it aborts or rolls back the entire sequence.

To send batch statements, provide D1Database::batch a list of prepared statements and get the results in the same order.

So Cloudflare didn’t allow us to do the full, freehand transaction, but they allowed us to send multiple statements that internally will be executed as a SQLite Transaction. When the request is handled, it’ll open a transaction, run the statements, and return the results.

Cool, let’s mix this soup together, as having that, I decided to:

1. Fail automatically if someone tries to create a transaction on Cloudflare D1 with an error:

D1 does not support SQL transactions (BEGIN/COMMIT/ROLLBACK/SAVEPOINT). Use { mode: “session_based” } to opt-in to session+batch semantics, or use ‘connection.execute.batchCommand() for atomic multi-statement execution.

Then they get clear information on the first try.

2. Allow users to explicitly open a session-based transaction by providing mode:

const users = await pool.withTransaction(
  async (tx) => {
    await tx.execute.command(
      SQL`INSERT INTO test_users (name) 
      VALUES (${firstUserName}), (${secondUserName})`,
    );

    return tx.execute.query<User>(
      SQL`SELECT *
              FROM test_users 
              WHERE ${SQL.in('id', userIds)}`,
    );
  },  
  { mode: 'session_based' }
);

When they do it, they will need to be aware of the limitations of the tool they have. So that this will internally create a D1 session, and only handle a single batch of operations properly. We’re mimicking the sequential processing by the session-based repeatable reads capability. Still, we need to remember that we won’t be able to roll back changes across multiple statements. Only a single command or batch command is an atomic operation.

We can’t, for instance, run a batch of updates, and fail the whole batch if one update didn’t change any record. The batch will only fail if the database throws an exception. An exception can be in SQLite only called by a table constraint or trigger.

By making this choice to require explicit mode and naming it explicitly, I didn’t manage to cover all cases, but at least made it safe, so people need to learn about this non-default behaviour and be more careful about it. When designing an API, it’s usually better to start with a more strict option and do a bit of “scarification” sometimes.

Still, when I implemented that in Emmett and Pongo, I intentionally used it to enable event appends, but document operations, etc.

If you’d like to try it, you can check Emmett’s or Pongo’s beta versions.

For Pongo, you can install it with:

npm install @event-driven-io/pongo@0.17.0-beta.21

And use it as:

import { d1PongoDriver } from '@event-driven-io/pongo/cloudflare';

const client = pongoClient({
  driver: d1PongoDriver,
  database,
  transactionOptions: { mode: 'session_based' },
});

Or in Emmett by installing:

npm install @event-driven-io/emmett-sqlite@0.43.0-beta.1

And use it as:

import { getSQLiteEventStore } from '@event-driven-io/emmett-sqlite';
import { d1EventStoreDriver } from '@event-driven-io/emmett-sqlite/cloudflare';

const eventStore = getSQLiteEventStore({
  driver: d1EventStoreDriver,
  database,
});

Still, even if you don’t care about Emmett, Pongo, and my Open Source project, I hope that this will give you decent inspiration for your own tradeoffs analysis.

I hope that you learned a bit about how design APIs work, how to check the guarantees of your tools and learn to walkaround them when you have to.

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

In the last article, I explained how to rebuild Event-Driven Read Models in a safe and resilient way. I asked readers to let me know if they find any blind spots in my design.

Well, I found one myself.

This article is about that edge case, but more importantly, it’s about the rabbit hole I went down thinking about how to “fix” it. At the end of this hole, there is a nice learning about dealing with distributed systems. Sometimes the best engineering isn’t about preventing all failures. It’s about recognising your blind spots and making sure they don’t catch you off guard.

Everyone has a plan until they get punched in the mouth

Let me recap the situation described in the previous article. We’re using PostgreSQL to store events and read models. We want to rebuild an inline projection, one that’s applied in the same transaction as the appended event.

In the design from my previous article, during a rebuild, we:

1. Mark the projection as “rebuilding”.

2. Skip inline projections (the rebuild process will catch up anyway).

3. Process all historical events.

4. Mark the projection as “active” again.

The hybrid locking strategy with advisory locks and status checks ensures that inline projections know when to skip and that only one rebuild runs at a time.

Sounds solid. Here’s where it breaks:

The rebuild process checks for new events, sees none (because event 1000 is still in an uncommitted transaction), declares victory, and sets the projection to active. Meanwhile, the append transaction has already decided to skip the inline projection. When it finally commits, the event exists but was never projected. The rebuild process is done, and only new events will update the projection during the next update to the new event append.

We could handwave it with it won’t ever happen!. But, well, under load, this will happen eventually. You might not even notice it, but it will. The timing window is small, but with enough throughput, it becomes a certainty.

The Rabbit Hole of “Fixes”

My first instinct was to engineer my way out of this. Surely with enough clever coordination, we can close this gap?

Let me walk you through the rabbit hole.

Attempt 1: Wait for In-Flight Transactions

PostgreSQL provides pg_snapshot_xmin(pg_current_snapshot()) which tells us the oldest transaction that’s still running. The idea: after processing events, wait until all potentially-skipping transactions have completed before marking the projection as active.

Why it fails: We don’t know what we’re waiting for. While we wait for existing transactions to complete, new transactions start and make their own skip decisions. The target keeps moving. We can never “catch up” because new skips happen while we’re waiting for old ones to become visible.

Attempt 2: Use Transaction IDs as Boundaries

PostgreSQL assigns monotonic transaction IDs to each transaction when it starts. This seems useful—what if we record a “sealing” transaction ID when we’re ready to complete the rebuild, and use it to make decisions?

The logic would be:

When rebuild is ready to complete, record sealing_txid = current_transaction_id

Any event from a transaction with ID lower than the sealing point was “in flight” during rebuild, so it should be handled by async Events from transactions with higher IDs started after sealing, so they can safely use inline projections (the read model will be ready)

Sounds reasonable?

Here’s why it doesn’t work.

The core problem: transaction ID order ≠ commit order.

When a transaction starts, PostgreSQL assigns it the next available transaction ID. But transactions don’t commit in the order they started. A transaction that started earlier (lower ID) might take longer to complete and commit after a transaction that started later (higher ID).

Let’s trace through a concrete scenario:

Here’s what happened:

1. Tx 99 started first and got the lowest transaction ID. It inserted an event and decided to skip the inline projection (status was ‘rebuilding’). But then it got slow—maybe network latency, maybe the application did other work.

2. Tx 100 (the rebuild) started second, recorded sealing_txid = 100, and prepared to complete.

3. Tx 101 started third. It checked: “Is my transaction ID (101) >= sealing_txid (100)?” Yes, so it assumed the read model was ready and processed its inline projection. It committed successfully.

4. Tx 100 marked the projection as active and committed.

5. Tx 99 finally committed. But it had already decided to skip the projection back when status was ‘rebuilding’. That decision was made, the skip happened, and the event is now missing from the read model.

The fundamental issue: we can’t see uncommitted transactions. When Tx 100 set the sealing point, it had no way to know that Tx 99 was still out there, holding an event that would skip projection. Transaction 99 is invisible until it commits, but by then it’s too late.

You might think: “Just wait until all transactions before the sealing point have committed!” PostgreSQL even provides pg_snapshot_xmin(pg_current_snapshot()) which tells you the oldest active transaction. But this leads us back to Attempt 1—while we wait, new transactions start and make their own skip decisions. The target keeps moving.

I wrote about this exact problem in How Postgres sequences issues can impact your messaging guarantees. The same visibility challenges that affect outbox ordering apply here. Transaction IDs are useful for ordering within committed data, but they can’t help us coordinate with transactions that haven’t committed yet.

Attempt 3: Lock Appends During Transition

What if we use advisory locks more aggressively? The idea: when rebuild is ready to complete, acquire an exclusive lock that blocks the entire append path. While holding this lock, flip the status to ‘active’. No appends can be in progress during the flip, so no race condition, right?

Here’s the proposed flow:

• Rebuild finishes processing historical events

• Acquire exclusive advisory lock (all new appends must wait)

• Set status = ‘active’

• Release lock

• Waiting appends resume and sees ‘active’, processes inline normally

It should work. In theory. We’re creating a synchronisation point where nothing can slip through. But there’s a subtle problem: the skip decision was already made inside the append transaction.

An append transaction doesn’t just check the lock and status at one instant. It does several things:

1. BEGIN transaction

2. INSERT the event

3. Try to acquire a shared advisory lock

4. Check projection status

5. Decide: process inline or skip?

6. Execute that decision

7. COMMIT

The decision in step 5 happens inside the transaction. If the transaction saw ‘rebuilding’ status, it was skipped. That decision is now part of the transaction’s pending work. The transaction might be waiting to commit or doing other work, but the skip decision has already been made.

Our exclusive lock in the completion flow blocks step 3 - new transactions can’t acquire the shared lock. But what about transactions that already passed step 5 and are just waiting to commit?

The timing here is tricky. Advisory locks in PostgreSQL can be either:

• Transaction-scoped (pg_advisory_xact_lock): released automatically when transaction commits

• Session-scoped (pg_advisory_lock): held until explicitly released or session ends

If we use transaction-scoped locks for inline projections (which makes sense—you want the lock tied to the transaction lifetime), then the append transaction might have already released its shared lock by the time we try to acquire an exclusive lock. The lock protected the status check, but the transaction is still running with its skip decision already made.

Even if we could perfectly synchronise the lock acquisition, there’s another problem: advisory locks are session-scoped, not durable.

If the connection dies while holding the exclusive lock:

• The lock releases immediately (that’s how advisory locks work)

• We might have partially updated the status

• The system is now in an unknown state

• Other transactions resume with potentially inconsistent data

We can’t build reliable coordination on something that vanishes when connections fail. This is exactly why the original article combined advisory locks with persistent status checks—but that combination doesn’t solve this particular race condition.

TLDR on Attempts

Every “fix” follows the same pattern:

1. Identify a coordination point.

2. Discover there’s a window before that point we can’t see.

3. Try to close that window.

4. Create a new window somewhere else.

5. Repeat.

We’re not solving the problem. We’re relocating it.

So is it a bug in the initial design? Depending how you look on that. I see it as an example of fundamental property of concurrent systems. PostgreSQL’s isolation guarantees mean uncommitted transactions are invisible to other transactions. That’s a feature, not a bug - but it means there’s always a window we can’t see into.

Stop Fighting, Start Tracking

After chasing my tail through various “solutions,” I stepped back and asked a different question.

Instead of: “How do we prevent events from being skipped?” (which requires blocking appends or seeing into uncommitted transactions—both unacceptable because of performance, guarantees etc.)

I started to think on: “How do we know when an event was skipped?” and “How do we ensure skipped events get processed eventually?”

Both of these are solvable.

If we can’t prevent skips from happening, let’s make them visible. When an inline projection skips, it could record that it skipped in the same transaction as the event append.

If the append transaction rolls back, the skip record also rolls back (there’s no event to worry about). If it commits, we have a durable record of exactly what was skipped.

In Emmett, I’m using a dedicated emt_system_messages table rather than reusing the regular emt_messages table or creating a simple “skipped events” table. This might seem like over-engineering—why not just create a simple table with projection ID and event position? Or why not just reuse the existing messages table?

Why a dedicated system messages table?

The regular emt_messages table is for business events—the actual domain events that drive your application. Mixing system-level concerns (like “this projection skipped this event during rebuild”) with business events pollutes the event log and makes it harder to reason about. A simple “skipped_events” table with just (projection_id, event_position) could be enough, but in the long term, it may be hard to maintain. As systems evolve, we’ll have more cases where we might want to skip events. Plus, having system events recorded with all metadata gives us full observability of the internals of our system. We could even make tables for projections and processors’ status, built as read models from system events! Still, let’s hold our horses and get back to our use case.

The system messages table could mirror the structure of regular messages with:

• Global position sequencing (with all the transaction visibility handling from the outbox ordering article)

• Transaction ID tracking for proper visibility checks

• Archiving support via is_archived flag

• Partitioning for performance

If we created a throwaway “skipped events” table, we’d need to solve all these problems again. We’d essentially be building a second event log with the same guarantees.

The skip could be stored as a system message where:

• Stream ID = the projection identifier (name + version), so we can query all skips for a specific projection

• Message data = reference to the original event (sequence ID from the event log)

• Message metadata = processor ID, reason for skip, timestamp

• Message type = indicates this is a “skip during rebuild” system event

Won’t this table grow too much? It may, but we can archive skip events when they’re no longer needed.

When the rebuild processor handles a skipped event, we could archive it. In Emmett, this means setting is_archived = true on the record. The table is partitioned by this flag. Archived records automatically move to a separate partition, which can even be on a different disk drive.

Why archive instead of delete?

• Audit trail: You can see what was skipped and when it was processed

• Debugging: If something goes wrong, you have history to investigate

• Idempotency: If a processor crashes and restarts, it won’t reprocess already - archived skips

Later, we could define retention policies and clean up old archived records. We could do it more aggressively than for business events, since these are operational records, not business history. You might keep business events forever, but archived skip records only need to stick around long enough for debugging (days or weeks, not years).

The same processor that handles regular events for the projection could also process its skipped records. When it reads an event from the main event log and applies the projection, it also checks for and archives any corresponding skip record for that event position. This keeps everything in sync.

As mentioned, this also opens the door for other system events in the future - not just rebuild skips, but potentially failed projections, poison messages, or audit records. The infrastructure would already be in place, separated from the business event log.

The Final Flow

Let me walk through how this works in practice, bringing together all the pieces.

During rebuild

The projection status is ‘rebuilding’. The async rebuild processor works through historical events from the beginning. Meanwhile, normal operations continue. Events are appended, and inline projections check the status. When they see ‘rebuilding’, they skip the projection but record a skip message in emt_system_messages within the same transaction.

When rebuild catches up

The rebuild processor eventually reaches the current position where no new events are visible. At this point, it sets the status to ‘active’. From now on, new appends will process their inline projections normally.

But what about those skip records? The events they reference exist, but their projections were never applied.

Draining the skipped events

The rebuild processor (or a dedicated processor, or a manual trigger—your choice) now queries emt_system_messages for skip records belonging to this projection. Using the same transaction visibility rules from the outbox pattern (transaction_id < pg_snapshot_xmin), it only sees skip records from committed transactions.

For each skip record:

1. Find the referenced event in the event log (or use its data if stored in skipped event)

2. Apply the projection for that event

3. Archive the skip record (set is_archived = true)

The skip record and the event are committed in the same transaction. This gives us a simple invariant: if an event exists, either its projection was applied inline (status was ‘active’) or a skip record exists (status was ‘rebuilding’). There’s no third option where an event exists without a trace.

The drain process might find new skip records appearing. Transactions that were in flight during the status change, committed after we started draining. That’s fine. We keep querying until no more visible skip records exist. They will stop appearing as we alread stopped rebuilding processor, so no more inline projections should be skipped.

The “drain skipped events” phase can be automatically triggered by finishing projection rebuild. It could be handled as the 2nd phase of rebuild processor, or triggering a dedicated one. It could be also just handled by a human initiating the drain.

This flexibility lets user choose the approach based on the specific operational requirements.

The Dead Letter Queue Pattern

What we’ve built is essentially a Dead Letter Queue (DLQ)—a place where messages that couldn’t be processed normally are stored for later handling.

This pattern exists in every serious messaging system:

• Apache Kafka: Dead Letter Topics for messages that fail consumer processing

• RabbitMQ: Dead letter exchanges for rejected or expired messages

• AWS SQS: Redrive policies that move messages to a DLQ after N failures

• Azure Service Bus: Built-in dead-letter sub-queues for each queue

The pattern is universal because all messaging systems face the same fundamental problem: sometimes messages can’t be processed immediately, and you need a place to store them without blocking the rest of the system.

It’s a topic in its own right, as it’s not perfect and should be used cautiously. Many teams fail to apply them correctly. The DLQ becomes like a car alarm in a parking lot, technically signalling a problem, practically ignored by everyone. And that’s what I see in many systems: teams set up DLQs and do nothing about them.

It starts innocently. You configure the DLQ “just in case.” A few messages end up there during a deployment. Someone says, “We’ll look at it later.”

More messages accumulate. The DLQ becomes background noise—a number on a dashboard that nobody checks. Eventually, something critical lands there, and nobody notices until a customer complains.

That’s why in the discussed design, skip records aren’t meant to accumulate indefinitely. The rebuild processor drains them during completion.

Retention policies clean up archived records after a reasonable period. If skip records exist for too long, that’s a signal that something is wrong with the rebuild process - and users should know about it.

A DLQ is only helpful if it’s monitored, processed, and understood why messages end up there. Otherwise, it’s just a fancy way to lose data slowly rather than immediately.

The Broader Lesson

This article isn’t really about PostgreSQL advisory locks or projection rebuilding. It’s about how we approach problems in distributed systems.

When we find a race condition, the instinct is to fix it. Add a lock. Add a check. Add a coordination phase. But each “fix” often just moves the race condition somewhere else. We went through three attempts—waiting for transactions, using transaction ID boundaries, locking appends—and each one failed for a different reason. We weren’t solving the problem; we were relocating it.

At some point, I had to ask myself: am I making this system more reliable, or just more complicated?

The answer came when I changed the question. Instead of asking “how do I prevent events from being skipped?” I asked, “How do I know when an event was skipped, and how do I make sure it gets processed eventually?”

The first question has no good answer, not without blocking appends, which defeats the purpose. The second question is straightforward: record the skip in the same transaction as the event, and process it later.

A system isn’t trustworthy because it never fails. That’s impossible for anything sufficiently complex. A system is trustworthy because you know when it can fail, how it will fail, and how to recover. The skip tracking approach doesn’t prevent failures during the transition period. It makes them visible and recoverable. That’s a stronger guarantee than complex coordination machinery with hidden edge cases.

Sometimes the best engineering decision is to accept what you can’t control and focus on what you can. We can’t control PostgreSQL’s transaction visibility rules. We can’t see into uncommitted transactions. We can’t make the append decision and the rebuild completion atomic without stopping the world.

What we can control is recording skipped items, ensuring those skips are processed, and making the whole thing observable. That’s enough.

The blind spot I found wasn’t really about the specific race condition. It was a reminder that distributed systems have fundamental constraints we can’t engineer around, at least not without trade-offs worse than the original problem.

Transaction isolation and concurrent systems mean we can’t have zero-downtime rebuilds with perfect inline projection consistency and no coordination overhead, all at the same time.

Something has to give.

What we get instead is explicit tracking of what was skipped, guaranteed eventual processing via the system messages table, observability into the transition period, and crash recovery that doesn’t lose data. The read model might be briefly inconsistent during the transition, but we know exactly what’s missing, and we have a clear path to fix it.

That’s the kind of system I can reason about and trust.

If you’re dealing with similar challenges, I’m happy to help through consulting or mentoring. You can also join the discussion in our Emmett Discord—we have a nice community working through these exact problems.

Or check also other related resources:

• Rebuilding Event-Driven Read Models in a safe and resilient way

• How Postgres sequences issues can impact your messaging guarantees

• Guide to Projections and Read Models in Event-Driven Architecture,

• Distributed Locking: A Practical Guide,

• Consumers, projectors, reactors and all that messaging jazz in Emmett,

• How to scale projections in the event-driven systems?,

• Checkpointing the message processing,

• Let’s talk about positions in event stores.

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

Let’s make a soup today: a blog soup. We’ll mix multiple ingredients like:

• events (obviously),

• read models,

• inline and async projections,

• rebuilding read models,

• backfilling new ones with data from existing events,

• scaling async processing horizontally,

• distributed locking,

• PostgreSQL and its Advisory Locks.

Sounds a lot? Well, the soup should be nutritious.

In an event-driven way, after handling business logic, we record new facts and call them events. They gather information about what has happened. That brings many benefits, such as business observability by keeping a log of them. Especially if we’re doing Event Sourcing, we can make the next decision based on them.

Typically, we’re using events in two ways:

• reacting to them, triggering and integrating the steps of our business workflow,

• projecting them, and getting the flattened interpretation of our system state inside read models.

Such processing can happen asynchronously, but doesn’t have to. If we’re using a transactional database (like PostgreSQL, SQLite, or even MongoDB), we can update our read models in the same atomic transaction that stores new events. Such a process is typically called inline projection. Event stores like Emmett and Marten allows that. Still, you can do the same if you’re using outbox pattern, then you can achieve the same without Event Sourcing.

Inline processing is tempting because we’re getting immediate consistency. Yet, there’s no free lunch here. We’re slowing down our event append as we need to process more, and our transactions will be open longer, which can cause deadlocks, etc. We also can’t take advantage of batching event processing.

Also, as Bastian Waidelich rightfully pointed out to me, using inline projections also increases the coupling and fragility of our business logic. If the inline projection fails (due to a bug in its projection logic, a database constraint or other random issue), then we won’t be able to append our event, which is counterintuitive, as why would read model block our business logic (e.g. confirming shopping cart).

My thumb rule is that for single stream, simple projections, I prefer inline projections, but for more complex or workflow processing, I’d go with async.

The big benefit of a durable event log is that we can correct past mistakes and gain more insights from existing data.

How does it look in practice? Let’s say we initially had a basic read model that showed a summary of the specific shopping cart. In TypeScript this could look as follows:

type ShoppingCartSummary = {
  _id?: string;
  productItemsCount: number;
  totalAmount: number;
};

Besides the data, we also need a method that represents how we apply events on top of the existing state to get the next, evolved state.

const evolve = (
  document: ShoppingCartSummary | null,
  event: ProductItemAdded | ProductItemRemoved,
): ShoppingCartSummary => {
  document = document ?? { totalAmount: 0, productItemsCount: 0 };

  switch (type) {
    case 'ProductItemAdded':
      return withAdjustedTotals({
        document,
        productItem: event.data.productItem,
        by: 'adding',
      });
    case 'ProductItemRemoved':
      return withAdjustedTotals({
        document,
        productItem: event.data.productItem,
        by: 'removing',
      });
  }
};

const withAdjustedTotals = (options: {
  document: ShoppingCartSummary;
  productItem: PricedProductItem;
  by: 'adding' | 'removing';
}) => {
  const { document, productItem, by } = options;
  const plusOrMinus = by === 'adding' ? 1 : -1;

  return {
    ...document,
    totalAmount:
      document.totalAmount +
      productItem.unitPrice * productItem.quantity * plusOrMinus,
    productItemsCount:
      document.productItemsCount + productItem.quantity * plusOrMinus,
  };
};

Sounds fine, but well, it may appear that either we, implementing it, or our business, through requirements, forgot about some requirements. Or we didn’t forget anything, but just requirements evolved as they tend to always do.

What if we need to handle now also:

• Show the shopping cart status and show whether the shopping cart is open, confirmed or cancelled.

• not only show totals, but also a list of product items with their details.

Our new read model would look like:

type ShoppingCartSummary = {
  _id?: string;
  productItemsCount: number;
  totalAmount: number;
  status: 'Opened' | 'Confirmed' | 'Cancelled';
  productItems: ProductItem[];
};

type ProductItem = {
  productId: string;
  name: string;
  quantity: number;
  unitPrice: number;
}

And here we should ask ourselves the following questions:

• Is it really the same read model or another one that’ll be used in another place in our UI? Maybe the initial just shows basic data in the menu bar, and this will be used as the summary before confirmation?

• If the read model is the same, then are you fine with downtime where you clean the old data, and reprocess events?

• If it’s the new read model, do we need to backfill it with the old data?

There’s no best practice here; we need to do the drill and be prepared for multiple options.

Add new vs update existing.

In our case, my initial guess would be that this should be a new read model. We need to add significantly more data and new event handling for confirmation and cancellation.

You could say that:

Why add a new read model with similar data? Can’t we just do a subquery?

Of course, you can. I wrote about that more in How to create projections of events for nested object structures?. This can make sense if those read models will always evolve together.

If you need to run multiple views on the same read models, you’re increasing coupling. As such, when you’re rebuilding the read model, then potential downtime will impact both. Also, each time you adjust one view, ensure you haven’t broken the others.

You’re getting a smaller storage size, and potentially don’t need to remember multiple read models, but are doing it just one.

My experience shows that optimising for the end storage until we check that it’s too big isn’t a good driver. Nowadays, storage is cheap, so my default is to keep read models separated, and also not reuse the same evolve logic. A bit of code duplication won’t harm us, but we’ll see benefits as our models evolve. We’ll decrease the cognitive load.

Still, if those models are indeed the same, or we bet they’ll constantly evolve together, then it could be fine to reuse them.

In place update, blue/green rebuild and backfilling data

Ok, I was already using rebuilding word multiple times. But how do we actually do it?

If you’ve read articles on checkpointing or positions in event stores, you already know that each event in the event store/outbox can have its unique, monotonic position. We can subscribe to notifications about new events and process them one by one. That’s how consumers and processors work in Emmett. Once we process a specific event, we can store the checkpoint in the end storage. This enables resilient failover when our processor dies for some reason. When we restart, we’ll read the last processed checkpoint first and start listening for events from that point.

We could reuse this not only for failover but also for rebuilds. Instead of starting processing from the last known checkpoint, we could reset the checkpoint in our database to a specific point (e.g., the beginning of the log). Then we’ll get recorded events again.

And this is the moment we need to decide whether to do an in-place update or a blue/green rebuild.

In-place means that we’re the same storage target. We need to truncate it and then apply data from scratch, typically. The downside is that when we clear it, our read models won’t be up to date with the current state of our event store. We already have newer events recorded, but our read models are empty. We need to fill them in.

The same happens when we create a new read model based on the existing events. When we create a new read model, it won’t magically contain data from existing events.

Typically, we need to spin up a background worker (e.g., Emmett consumer with a projector plugged in) that pulls all events since the beginning and runs projection logic. Obviously, that means we need to run it asynchronously, and depending on the data size, we’ll need to wait until it catches up. During that time, querying for the data will return outdated information - eventual consistency in practice.

That’s why there’s another option. Instead of truncating existing data, we can keep it as it is, or even keep updating it in the old form. Having that, we can build a new read model in parallel. This read model could have a different name, or just a suffix, e.g. V2, V2.0.1, whatever we find helpful.

Then, once this new read model catches up (so it has processed all events, or is close enough to the latest event with a defined threshold), we can switch queries from the old to the new read model storage.

This is actually the preferred way, but it’s a bit more challenging when it comes to dynamically switching the query target. If you’re using Pongo, that’s not that hard, since you just switch the text-based collection name, which is just another table. But if you’re using an ORM, adding a new table dynamically and mapping it can be much more challenging.

Concurrency issues while (re)building read models

To make it harder, eventual consistency is not the only challenge we’ll face. We also need to deal with concurrency and parallel processing.

What should we do when a new event is appended while we’re rebuilding/backfilling the inline projection? If we try to update the end storage in the middle of the processing, we can end up with an inconsistent or erroneous state.

Another issue is what to do if we’re in a Kubernetes-like setup and don’t have full control over the number of instances of the same service? Or what happens if we accidentally spin up multiple rebuild workers?

Then we’re doomed, or at least the consistency of our read model data.

How do we solve it? Let me explain my plan for Emmett.

Distributed Locking and PostgreSQL advisory locks

I encourage you to check my other article: Distributed Locking: A Practical Guide. Yet, don’t worry, I won’t leave you with Read-The-Fucking-Manual type of answer.

Distributed locks are a fundamental tool for coordinating concurrency across systems. We can use a central place, typically scalable on its own, that’ll be used in multiple instances of our service, to ensure that exactly one can request a lock and run specific code.

In the mentioned article, I described multiple options and popular tools for handling that (e.g., Redis, Zookeeper, Kubernetes Replica Sets), as well as PostgreSQL and its Advisory Locks. Let me focus today on the last one and describe how I want to use it in Emmett’s PostgreSQL projection handling.

PostgreSQL gives us two options for coordination.

Row-level locks lock individual table rows. You do SELECT … FOR UPDATE, and anyone else trying to modify that row waits. The lock is tied to the specific row in a table.

BEGIN;
SELECT * FROM some_table WHERE id = @key FOR UPDATE;
-- make changes
COMMIT;

Such locks are straightforward, as we explicitly state what we want to lock. We could define the following table for projections:

CREATE TABLE IF NOT EXISTS emt_projections(
      version                       INT                    NOT NULL DEFAULT 1,  
      type                          VARCHAR(1)             NOT NULL,
      name                          TEXT                   NOT NULL,
      status                        TEXT                   NOT NULL,  
      PRIMARY KEY (name, version)
);

The background worker could lock the projection by a specific name and version and set the status to “rebuilding”. Then, during handling the inline projection, we could use the same lock and check whether the status is “active”; if not, skip processing. Using a lock-in inline projection would also prevent the rebuilding process from starting, as they wouldn’t acquire the lock.

And that’s a nuke option, as it would work but could create a performance problem. If every inline projection needs to grab a row lock on a coordination row, they will all be processed sequentially, one at a time. That’s a throughput killer when you’re appending thousands of events per second. Each of these append would require access to the lock for the specific projection type (e.g. our shopping cart summary), making not only updates but also event appends sequential. That’s not acceptable for most cases.

Advisory locks are different. They’re locks on arbitrary integers. PostgreSQL doesn’t care what the integer means—it just manages who holds the lock. No table rows involved. It also allows two modes: exclusive or shared.

-- Shared lock: many can hold simultaneously
SELECT pg_try_advisory_xact_lock_shared(12345);

-- Exclusive lock: only one holder, blocks shared locks
SELECT pg_advisory_lock(12345);

Those locks are either scoped to an open connection session or an opened transaction (with xact with name) and released automatically once they end.

Shared locks allow multiple sessions to access the lock with the same value. Exclusive lock blocks both other exclusive locks and new shared locks.

This allows a design where shared locks are used for readers, and exclusive locks for writers and maps directly to our problem:

• Inline projections take shared locks as they run concurrently, and just need to check if there’s no async job updating these projections

• Rebuilds take exclusive locks - they block inlines and other instances of async processing.

We just need to do one more thing: since advisory locks take integers, we need to map our projection name and version to them. We can do it by a consistent hash algorithm, either in the application code or in PostgreSQL.

PostgreSQL provides a built-in MD5 hash function. It’s not perfect, as it’s not a sophisticated hash, but it’s fast enough and predictable. In our case, we won’t have thousands of projections in our application, so the risk of a hash collision is negligible. If you’re still worried it’s too high, we could store id in our projections table and use it instead of hash-mapping. Still, if we used md5 function, it could look as follows:

// shared for inline projections
SELECT pg_try_advisory_xact_lock_shared(
        ('x' || substr(md5(?), 1, 16))::bit(64)::bigint
) AS acquired;

// exclusive for inline projections
SELECT pg_try_advisory_xact_lock(
        ('x' || substr(md5(?), 1, 16))::bit(64)::bigint
) AS acquired;

Where as query param, we’d pass the joined projection name and its version.

Thanks to that, multiple inline projections can access the lock if it’s not held exclusively by the async (re)building worker. Thanks to that, we’re not blocking event appends because of the lock on the inline projections.

An exclusive lock can be held only when there’s no single inline projection being applied at the moment.

What’s also cool is that advisory locks can be used with two strategies: fail fast when the lock is held, or wait for the lock to be released. The first option would be useful for inline projections, and the second for rebuilds.

Is that all? Well, not quite.

Why advisory locks alone aren’t enough

Advisory locks have a gap: they’re session-scoped. If the connection holding the lock dies, the lock releases automatically. For most cases, that’s fine and expected, but think through this scenario:

1. We have 1000 events in our event store.

2. Rebuild starts, acquires exclusive lock.

3. It truncates the projection and processes events 1-500.

4. Connection dies (network blip, out of memory kill, whatever).

5. Lock releases automatically.

6. Projection contains data only for events 1-500.

7. Inlines see no lock, start processing.

8. Inline applies event 1001, ending up with a potentially corrupted state (as there may be some events between 500 and 1001 that would impact the state).

Advisory locks can’t persist across connection failures. We need to add something that does.

The hybrid-locking approach

Emmett already maintains tables for tracking processors and projections. The relevant ones:

CREATE TABLE IF NOT EXISTS emt_projections(
    version         INT         NOT NULL DEFAULT 1,  
    type            VARCHAR(1)  NOT NULL,
    name            TEXT        NOT NULL,
    partition       TEXT        NOT NULL DEFAULT 'emt:default',
    kind            TEXT        NOT NULL, 
    status          TEXT        NOT NULL, 
    definition      JSONB       NOT NULL DEFAULT '{}'::jsonb, 
    PRIMARY KEY (name, partition, version)
) PARTITION BY LIST (partition);

CREATE TABLE IF NOT EXISTS emt_processors(
    last_processed_transaction_id XID8    NOT NULL,
    version                       INT     NOT NULL DEFAULT 1,
    processor_id                  TEXT    NOT NULL,
    partition                     TEXT    NOT NULL DEFAULT 'emt:default',
    status                        TEXT    NOT NULL DEFAULT 'stopped', 
    last_processed_checkpoint     TEXT    NOT NULL,    
    processor_instance_id         TEXT    DEFAULT 'emt:unknown',
    PRIMARY KEY (processor_id, partition, version)
) PARTITION BY LIST (partition);

The status column in the projections table can do what advisory locks can’t: persist status between connection crashes. Even if the connection dies, status = ‘rebuilding’ stays in the table. Inlines could check this and skip processing.

The processor’s table tracks checkpoint progress. When a rebuild runs, it updates last_processed_checkpoint as it goes. If it crashes and restarts, it can resume from where it left off rather than starting over.

Potentially, we could reuse the processor’s table, but having a dedicated projection table could also be useful for diagnostics (e.g., storing the projection definition) and for switching queries when a new projection version catches up. We could select the highest active projection version.

Having that, the query checking if the inline projection should be processed or skipped could look as follows:

WITH lock_check AS (
    SELECT pg_try_advisory_xact_lock_shared(
        ('x' || substr(md5($1), 1, 16))::bit(64)::bigint
    ) AS acquired
),
status_check AS (
    SELECT status = 'active' AS is_active
    FROM emt_projections
    WHERE partition = $2 AND name = $3 AND version = $4
)
SELECT
    COALESCE((SELECT acquired FROM lock_check), false) AS acquired,
    COALESCE((SELECT is_active FROM status_check), true) AS is_active;

Two checks, both must pass:

1. Can we get a shared advisory lock? Fails if a rebuild holds exclusive.

2. Is the projection status ‘active’? Fails if it’s ‘rebuilding’.

Then for async (re)building worker:

WITH lock_check AS (
    SELECT pg_try_advisory_lock(
        ('x' || substr(md5($1), 1, 16))::bit(64)::bigint
    ) AS acquired
),
ownership_check AS (
    INSERT INTO emt_processors (
        processor_id,
        partition,
        version,
        processor_instance_id,
        status,
        last_processed_checkpoint,
        last_processed_transaction_id
    )
    VALUES ($2, $3, $4, $5, 'running', '0', '0'::xid8)
    ON CONFLICT (processor_id, partition, version) DO UPDATE
    SET processor_instance_id = $5,
        status = 'running'
    WHERE 
       -- We already own it
       emt_processors.processor_instance_id = $5 
       -- Unclaimed
       OR emt_processors.processor_instance_id = 'emt:unknown' 
       -- Previous instance finished or crashed
       OR emt_processors.status = 'stopped
    RETURNING last_processed_checkpoint
)
SELECT
    COALESCE((SELECT acquired FROM lock_check), false) AS acquired,
    (SELECT last_processed_checkpoint FROM ownership_check) AS checkpoint;

Together Advisory locks prevent the race at the transition point where the connection was closed, and the rebuilding job is restarting. The status check handles crash recovery

The rebuild acquires the exclusive lock before updating the status. It waits for in-flight inlines (which hold shared locks) to finish.

If the checkpoint is null, the UPDATE matched no rows, then another instance owns the processor and is actively running. Back off.

If both succeed, you own the lock and the processor. The checkpoint tells you where to resume from (with a fallback to the beginning). Only then does it flip the status and start async processing.

If the rebuild crashes, the lock is released, but the status remains ‘rebuilding’. Inlines check status and skip.

The scenarios walkthrough

As a single good image speaks more than thousands of words, let me give you some diagrams to summarise how it works.

1. During inline projection while event appends:

• Each inline projection grabs a shared lock before applying,

• Multiple inlines can run concurrently (shared locks are compatible),

• Projection updates happen normally.

2. When a rebuild starts:

• Rebuild grabs an exclusive lock (waits for any in-flight inlines to finish),

• Marks projection as “rebuilding”,

• New inlines see the lock or status and skip,

• Rebuild processes all historical events,

• Marks the projection as “active” and releases the lock,

• Inlines resume.

3. If rebuild crashes:

• Lock releases automatically

• Status stays “rebuilding”

• Inlines keep skipping until another rebuild completes

4. Multiple async processors:

• Each processor tries to acquire the exclusive lock,

• First one wins, others wait or skip,

• Guarantees that only one processor handles a projection at a time.

5. When you add a new projection:

• No status row exists yet,

• Inlines skip automatically,

• First rebuild creates the row and backfills data.

TLDR

Just like making delicious soup, designing robust, fault-tolerant and performant distributed systems is not that easy. Building an event store is not that hard. But only if we exclude the async processing part.

I hope this walkthrough covers both the conceptual and practical aspects of handling projection rebuilds.

We used PostgreSQL and Advisory Locks, as PostgreSQL is cool and is a driving force in Emmett. But all the same principles apply to other tools and storage (with their specifics).

I explained why advisory locks and status columns complement each other:

• Advisory locks handle the fast path (in-memory, no disk I/O for normal operations) and prevent races at transition points (rebuild can’t start until in-flight inlines finish)

• Status column handles crash recovery (persists across connection failures) and new projection bootstrapping (inlines skip until first rebuild completes)

Neither alone is sufficient. Together, they provide the guarantees we need without external infrastructure. Just PostgreSQL doing what PostgreSQL does.

The cost on the hot path is microseconds: one in-memory lock check, one indexed read on a tiny cached table. For most systems, that’s an acceptable tradeoff.

I hope this article also shows you how to use distributed locking in practice.

Please tell me your thoughts and concerns, especially if you see any blind spots in this design! You can do that in our Emmett Discord, come on in, we have a nice community!

If you’re dealing with such issues, I’m happy to help you through consulting or mentoring. Contact me and we’ll find a way to unblock you!

Or check also other related resources:

• Emmett’s Pull Request implementing described approach

• Guide to Projections and Read Models in Event-Driven Architecture,

• Distributed Locking: A Practical Guide,

• Consumers, projectors, reactors and all that messaging jazz in Emmett,

• How to scale projections in the event-driven systems?,

• Checkpointing the message processing,

• Let’s talk about positions in event stores.

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

Who said that LinkedIn notifications are useless? I did, several times. Yet! Today, I opened my computer for the first time since last Tuesday and just saw a notification that Architecture Weekly is 5 years old now!

At this point, you probably know that I’m not the anniversary-type-of-guy (my wife loves that part of me…), but well, that’s a nice milestone!

Architecture Weekly started as a way to organise my readings. I’m a curious guy, and I like to read a lot about different topics. During my career, I worked on a lot of weird projects, which exposed me to many technologies and approaches. It was not always pleasant, but it taught me that technology is a tool, and there’s a big diversity of tools that can lead you to a proper solution.

So, I had always had hundreds of open browser tabs until my Firefox crashed from time to time (yes, I’m a Firefox-type-of-guy). I’m sure you know that too. I tried multiple ways to organise that: tools like Notion, e-mail threads, etc. None of them was perfect. At some point, my friends treated me like a librarian and asked me for interesting links on a topic or what interesting things I’d read recently. I decided to put my readings publicly into a Git repository and use Markdown, so I can easily share them. This repository still exists: https://github.com/oskardudycz/ArchitectureWeekly. At first, just links organised in groups.

I started sharing it with my friends, and they told me it was useful, so I started sharing it with their friends. It seemed so useful to others that I shared it on the other channels, socials, and groups I was part of. Feedback was still positive, and some people started pushing me to create the newsletter, even though they could subscribe to GitHub release e-mail notifications; they still landed in the same notification swamp, as usual, with other notifications. So they asked if I could set up “a proper newsletter”. I was reluctant to do it, but well, as you already know, I did. I used Substack because it helped me avoid the usual accidental complexity by writing my own blogging engine from scratch.

At some point, I decided to try making a paid version of the newsletter. I decided to build a Discord community for paid subscribers and my GitHub Sponsors. I started to also run webinars every month, also inviting guests.

Complete list of webinars we had so far:

• #1 - From CRUD to Event Sourcing

• #2 - Keep your streams short! Or how to model Event-Sourced systems efficiently

• #3 - Implementing Distributed Processes

• #4 - From CRUD to CQRS in Practice

• #5 - Architecture Weekly 100 Edition - Live Q&A

• #6 - Alexey Zimarev - You don’t need an Event Sourcing framework. Or do you?

• #7 - Design and test Event-Driven projections and read models

• #8 - Slim down your aggregates!

• #9 - Radek Maziarka - Modularization with Event Storming Process Level

• #10 - PostgreSQL Superpowers in Practice

• #11 - Maciej “MJ” Jędrzejewski - Evolutionary Architecture: The What. The Why. The How.

• #12 - Jeremy D. Miller: Simplify your architecture with Wolverine

• #13 - Yves Goeleven - The Fantastic 9

• #14 - Mateusz Jendza - Why Verified Credentials is the Future of Digital Identity!

• #15 - Mário Bittencourt: Leveraging BPMN for Seamless Team Collaboration in Software Development

• #16 - Papers We Love #1 - Sagas (Hector Garcia-Molina, Kenneth Salem)

• #17 - Simple patterns for events schema versioning

• #18 - Andrea Magnorsky: Introducing Bytesize Architecture Sessions!

• #19 - Laïla Bougriâ: Debug your thinking

• #20 - Papers We Love #2 - How do committees invent? (Melvin E. Conway)

• #21 - Michael Drogalis: Building the product on your own terms

• #22 - On Performance Testing with Jarosław Pałka

• #23 - Gojko Adzic on designing product development experiments with Lizard Optimization

• #24 - Frontent Architecture, Backend Architecture or just Architecture? With Tomasz Ducin

• #25 - Applying Observability: From Strategy to Practice with Hazel Weakly

• #26 - React Query: A solution for Frontend State Management challenges? With Tomasz Ducin

• #27 - Documenting Event-Driven Architecture with EventCatalog and David Boyne

• #28 - Practical Introduction to Event Sourcing with Emmett

And now, yes, all of them are available for free. You can watch them. It’s around 50 hours of free knowledge. I believe that it is of much better quality than some paid courses. Why did I make them public?

At some point, I realised that I don’t have much fun with organising those links; it was more of a burden, as curating such lists takes a lot of time. I’m also a perfectionist-type-of-guy, so I wanted to ensure that what I recommend is worth the read. It triggers the movement of brain cells. So if you read the article, you might disagree with it, but at least it’s sparking some thought. I also wanted to synthesise them and put my comments. It may seem like a simple thing to provide such articles. But try to keep it at the proper level week after week, and you’ll see it might not be as easy as it seems.

There were not enough paid subscribers to justify the time spent on it, even though I was named a Substack Bestseller. Yes, running a newsletter is not a great business plan. There are some exceptions to that, but even if you’re in the top few per cent, then it might not be enough to even justify the time spent on it.

I decided to try a different angle and made Architecture Weekly a fully paid newsletter. Starting with a deep dive into Database Connection Pooling. I tried to see if I could grind for a while, providing high-quality content that’s not available elsewhere. I think I managed to deliver a lot of articles like that, and the paid subscriber count grew, but it was still not enough to justify my work. I enjoyed writing such articles a lot, and I still do, but it has put more pressure on me. And preparing such content takes time and effort. The effort I could spend elsewhere, e.g., with my family, in better-paid work like my workshops and consulting, or in my OSS projects.

And now, here we are, 10 months later and well, I still delivered a lot of deep dives, but shifted closer to the stuff I do daily, so helping other humans build systems and getting benefit from the Event-Driven approach. As you see, recently I cross-posted articles to my blog and here. I also used the angle of my work on tools like Emmett to explain general concepts.

What’s next? I’ll probably continue doing it as is; I might also consider merging the blog and newsletter into one. I even own domain event-driven.news, so maybe some rebranding will happen. But it’s not yet decided.

I’ll definitely continue sharing what I learned, blogging, but the exact form? We’ll see. I’m open to suggestions! Comment under this post and tell me your thoughts.

What type of content would you like to see here? Or is there some that you’d be even open to paying for?

Can’t promise that I’ll deliver all of them, but it’ll give me a food for thought.

Nevertheless, it’s time for a small celebration!

Thanks for being here with me through this journey!

Cheers!

Oskar

The first one is that I forget, and writing helps me to remember and organise my findings.

The other is that I like to share my journey with the intention of sparing you, my dear reader, some of my struggles.

Last but not least, it gives me the chance to learn from discussions inspired by them. It’s always a chance to meet new perspectives, correct what I did or just trigger some recollection.

After the article about consumer, processors and all that messaging jazz, there was a great discussion on Emmett Discord, there were a lot of interesting threads there, but the one that inspired what you read today came from the Ismael Celis question. Ismael was curious about:

Distributing the workload dynamically between processors

And as that’s something I haven’t planned to deliver initially, but it’s a great question, let me put down some notes on my plan around it.

Why would we want to distribute the workload dynamically? What does that even mean?

Let’s start from the classical approach. We can define projections that will build read models based on the upcoming events. In Emmett, it can look like that when using Pongo as the storage tool:

const cartsSummaryProjection = pongoSingleStreamProjection({
  collectionName: shoppingCartsSummaryCollectionName,
  getDocumentId: (event) => event.data.shoppingCartId,
  evolve,
  canHandle: [’ProductItemAdded’, ‘ShoppingCartConfirmed’],
  initialState: () => ({
    status: ‘pending’,
    productItemsCount: 0,
 }),
})

This means that when the projection handles only ProductItemAdded and ShoppingCartConfirmed event types. It’ll insert or update rows in the table based on the shopping cart id from the event data.

We can plug this projection into a projector (the specific type of processor responsible for running projections).

const cartsSummaryProjector = postgreSQLProjector({ 
  processorId: ‘shoppingCartSummary’,
  projection: cartsSummaryProjection,
 });

And plug it into the consumer:

const consumer = postgreSQLEventStoreConsumer({
  connectionString,
  processors: [
    cartsSummaryProjector
  ],
});

Now, consumers will poll the PostgreSQL event store and pipe filtered events by type into our projector.

That’s simple, we know the filtering criteria, as consumers will know which event types their processor(s) can handle.

Things get harder when we don’t know those criteria upfront. When can that happen? What if we were building an e-commerce SaaS product that lets shop owners buy a subscription to run their shops? Then we won’t know all of our tenants upfront. They will register as they go.

What options do we have for such a multi-tenant setup?

The simplest option would be to add a custom filter that allows filtering events on the consumer by the tenant.

Emmett doesn’t support such a feature yet, but it could (and will at some point). We could extend our event metadata to contain the tenant and spin up a consumer for a dedicated tenant.

We could even wrap it into a dedicated function.

function tenantedConsumer(
  connectionString: string,
  tenant: string,
  processors: MessageProcessor[]) {
  return postgreSQLEventStoreConsumer({
    connectionString,
    processors,
    filterBy: (event) => event.metadata.tenant === tenant1,
    projection,
  });
}

Of course, this wouldn’t be fully dynamic, but we could make it dynamic at the infrastructure level, e.g., by passing the tenant ID from an environment variable and spinning up a new Container (e.g., a Kubernetes Pod).

const consumer = tenantedConsumer(
  process.env.PG_CONNECTION_STRING,
  process.env.TENANT_ID,
  [ cartsSummaryProjector ],
);

We could also start a new job from the API endpoint, running it in the existing deployment; there are plenty of options. You can go wild and think about other ways.

Still, logically, they would look more or less like that. Either you’re scaling horizontally and sharding physically, tenants

Or you’re scaling vertically and running multiple tenants inside your box. Then, instead of spinning up new containers, you’re spinning up new jobs (processes, threads, virtual threads, etc.).

That’s nothing special for Emmett; other tools also do this, e.g., Spring Boot Kafka concurrent listeners.

And hey, btw. there’s one more reason why I’m writing about my Emmett design: to let you benchmark your design against it. Even if you’re not planning to use it, then those are considerations for you either as:

• internal tooling creator,

• user of other OSS tooling, to check how they solved it.

Separating tenants through sharding give you:

• better option to scale and align the needs to the specific tenant workload,

• makes possible full separation in terms of networking, storage, etc.

• can be more costly.

Separating tenants through partitioning the load inside one box:

• is usually cheaper,

• easier to manage,

• doesn’t give you a full separation and can fall into Noisy Neighbour issue.

So there’s no golden rule to choose, it depends on your tooling, needs, etc.

You can, of course, have a mixed solution, so running most of the small tenants in the same box, and the one with bigger security needs gets a special setup. Especially that dynamic split doesn’t have to be only per tenant. You can, e.g., have per region, per product range, per chain, etc.

Also, such a dedicated setup can work for a set number of dynamic options. If you have 1000 tenants, would you spin up 1000 containers or jobs? You can, but just because you can doesn’t mean that you should. The more containers or threads you have, the more you pay for the coordination costs. At some point, it’s better to group processing into a manageable range of containers or threads.

Ok, but how do we know which messages to process where? We could use a consistent hash. I wrote about it in detail in Understanding Kafka’s Consumer Protocol: A Deep Dive into How Consumers Talk to Brokers.

Kafka, by default, partitions its data on the producer side. The topic represents a logical split (e.g. all messages from the E-Commerce module), and the partition represents physical layout. Consumer groups receive messages. Kafka guarantees that precisely one consumer from the consumer group will receive messages from the specific partition. If we have fewer consumers than partitions, then of course, we can get more partitions to handle.

The pseudo code for distributing load to consumers could look as follows:

const partitionId = message.headers.partitionId;
const hash = consistentHashFunction(partitionId);
const consumerId = hash % totalNumberOfConsumersWithinGroup;

In our case, we could use the tenant id as the partition id.

Kafka’s strategy is also simpler than we might need. The partitioning is done on the producer side. The producer sends a message to the topic that already has a certain number of partitions:

const hash =  consistentHashFunction(message.header.recordId);
const partition = hash % totalNumberOfPartitionsWithinTopic;

In our case, if we’d like to allow distribution by any property, we’d need to load the message, read the field we want to partition/shard on, and send it to the appropriate container or thread.

The mechanism can get pretty hefty. You’d need a more sophisticated mechanism to know where to spin up what, how to distribute the load between containers or threads, and to make it resilient. We’re getting into the area of distributed consensus algorithms such as Raft and Paxos.

Do I want to go into this area with Emmett? Definitely not for free! I don’t think that’d be even worth it, as we have mature solutions like Kafka, RabbitMQ, and other messaging systems that implement such algorithms and specialise in that. I’d prefer to make it easier to forward messages to them and let them do the work.

What is definitely on the plate is the second option, which allows partitioned producers. You could define it as such:

const cartsSummaryProjector = postgreSQLProjector({ 
  processorId: ‘shoppingCartSummary’,
  projection: cartsSummaryProjection,
  partitionBy: (event) => event.metadata.tenant,
 });

Then the consumer would poll messages from all tenants, forward them to projectors, and the projectors would internally spin up worker threads per tenant or group them by consistent hashing.

If we add to that distributed locking or detecting conflicting checkpointing to ensure that there’s only one worker instance handling messages for the processor and partition id, then this should be good enough for the majority of cases.

Keeping in mind the flexibility in which you can group (or not) projectors within consumers, you can define your own topologies.

What are your thoughts? How do you deal with such cases? Would you like me to expand more on some cases?

Or maybe you’d like to help me and sponsor my work in this area in Emmett? Then your project could also benefit faster from it!

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

1. What Super Frog has to do with messaging?

2. When was the last time you wrote if statements in SQL? If it’s been a long time, have you at least seen them? If not, (don’t) worry, you’ll see them today.

Will it be a post about weird SQL usage? Not necessarily.

We’ll talk today about checkpointing our processing.

I’ve started my relationship with computers with games. I still have my Amiga 500. In those days, computers didn’t always have a hard disk. You’ve got a bunch of diskettes with different chapters of the game. Not all of them were simple games; many were quite sophisticated, and it took some time to finish them.

Yet they were dealing with limited diskette space, so it was best if they didn’t have to use any of it. How can you then allow you to stop playing and return to the previous state? Or how to not force you to start from the beginning of the game when you fell from the platform, and well, you died? You died in the game, ofc, that at least you should be able to recover, right?

As mentioned earlier, the limited space on diskettes and the additional complexity that came with it led many game makers to adopt a simple solution: checkpoints with codes.

After you passed a level, you got a code you could type when you started the game, and instead of starting from the beginning, you could go directly to the place where you left off. That worked pretty well for the platform and race car games, since your game’s storyline was always the same, immutable. If you had to go to level 27, the starting point and your character would always look the same. Of course, for RPG and strategy games, that’s a different story.

Surprisingly, this parallel also matches the recovery from a business process failure.

Let’s say we’re using message-based communication to streamline and make it more resilient. We don’t want to make it vulnerable to scenarios where we store information in one system, our process dies, and we don’t manage to notify the other parts.

We’re using Outbox pattern to enable that technically. We’re storing messages in the relational table within the same transaction, updating the state after running business logic. Thanks to that, either both states are updated, and the message is scheduled, or none of it is. We’re getting (eventual) consistency thanks to that.

Now we’re on the receiving end, so where we were in the previous article with the explanation of Consumers, Processors and all that jazz.

Let’s say that we’re using PostgreSQL and our Outbox structure looks as explained in the other article:

CREATE TABLE outbox(
   -- the autoincremented position of the message to respect the order
   position        BIGSERIAL                 PRIMARY KEY,
   -- used to detect gaps in numbering
   transaction_id    xid8 NOT NULL,
   -- unique message id, which can be used for deduplication or idempotency
   message_id       VARCHAR(250)             NOT NULL,
   -- the message type, e.g. `TransactionRecorded`
   message_type     VARCHAR(250)             NOT NULL,
   -- serialised message data, e.g. to JSON
   data             JSONB                    NOT NULL,
   -- diagnostic information on when the message was scheduled
   scheduled       TIMESTAMP WITH TIME ZONE  NOT NULL    default (now())
);

As you can see, besides the message ID, type, and data, we’re also storing the (global) position number and transaction ID (to ensure we don’t skip in-flight transactions that have requested the global position number, read more here for reasoning).

Now we can be polling it with the query like:

SELECT 
     position, message_id, message_type, data
FROM
     outbox
WHERE
     (
          (transaction_id = last_processed_transaction_id
               AND position > last_processed_position)
          OR
          (transaction_id > last_processed_transaction_id)
     )
     AND transaction_id < pg_snapshot_xmin(pg_current_snapshot())
ORDER BY
    transaction_id ASC,
    position ASC
LIMIT 100;

Now, thanks to that, we can have the global ordering guarantee on the receiving end. We’re trading a bit of performance for greater correctness. Not always acceptable, but for internal module communication or forwarding to the messaging system, that’s usually more than enough.

That’s also the place where we’re back to our checkpointing. How do we know the last processed position?

By default, that’s simple, we could either say:

• start from the beginning and use -1 or some other hardcoded position,

• start from the end and use the position of the last message in the table plus one.

Those cases can be fine when we need to handle all messages (e.g., adding a new process or reading a model), or when we don’t care about the past and need to process the newest notifications.

Still, the reality is just like in the old games, we’d like to start somewhere in the middle, precisely where we left off.

We need one more table for storing our checkpoints. It can look as follows:

CREATE TABLE processor_checkpoints
(
   -- subscription name
   processor_id          VARCHAR(100) PRIMARY KEY,
   -- information about the position of the last processed message
   last_processed_position  INTEGER      NULL,
   -- used to detect gaps in numbering
   last_processed_transaction_id    xid8 NOT NULL
);

Simple stuff: the processor ID should be unique and reflect the processor’s logical name and the last processed position(s).

If that looks simple, then let me follow up with the next set of potentially simple questions:

• How to store it?

• When to store it?

If your answer is: “just do upsert statement”, then you’re kinda right, but that wouldn’t be simple, it’d be over-simplification. At least if you’d like to run it in production.

Let’s start with how. And, for that, let me bring you now the promised stored procedure with if statements, it’s a bit simplified version from Emmett:

CREATE OR REPLACE FUNCTION store_processor_checkpoint(
  p_processor_id VARCHAR(100),
  p_position BIGINT,
  p_expected_position BIGINT,
  p_transaction_id xid8
) RETURNS INT AS $$
DECLARE
  current_position BIGINT;
BEGIN
  -- Handle the case when p_check_position is provided
  IF p_expected_position IS NOT NULL THEN
      -- Try to update if the position matches p_check_position
      UPDATE processor_checkpoints
      SET 
        “last_processed_position” = p_position, 
        “last_processed_transaction_id” = p_transaction_id
      WHERE “processor_id” = p_processor_id AND “last_processed_position” = p_check_position;

      IF FOUND THEN
          RETURN 1;  -- Successfully updated
      END IF;

      -- Retrieve the current position
      SELECT “last_processed_position” INTO current_position
      FROM processor_checkpoints
      WHERE “processor_id” = p_processor_id;

      -- Return appropriate codes based on current position
      IF current_position = p_position THEN
          RETURN 0;  -- Idempotent check: position already set
      ELSIF current_position > p_expected_position THEN
          RETURN 2;  -- Failure: current position is greater
      ELSE
          RETURN 3;  -- Default failure case for mismatched positions
      END IF;
  END IF;

  -- Handle the case when p_check_position is NULL: Insert if not exists
  BEGIN
      INSERT INTO processor_checkpoints (”processor_id”, “last_processed_position”, “last_processed_transaction_id”)
      VALUES (p_processor_id, p_position, p_transaction_id);
      RETURN 1;  -- Successfully inserted
  EXCEPTION WHEN unique_violation THEN
      -- If insertion failed, it means the row already exists
      SELECT “last_processed_position” INTO current_position
      FROM processor_checkpoints
      WHERE “processor_id” = p_processor_id;

      IF current_position = p_position THEN
          RETURN 0;  -- Idempotent check: position already set
      ELSIF current_position > p_expected_position THEN
          RETURN 2;  -- Insertion failed, row already exists with a greater position
      ELSE
          RETURN 3;  -- Default failure case for mismatched positions
      END IF;
  END;
END;
$$ LANGUAGE plpgsql;

Oooh, even by copy and pasting, I’m already tired; there’s a fair reason why we’re not doing that too often nowadays.

Let me untangle that for you:

1. We’re trying to update the existing position or insert it if we’re storing it for the first time.

2. If all went fine, we’re returning 1 as a result to denote complete success.

3. If we saw that the checkpoint in the database had the same value, we’re returning 0.

4. If we saw that the checkpoint is different from what was expected and further away from it, then we’re returning 2.

5. Otherwise, we’re returning 3, which means that the checkpoint is different from the expected and older.

Essentially, by passing the expected position, we can detect whether we:

• already handled the specific position,

• have some competing instance of our processor handling our data.

That’s why we’re doing this fancy dance with IF statements and a stored procedure.

Detection assumes that we have a global ordering processing guarantee (thus, tricky bits with transaction ID).

It also shows why global ordering is useful.

By detecting that we’ve already handled a specific position, we can skip processing handling idempotency on the processor level.

By detecting that there’s another processor with the same id processing messages, we can make it more resilient and detect the noisy neighbour issue.

How would that look in the code?

async function handleBatch(messageBatch: RecordedMessage[], context: ProcessorContext): Promise<BatchHandlingResult> {
  const { checkpoint } = messageBatch[messageBatch.length - 1].metadata;

  return context.pool.withTransaction(async (transaction) => {
    for (const message of messageBatch) {
      await context.onMessage(message);      
    };

    // No error was thrown: proceed to store checkpoint of the last processed message
    const result = await storeProcessorCheckpoint(transaction.execute, {
      processorId: context.processorId,
      newCheckpoint: checkpoint,
      lastProcessedCheckpoint: context.lastProcessedCheckpoint,
    });

    if(result.success) {
      await transaction.commit();
    } else {
      // no need to do here, either we already handled it
      // or we have a mismatch of expected and existing checkpoints
      await transaction.rollback();
    }

    return result;
  });
}

type ProcessorContext = {
  processorId: string;
  lastProcessedCheckpoint: bigint | null;
  onMessage: (message: AnyMessage) => Promise<void>;
  pool: ConnectionPool;
}

type BatchHandlingResult =
  | {
      success: true;
      newCheckpoint: bigint | null;
 }
  | { success: false; reason: ‘IGNORED’ | ‘FURTHER’ | ‘OLDER’ };

async function storeProcessorCheckpoint(
  execute: SQLExecutor,
  options: {
    processorId: string;
    newCheckpoint: bigint | null
    lastProcessedCheckpoint: bigint | null;
    partition?: string;
 },
): Promise<BatchHandlingResult> {
  const { result } = await single(
      execute.command<{ result: 0 | 1 | 2 | 3}>(
        SQL`SELECT store_processor_checkpoint(
            ${options.processorId}, 
            ${options.newCheckpoint}, 
            ${options.lastProcessedCheckpoint}, 
            pg_current_xact_id()
        ) as result;`,
      ),
    );

    return result === 1
      ? { success: true, newCheckpoint: options.newCheckpoint! }
      : { success: false, reason: result === 0 ? ‘IGNORED’ : result === 2 ? ‘FURTHER’: ‘OLDER’ };
};

As you can see, thanks to:

• global ordering,

• checkpoint detection,

• storing checkpoint where our side effects will be stored,

• transactional capabilities of our end storage,

We can ensure that the entire batch is processed or not. We could even optimise it by storing the checkpoint first and not processing the business logic if there’s a mismatch, then committing only if the logic succeeds.

We’re getting by that generic idempotence check and detection of the noisy neighbour.

Of course, I still believe that idempotence check should happen on the business logic side. But why not both?

Being able to detect a noisy neighbour can help you automatically stop (or pause) one of the competing consumers and avoid inconsistency conflicts.

What are the tradeoffs of this approach?

1. This will work if we have a global ordering guarantee. Not many messaging solution gives us such. If we have a subscription-based outbox as explained, event store like Emmett, Marten or KurrentDB, Kafka, this will work, but not necessarily for solutions like RabbitMQ, SQS, Google Pub Sub, etc.

2. This works best if you have transaction capabilities. Batching updates generally improves performance, but sometimes can lead to long-lived transactions; beware of that. The subscription-based solution with a transaction ID also works best if your transactions are short. If they’re open for a long time, it can cause delays.

3. Still, even without transactions, it will work fine, as long as you’re fine with having retries more often. If the business logic fails and the checkpoint is not committed, it’ll reprocess the already-handled messages from the previously stored messages. Which means that they can be handled more than once, but you should not lose any messages. You may also not fully benefit from the idempotence check for skipping already handled messages.

As always, the choice is yours.

Still, I hope that this article will show you why:

• having a global ordering guarantee can be useful,

• why and how to checkpoint your processing, how they relate to level codes from old games like Super Frog,

• What are the tradeoffs, and how to consider them,

• …and that SQL IF statements are sometimes justified. But don’t go wild with them!

And hey, I also hope that’s not something that you’d like to maintain on your own. There are mature tools to deal with such stuff, like Emmett, which implements this for you.

What are your thoughts? Questions? Concerns?

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

Yet, my final thought was: Kids don’t do it at home.

It’s a fun coding exercise, but using the outcome in production? Not as much fun running and maintaining it. Why though? How hard can it be?

Event Sourcing systems have two phases: appending events and processing them afterwards. The write side gets most of the attention in tutorials and talks - commands, deciders, event stores, optimistic concurrency, as you saw, I’m also one to blame.

Providing the guarantees on the write side is relatively simple, especially if you use a database like PostgreSQL as a storage. You need to provide features like:

• appending an event at the end of the stream,

• reading all events from the stream,

• a guarantee of the ordering within the stream,

• being able to read your writes,

• strong-consistent, atomic writes and optimistic concurrency.

That can be solved with knowledge about transactions, database design, etc. So again, why so hard?

The processing side is where systems often struggle as they grow. This is where the Event Sourcing solution becomes an Event-Driven Messaging tool. And if you’ve read my previous articles, you know that this can be tricky at times.

How do you reliably process events to build read models? How do you trigger side effects without losing messages? How do you scale processing independently from writes? How do you make it performant and run multiple handlers in parallel?

I’ve been working on the message processing architecture in Emmett for a while now. I’ll try to explain how I designed the split between Consumers and Processors, the problems it solves, and the tradeoffs involved.

Why Split Consumers and Processors?

When processing messages, we already know that someone produced them. We’re on the receiving end. Facts are already known; now we need to do something about them.

When we process them, do we care about the source? Typically, we take the information it gathers and reason about it. For instance, when we received an event indicating that a room reservation was made, we may need to send an email with details to the consumer, update the reservations dashboard, and generate a pro forma invoice. We may have specific logic, whether it came from our internal reservation platform or Booking.com, but we know the source from the message payload.

That seems obvious, but it was an important realisation for me. When we’re building a read model in MongoDB, we don’t care if events come from PostgreSQL event store, EventStoreDB, RabbitMQ queue or Kafka topic.

It needs events and the projection logic. Of course, it needs to know the guarantees around: delivery, ordering, idempotency, etc., but besides that? The message’s source doesn’t matter to its logic.

Similarly, a component polling PostgreSQL for messages to publish them doesn’t care what happens to those events - whether they update read models or trigger webhooks is irrelevant to polling logic.

These concerns are orthogonal.

I realised that much of the complexity comes from coupling those two together. We wouldn’t like to change our processing logic because of an internal change in how they’re produced, or vice versa. I concluded that separating them means each can evolve independently. And I came with the initial idea for the split: Consumers and Message Processors.

Consumers are responsible for getting messages from a source and forwarding them to processors. Think of them as the “delivery mechanism.” They handle the “where do events come from” concern. A consumer might connect to:

• A PostgreSQL event store, polling the events table,

• EventStoreDB, using push-based catch-up subscriptions,

• Kafka, consuming from topics,

• Any other message source you might have.

Processors are responsible for doing something meaningful with those messages. They handle the “what do we do with messages” concern. A processor might:

• Update a read model in PostgreSQL or MongoDB,

• Call an external API when certain events occur,

• Publish events to Kafka or send webhooks,

• Trigger workflow steps or saga operations.

This separation follows Unix philosophy: small, focused components connected by simple interfaces. Each piece does one thing well. You can plug any processor into any consumer. This gives you flexibility that matters in practice:

• You can run the same projection logic against different event sources

• Adding new processors doesn’t require changing consumer code

• You can test processors in isolation with fake event streams

• Consumers and processors can be scaled independently

Let me show you how this looks in practice.

How Consumers Work

I deliberately decided to keep Consumers as dumb as possible. A consumer’s entire job is:

1. Connect to a message source,

2. Poll messages in batches or subscribe to notifications (depending on the source specifics).

3. Forward them to all registered processors.

4. Go back to step 2.

That’s it. No business logic. No complex state management. No decision-making about what to do with messages. Consumers are essentially routers.

Why this simplicity? I’m a simple guy; I like clear boundaries for responsibility. They help me reason about both how to use the tool and how to handle it when things go wrong. When message delivery breaks, you want to know exactly where to look. With a simple consumer, the question is binary: did it deliver messages or didn’t it? There’s no complex interaction between delivery logic and processing logic to untangle at 3 AM.

For EventStoreDB, the consumer creates a single subscription and fans out messages to all registered processors. For PostgreSQL, it polls the message table in batches, handling ordering guarantees (we’ll get to why that later).

Here’s what a basic consumer setup looks like for PostgreSQL :

const consumer = postgreSQLEventStoreConsumer({
  connectionString,
  processors: [
    shoppingCartDetailsProjector,
    customerAnalyticsProjector,
    orderNotificationReactor
  ]
});

await consumer.start();

Set up looks accordingly for the other sources providing options specific for the source, e.g. fthe or EventStoreDB store, you may want to provide the category stream name:

const consumer = eventStoreDBEventStoreConsumer({
  connectionString,
  from: { stream: ‘$ce-roomRservations’, options: { resolveLinkTos: true } },
});

The consumer receives messages, batches them and forwards them to all processors. Each processor handles messages independently - they don’t know about each other, and they don’t need to.

Why Batching Belongs to Consumers

Who decides how many messages to fetch at once? That’s a tricky question. I think that batching can happen both at the consumer and processor levels. The consumer decides the batch size for the polling or receiving to tune the receiving throughput. The processor can either align with it, using those batches as a safe default or diverge to its specifics. Read more on Why you should batch message processing in my other article.

Different message sources have different optimal batch sizes. PostgreSQL might be efficient with 100-row fetches. EventStoreDB subscriptions don’t have built-in batching; they deliver events as they arrive. Kafka has its own batching semantics. These are all source-specific optimisations we should be able to apply without ending up with the lowest common denominator.

Processors, by default, can just receive batches and process them. Then they can decide whether to split batches into smaller chunks, group them into even larger chunks, or process them as single messages. For instance, PostgreSQL can handle random single updates pretty well, whereas Elastic prefers batching updates.

How Processors Work

If consumers are simple routers, processors are where the interesting work happens. They’re the smarter ones in this relationship. A processor is responsible for:

• Processing logic: Actually doing something useful with events. Updating a read model, sending an email, calling an API.

• Checkpointing: Tracking which messages have been processed. This is crucial - without it, you’d reprocess everything from the beginning every time you restart.

• Error handling: Deciding what to do when processing fails. Retry? Skip? Stop everything?

• Idempotency: Doing their best to ensure that reprocessing the same event doesn’t cause problems. Of course, still assuming that handlers should be idempotent, read more in my other article,

• Backpressure: They need to be able to tell consumers that they cannot process more messages at the moment, and that the consumer needs to slow down delivery.

Those are general promises and common stuff for the message processing logic. Still, there are multiple reasons why you want to process incoming messages:

• Read models (projections) transform events into queryable state. For instance, shopping cart events - ProductItemAdded, ProductItemRemoved, ShoppingCartConfirmed - need to become a document showing current items, quantities, and totals. Something your API can quickly return when a user opens their cart.

• Reactions trigger side effects after a business fact has happened. When a shopping cart is confirmed, you may want to send a confirmation email, notify the shipment module, and register a new order. These things need to happen, but they’re not part of the core business logic.

• Workflows coordinate multi-step processes across multiple streams. An order might involve payment processing, inventory reservation, and shipping coordination - each with its own state and events.

• Integration means forwarding events to other systems. Other services in your systems might need to know about orders. External partners might need webhook notifications. You might publish to messaging systems for downstream consumers.

All of those processing needs a bit different ways to handle reliability, ordering, throughput, etc. Also, all tools we integrate with require a different approach: storing the read model in PostgreSQL will be quite different from forwarding a message to Kafka.

I wouldn’t like to handwave all of those specifics and end up with the lowest common denominator. That’s why I decided to group them into the following archetypes:

• projectors,

• reactors,

• workflows,

• allow custom message processors to allow people to tune it fully to their needs,

• and in the future, stuff like forwarders, web hooks and others we find useful.

All of them should have a unified API that allows them to be plugged into different consumers, while also embracing differences in message processing and target API specifics.

That’s also why each message processing target (PostgreSQL, EventStoreDB, MongoDB, InMemory, Kafka, SQS, etc.) will have its own implementations.

I believe that this focused responsibility, different archetypes, and specific implementations for different tools will strike the right balance between reusability and avoiding the lowest common denominator. We’ll see if that’s not a famous last words.

Read also more in:

• My RFC for Workflow Processing

• How message pipelines can be technically implemented.

The example projector can look like that:

const projection = pongoSingleStreamProjection({
  collectionName: shoppingCartsSummaryCollectionName,
  evolve,
  canHandle: [’ProductItemAdded’, ‘ShoppingCartConfirmed’],
  initialState: () => ({
    status: ‘pending’,
    productItemsCount: 0,
 }),
});


const postgreSQLProjector = postgreSQLProjector({ projection });

const reactor = postgreSQLReactor({
    processorId: ‘order-notifications’,
    canHandle: [’ShoppingCartConfirmed’],
    eachMessage: (event) => 
      emailService.sendOrderConfirmation(event.data.customerId);    
  }

Native implementations of processors

Different storage requirements require different capabilities, and getting proper guarantees might involve deeper knowledge. For instance, Postgres sequences issues can impact your messaging guarantees. Those are cases where, when you’re starting, you might not anticipate. Test environments may not even catch it; you might realise you’re losing business data when you reach production. That’s why it’s, imho, better to have a tool that solves it rather than trying to maintain it on your own, making technical infrastructure something you need to keep working on instead of your business features. How does Emmett solve them? Let’s discuss them briefly. I’ll try to expand in the future posts about the details.

Resilience

What happens if the processor fails? By default, it stops processing. But only this one, the consumer keeps pushing events to the other processors that can continue. Consumer stops when all their processors are inactive.

Why? Consider this scenario: you have two processors, one updating MongoDB and another updating PostgreSQL. MongoDB becomes temporarily unavailable. Should that stop PostgreSQL updates?

Still, failure behaviour is configurable; your message handler can return:

• void/ACK: message processed successfully, continue to the next one.

• Skip: Skip this message, useful for poison messages that consistently fail

• Stop: Stop this processor entirely.

Why have skip separate from ACK? Consider a poison message - a message that causes your processor to fail every time. Without skip, you have two bad options: fail forever (blocking all processing) or ACK it (pretending you processed it). With Skip, you can move it to a dead-letter queue for investigation while continuing to process other messages.

For now, Emmett doesn’t support Dead Letter/Poison Message Queues out of the box, but they will be supported in the future. You could already append those events to some specific stream.

In upcoming releases, we’ll also have configurable retry policies based on error type and other factors. Just like we already have for command handlers (e.g. to retry 3 times with exponential backoff for Optimistic Concurrency error).

There’s no easy answer to when to stop and when to skip poison messages. Neither choice is universally correct. A financial system might need all-or-nothing semantics. A social media feed can tolerate inconsistency between views.

That’s also why you can freely group processors within consumers. Best if they share similar resiliency and desired throughput characteristics. If they’re very different, you can always spin up another consumer for the same source and process it differently.

Checkpointing processing

In Emmett, processors own their checkpoints. Each processor independently tracks the last message it processed. The consumer doesn’t maintain any checkpoint state.

When a consumer starts up, it asks all registered processors for their last processed position and starts polling from the earliest one.

It has several benefits:

• Independent progress: Processors can move at different speeds. If your MongoDB projector is fast and your analytics processor can get slow at times, they each track their own progress. The slow one doesn’t hold back the fast one.

• Isolated failures: If one processor’s checkpoint storage fails, only that processor is affected. Others continue working.

• Easy replay: To rebuild a single projection, you just reset that processor’s checkpoint. No need to coordinate with other processors or manage a global position.

• Flexibility: Processors can store checkpoints wherever makes sense - in the same database as their read model, in a separate checkpoint table, or anywhere else.

• Capability to redistribute the load. As mentioned in the previous points, if you observe that one of the processors is slower or demands more resources, you can freely deploy it separately in a different consumer, and it’ll start where it left off.

The tradeoff? When a consumer restarts, it might poll events that most processors have already seen. If one processor is significantly behind, all processors receive those events again (they just skip them based on their checkpoints). This is why you should group processors by their typical processing pace - don’t put a real-time dashboard projector and a monthly analytics batch processor on the same consumer.

Backpressure: When Processors Can’t Keep Up

Backpressure occurs when processors can’t process messages fast enough for the consumer to deliver them. This is a real operational concern that needs explicit handling.

Emmett doesn’t support it at the moment, but here’s what I’m thinking about it.

There are several strategies, each with tradeoffs:

1. Ignore backpressure: Consumer keeps polling and pushing regardless of processor state.

• Pro: Simple, maximum throughput when processors can keep up

• Con: Memory grows unbounded, possible OOM, cascading failures

2. Stop on any slowdown: If any processor signals it’s overwhelmed, stop polling.

• Pro: Safe, no resource exhaustion

• Con: Slowest processor determines overall throughput

3. Force synchronised pace: All processors must process each batch before the next is fetched.

• Pro: All processors stay in sync, predictable memory usage

• Con: The Slowest processor becomes the bottleneck for all

4. Slow down ingress: Adaptively reduce polling rate based on processor feedback.

• Pro: Balances throughput and stability

• Con: More complex, needs tuning

5. Rolling buffer: Buffer messages up to a limit, retry delivery to slow processors.

• Pro: Absorbs temporary slowdowns, maximises throughput

• Con: Needs memory limits, complex failure handling

Different systems need different strategies. Real-time dashboards might use strategy 1 (drop messages rather than lag). Financial transactions might use strategy 3 (consistency over throughput). Event forwarding to Kafka might use strategy 5 (buffer temporary network issues).

I’m leaning toward making this configurable per consumer, with sensible defaults. The default would be a bounded buffer with adaptive polling slowdown.

Scaling: Current State and Future Plans

For now, the big benefit of having dumb consumers is that you can scale them horizontally. Of course, this works for offset-based solutions like event stores and streaming tools like Kafka. It may not always work for systems that remove the message once it’s handled. Still, current consumers are using only event stores as sources; Kafka will likely come next.

You can group processors into consumers by that, reducing the number of polling jobs (one consumer polls/subscribes to one source).

I already mentioned batching, which should also increase the throughput.

Running multiple instances of the same processor causes conflicts. Both process the same events, update the same read models, and corrupt the state. Emmett already has the basic capability to do distributed locking, but it’s not fully plugged yet. This will come in future releases.

For now, checkpointing can detect whether a newer checkpoint is already stored (which can suggest another processor is running) and stop processing.

The recommended approach is to run the consumer as a separate service from the API. Then you can scale it separately. You can also set replicas=1 for the specific consumer to ensure one instance.

Rebuilding Projections

Event Sourcing enables rebuilding read models from events. Bug in projection? Fix code, rebuild. New read model? Populate from history.

With processor-owned checkpoints, you can either rebuild read model from scratch by:

1. Stopping the processor.

2. Delete read model data.

3. Reset the checkpoint to the beginning.

4. Restart processing.

Or doing blue greeen by:

1. Creating a new version of your storage (with Pongo, it’s just adding a suffix or prefix to your collection name).

2. Start consumer since the beginning.

3. Check if read models are close enough, and stopthe old processor

4. Start processing.

In Emmett you have even some syntactic sugar on top of consumers and processors to make this easier:

import { rebuildPostgreSQLProjections } from ‘@event-driven-io/emmett-postgresql’;

const rebuilder = rebuildPostgreSQLProjections({
  connectionString,
  projection: shoppingCartsSummaryProjectionV2
});

await rebuilder.start();

This will spin up a new consumer; other consumers and processors continue normally, with their checkpoints unaffected. You can specify the position from which you want to start, and also whether to truncate the end storage.

We’ll need more metrics like gap detection and distributed locking to make it more plug-and-play.

Wrapping Up

The consumer/processor architecture in Emmett is about making event processing concerns explicit and separable:

Consumers handle delivery - getting events from sources to processors. They’re simple by design. When delivery breaks, you know where to look.

Processors handle processing - doing useful things with events. They own their checkpoints, track their own progress, and handle their own failures.

This separation gives you:

• Flexibility to mix and match consumers and processors,

• Independent scaling of different processing workloads,

• Isolated failure domains,

• Easy projection rebuilds,

• Testability at multiple levels.

The design makes tradeoffs explicit:

• Partial progress over all-or-nothing (configurable soon),

• Processor-owned checkpoints over global tracking,

• Simplicity in consumers, complexity in processors,

• Eventual consistency for async operations.

There’s more to build - distributed locking, partitioning, better backpressure handling. There’s still a lot to do, but I believe the foundation is there, and I know real applications are using it already.

I hope that this is a good food for thought, even if you’re not using Emmett. I’m curious about your thoughts and feedback. I’ll try to tackle those cases in more detail in dedicated articles.

If you have questions, feedback, or would like to help me speed up the planned stuff, come chat in the Emmett Discord. We have a small, but welcoming and awesome community.

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

I’m always saying that there’s a thin line between good and bad practice, and this thin line is named “Context”.

That’s also true for the (anti-)pattern I’m calling “Requeuing Roulette”. Let’s discuss it today, continuing the “race condition series”:

• Dealing with Race Conditions in Event-Driven Architecture with Read Models,

• Handling Events Coming in an Unknown Order.

What’s the Requeuing Roulette? As the name suggests, this technique involves putting a message back into the queue. It also (correctly) suggests that we’re hoping for the best. And sometimes we may be lucky to be true.

The basic primitive for a messaging system is a queue. The producer is putting messages into the queue, and the consumer is getting them on the other end. If everything goes well, the consumer receives them in the order the producer put them (thus, a queue, like a queue in a shop).

If the consumer is not available, the messaging system will try to deliver messages and handle retries for us.

We discussed it in detail in:

• Queuing, Backpressure, Single Writer and other useful patterns for managing concurrency

• Ordering, Grouping and Consistency in Messaging systems

• The Order of Things: Why You Can’t Have Both Speed and Ordering in Distributed Systems

• Dealing with Eventual Consistency, and Causal Consistency using Predictable Identifiers

Ordering of processing works if we have a single consumer for a single queue. If we have more than one consumer, we lose the ordering guarantee. Why would we want to have more than one consumer? Obviously, to speed up processing. If messages in the queue are not causally correlated, then we can process them in parallel.

What does the smartass “causally correlated” even mean? For instance:

• depositing money into a bank account is causally correlated to opening it, as we can’t deposit money if we don’t open it.

• depositing money into a bank account is NOT causally correlated to other deposits, as we can deposit as much money as we have (of course, ignoring weird regulations),

• money withdrawal is causally correlated to depositing money and other withdrawals, as we need to check the balance, and they may impact it.

• Withdrawals and deposits are only causally correlated if they happen on the same bank account; other bank account operations can happen at any time.

You get the idea, aye?

So if we set up a queue to process money transfer events, then it could look as follows:

I assumed that we’re following the advice from the previous article. Besides the event type and payload, we’d also pass the record revision, which represents the logical order of events. It comes from the number incremented with each change. Assuming we’re publishing events after each successful business logic handling, it should be gapless.

You may notice that our queue actually has multiple timelines for each causally correlated message sequence. If we simplify our considerations and assume that all events from a certain account are causally correlated, then we could visualise them as:

It’s fine to process events that are not causally correlated, as by that we’re increasing throughput, not trading off correctness:

But it’s not fine to process messages from the same timeline in parallel, as we may get race conditions when the consumer processing the earlier message will be slower than the one processing the later message.

We actually have two orders in messaging systems:

1. Queue Order: The order messages are produced to the queue

2. Processing Order: The order messages are actually consumed and processed.

We already learned that we can detect the out-of-order processing by:

• business rules or,

• detecting the gap between the last processed revision and the event revision.

We can also apply techniques like the Phantom record, where we keep data as it comes and take the next steps only when defined conditions are met. And that’s also what I’d recommend in general, but…

But I promised you to talk today about the Requeuing Roulette (anti)pattern, aye? So let’s do it!

If we’re using tools like RabbitMQ, SQS and other classical messaging tooling (so not you Kafka! You’re a streaming or log solution!), then we can put the message back in the queue.

RabbitMQ message ordering documentation states:

Messages published in one channel, passing through one exchange and one queue and one outgoing channel will be received in the same order that they were sent. RabbitMQ offers stronger guarantees since release 2.7.0.

Messages can be returned to the queue using AMQP methods that feature a requeue parameter (basic.recover, basic.reject and basic.nack), or due to a channel closing while holding unacknowledged messages. Any of these scenarios caused messages to be requeued at the back of the queue for RabbitMQ releases earlier than 2.7.0. From RabbitMQ release 2.7.0, messages are always held in the queue in publication order, even in the presence of requeueing or channel closure.

With release 2.7.0 and later it is still possible for individual consumers to observe messages out of order if the queue has multiple subscribers. This is due to the actions of other subscribers who may requeue messages. From the perspective of the queue the messages are always held in the publication order.

The last paragraph seems promising, as it suggests the message will be put back before the next messages, since it was placed in the queue.

Unfortunately, it’s only a best effort. Another place in the documentation states:

When a message is requeued, it will be placed to its original position in its queue, if possible. If not (due to concurrent deliveries and acknowledgements from other consumers when multiple consumers share a queue), the message will be requeued to a position closer to queue head.

So in the worst case, it can even end up like that:

Then the fun will begin. Now we have two messages that we already see will be handled out of order, and we’ll need to requeue them, hoping they land after the message we need to process as the next one. What if we have multiple messages like that? What if our consumer randomly fails when processing our message with revision 12, and we need to requeue it again? How lucky will it be with Requeuing Roulette?

As you can see, the more correlated our messages are and the more we’d like to parallelise them, the more likely we are to face Requeuing Roulette. Typically, we put messages into the same queue so we can process them in order.

When Requeuing Roulette is helpful?

It can be useful if:

• We want to get the best parallelism for message processing, and ordering best effort is good enough for us.

• Our messages are most of the time not causally correlated or…

• Events for the same records/processes are not quickly published one after another, so we could safely retry message before the next one will arrive,

• Our consumers are stable, not failing too often.

As you see, those assumptions can be fragile and classical famous last words.

Of course, we can use one of the techniques like:

• RabbitMQ routing key, correlation id,

• AWS SQS message group id, visibility timeout,

• Azure Service Bus sessions,

• etc. see Ordering, Grouping and Consistency in Messaging systems for details.

Making this trade-off more in favour of the ordering guarantee or parallelism may reduce it to an acceptable level, but you need to be aware of the risk of an unexpected traffic spike or a different message distribution than you expected.

Dangers of Requeuing Roulette

Even when order doesn’t matter, requeueing has a hidden cost that becomes visible under load.

Suppose you reject the message with requeue set to true. In that case, it can be redelivered to your consumer almost instantly, resulting in a very high workload, since your consumer will reject it again.

Let’s say you have a message that fails because a downstream service is down. You requeue it. It immediately returns to the consumer (maybe even the same one), fails again, and is requeued. This can happen hundreds of times per second. It can also swamp the slow consumer, preventing it from even recovering.

In the worst case, your CPU can be spent processing and requeueing the same 10 messages over and over, while thousands of processable messages sit behind them in the queue (because RabbitMQ will try to put requeued messages before the next messages).

What about Kafka?

Well, in Kafka, this issue doesn’t exist as Messages with the same record key go to the same partition, maintaining order within that partition while allowing parallel processing across partitions.

So Kafka, for the win? Hold your horses!

Only one consumer from the consumer group can handle a specific partition. So, within a single partition, parallelisation isn’t possible. If we map the RabbitMQ queue to Kafka’s partition, then the conclusion can be that Kafka solved it by removing this feature.

Also, when we consume a message from the classical messaging system (like RabbitMQ), it will be removed from the queue. In a streaming solution like Kafka/Pulsar, etc., they will remain in the log until the retention policy kicks in and drops old messages from the partition.

Kafka maintains the offset of the last processed message in each topic partition. You don’t need to requeue messages; you can just rewind the offset to an older position when you want to reprocess messages.

Read more in Kafka Consumers: Under the Hood of Message Processing

TLDR

The “requeueing roulette” is a symptom of trying to solve a distributed systems problem with a technical solution.

The requeueing roulette is seductive because it promises something impossible: maintaining strict order in a distributed, concurrent system without sacrificing throughput. It’s trying to cheat the fundamental trade-offs of distributed systems.

Still, cheating can take us far enough, but there’s always a danger that we’ll be caught and handcuffed.

If you’re considering using Requeuing Roulette, then consider the other techniques I described in previous articles. I’d treat Requeuing Roulette as a temporary solution and a tradeoff.

In my opinion, if you decide to use it, then the question isn’t whether you’ll abandon it, but how much pain you’ll endure before you do.

The real skill isn’t in making requeueing work - it’s in understanding your actual ordering requirements and choosing the most straightforward solution that meets them. Often, that means accepting that perfect ordering is neither necessary nor worth its cost, especially in the long term.

Read also more in:

• Dealing with Race Conditions in Event-Driven Architecture with Read Models

• The Order of Things: Why You Can’t Have Both Speed and Ordering in Distributed Systems,

• Internal and external events, or how to design event-driven API,

• Dealing with Eventual Consistency and Idempotency in MongoDB projections

• Queuing, Backpressure, Single Writer and other useful patterns for managing concurrency

• Ordering, Grouping and Consistency in Messaging systems

• The Order of Things: Why You Can’t Have Both Speed and Ordering in Distributed Systems

• Dealing with Eventual Consistency, and Causal Consistency using Predictable Identifiers.

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.