Architecture Weekly

Vibing, Harness and OODA loop

Oskar Dudycz — Mon, 27 Apr 2026 05:27:27 GMT

Hey, have a look at what I made during the weekend. I had some time, grabbed a beer, turned on the computer and tried to code this feature. If I could do so much during the weekend, how much could you and your team do with it in 2 weeks?

It’s almost a 1:1 quote of what I heard from the startup founder I worked with over 10 years ago. I’m sure that you’ve heard similar phrases from people you worked with. We all know the annoying type of person who doesn’t code anymore but thinks, “I still got it!”. Then they threw a piece of stuff at you to “just fine-tune it a bit and do final touches”. Then they’re the first ones to ask “Why so long?“.

Nowadays, the Internet is full of such people. They shout about what they did with Claude or how much progress LLM tools have made. Some even predict the end of coding. I already wrote that this is wrong perspective. I won’t repeat that, but I want to say that…

Vibing isn’t new and isn’t always an issue.

I’m saying that LLM tools are an appraisal for ignorance. The more ignorant we are of the topic we’re working with, the better we see the outcomes. And that, by itself, is not always bad, as there’s power in ignorance if we focus on getting it done with the simplest tools we have.

Still, this can be terrible if we fall in love too much with what we’ve vibed.

To understand why that “weekend beer” energy is both a superpower and a liability, we need to look at the OODA Loop.

Disclaimer, it’s not a competition for Ralph Wiggum Loop. It’s much older and generic.

Military strategist John Boyd developed the OODA loop (Observe, Orient, Decide, Act) for fighter pilots. In a dogfight, the pilot who cycles through these four stages the fastest and most accurately survives.

In software, the “dogfight” is the gap between your intent and the production-ready feature.

OODA loop is built from four steps:

Observe - This is the intake of raw, unfiltered information. In our world, this means looking at the state of the system.
Orient - This is the most critical and difficult stage. It’s where you filter your observations through your experience, culture, and technical knowledge.
Decide - Based on your orientation, you formulate a hypothesis.
Act - You execute.

Getting back to my favourite founder and LLM-based tools.

The reason founder could build a PoC in a weekend while the team needed more than two weeks is that he bypassed the Observe and Orient phases. He went straight from a vague idea to Act.

If we skip or brush past the observation step, it feels like lightning speed. If the fancy UI grid is there and it does something we wanted, we move on. We’ve outsourced Orientation to our own ego. It’s too easy to assume that because we wrote it, it works.

Observation is the intake of raw data. In a professional environment, our eyes aren’t enough. We need a Harness. If we don’t have automations, tests, integration tests, and pristine traces, we aren’t observing the system; we’re just looking at it. If the inputs are messy, our observation is clouded.

But real engineering, the kind that takes those “two weeks”, is about closing the loop properly. That’s also where we need different perspectives and knowledge sharing.

Orientation is where you process those observations. This is the part where LLMs make us feel smarter than we are. If we don’t understand how a database handles concurrent connections, our “orientation” of a generated script will be shallow. We’ll see code that “looks” right, decide it’s fine, and act by deploying it.

The “I still got it” crowd loves the Decide and Act phases because that’s where the visible progress happens. LLM tools have made these phases nearly instantaneous. We can decide to build a feature and have the code for it in ten seconds.

The problem is that the faster we Act, the faster we need to Observe. If our “Act” phase takes seconds but our “Observe” phase requires a manual weekend of clicking around and drinking beer, our OODA loop is broken. We’re just generating a pile of stuff that we haven’t actually verified.

That’s why the team usually needs more than an imaginary “two weeks”. They are not “fine-tuning” the single-brilliant-dude masterpiece. They are building the infrastructure required to make the OODA loop sustainable.

And to make that possible, they need to run the full loop: Observe, Orient, Decide, Act. And do it multiple times. That takes time, but it’s required to assess the direction, automate what needs to be automated, and ensure they can iterate further and run this loop sustainably. That’s critical for delivering the outcome at the expected pace.

Of course, there’s a danger here, overfocusing on the Orient and Decide can lead to overengineering, building stuff we don’t need. That’s where ignorance can be blissful, especially when we connect it with humility. Being humble about what we don’t know and trying things the easiest way, then learning and making enhancements. Still, humility fails under deadline pressure. The harness doesn’t.

Let me give you…

The example

I’m adding proper Observability and Open Telemetry to Emmett right now. I spent some time working on it and instrumented the first component: Command Handling.

Of course, I had tests to prove it works, but I don’t trust them enough, and I wanted to try it on a real sample, since you never know until you run it. Even the best test suite won’t tell you all.

So I decided to plug it into the sample. See if it works, how ergonomic the API is and how it fits conventions in this area.

To do it, I decided to use Grafana stack and set it up with Docker Compose. So, stable, boring stack. Not going to lie, I vibed the config. Not that there are no docs, but I intentionally wanted to see the typical config people use.

If someone says LLM-based tools are great at proof of concepts, they don’t run the stuff they vibed. If I made the observation based on the initial config, then an oriented decision would be that it won’t work. Of course, then I did the typical back-and-forth, with the LLM tool doing some Linux command Voodoo to make it work. Once. Then, if you try to repeat it, you won’t know how to do it without doing Voodoo again.

Again, that’s not much different from the other stuff we do. I’m sure that you had multiple cases, when someone didn’t use Continuous Deployment tools, but clicked through Azure, AWS, GCP portal, deployed the stack, and then there was no trace on how to set it up again (e.g. to have a different environment for testing or demos for customers).

So, we need a harness, we need a leash to keep our process on track.

How to do the harness? My advice is to start simple. We may ask LLMs to give us shell scripts, and we may ask them to run them multiple times. We also need experience and knowledge of what we want to achieve and the tools we use. It’s fine not to remember all the YAML config to set up the Grafana stack, but it’s not fine not to understand why you even use it, how it relates, and how to set it up.

Still, our first loop can close on the first working solution, even a manually vibed one. But that’s not even a PoC. We need to automate them.

I asked LLM to take notes on what issues it had, and it solved them. Then, based on that, I asked to research how to code it in TypeScript. And to use tools I know, used in past, validating if there are no new more modern ones. For instance, I was a big fan of Gulp.js and Bullseye in the past, but they’re mostly dead. I wanted to have something in the same spirit, using native, maintained tooling.

I ended up with the following tools:

execa for running shell scripts,
native fetch for calling http endpoints,
native Node.js test tools for checking if the stack works as expected.

Then I asked it to create the script to automate the shell Voodoo they did to make Grafana stack and Docker Compose work.

Essentially, it should:

Run Docker Compose script starting up services (Grafana, Prometheus, Loki, Tempo, PostgreSQL, etc.).
Wait for them to check when they’re ready (it usually takes some time).
Start the application and make a request.
Check if the predefined dashboard with Emmett metrics appears, and shows expected traces and metrics.

Initial diagnostic tools looked like that

async function fetchWithDiag(label: string, url: string, init?: RequestInit) {
  const res = await fetch(url, init);
  if (!res.ok) {
    const body = await res.text().catch(() => '(could not read body)');
    console.error(`\n  ✗ ${label} → HTTP ${res.status}\n  body: ${body}\n`);
  }
  return res;
}

async function diagnoseCollector() {
  const text = await fetch(URLS.otelCollectorMetrics)
    .then((r) => r.text())
    .catch(() => 'unreachable');
  const emmett = text
    .split('\n')
    .filter((l) => l.startsWith('emmett_') && !l.startsWith('#'))
    .slice(0, 5);
  console.log(
    emmett.length
      ? `\n  collector /metrics (emmett lines):\n  ${emmett.join('\n  ')}`
      : '\n  collector /metrics: no emmett_* lines found',
  );
}

async function diagnosePrometheus() {
  const json = await fetch(
    `${URLS.prometheus}/api/v1/label/__name__/values`,
  )
    .then((r) => r.json() as Promise<{ data: string[] }>)
    .catch(() => ({ data: [] as string[] }));
  const emmett = json.data.filter((n) => n.startsWith('emmett_'));
  console.log(
    emmett.length
      ? `\n  Prometheus emmett_* metrics: ${emmett.join(', ')}`
      : '\n  Prometheus: no emmett_* metrics found yet',
  );
}

async function diagnoseLoki() {
  const labels = await fetch(`${URLS.loki}/loki/api/v1/labels`)
    .then((r) => r.json() as Promise<{ data?: string[] }>)
    .catch(() => ({ data: [] as string[] }));
  console.log(`\n  Loki labels: ${(labels.data ?? []).join(', ') || '(none)'}`);
}

async function diagnoseDockerLogs(service: string, lines = 10) {
  const { stdout } = await execa('docker', [
    ...COMPOSE,
    'logs',
    '--tail',
    String(lines),
    service,
  ]).catch(() => ({ stdout: '(could not get logs)' }));
  console.log(`\n  docker logs ${service} (last ${lines}):\n  ${stdout.split('\n').join('\n  ')}`);
}

Are they pretty? No. Can they be improved? Yes. Do they have to be improved at this specific moment? No.

The setup uses test infrastructure


const CLEANUP = process.env['CLEANUP'] === '1' || process.env['CLEANUP'] === 'true';
const CLEANUP_AFTER = process.env['CLEANUP_AFTER'] === '1' || process.env['CLEANUP_AFTER'] === 'true';
const NO_START = process.env['NO_START'] === '1' || process.env['NO_START'] === 'true';

// ─── configuration ───────────────────────────────────────────────────────────

const COMPOSE = ['compose', '-f', 'docker-compose.yml', '--profile', 'observability'];

const URLS = {
  app: 'http://localhost:3000',
  prometheus: 'http://localhost:9090',
  tempo: 'http://localhost:3200',
  loki: 'http://localhost:3100',
  grafana: 'http://localhost:3001',
  otelCollectorMetrics: 'http://localhost:8889/metrics',
};

// Fresh client per run — avoids stale cart state from previous runs.
const SERVICE_NAME = 'expressjs-with-postgresql';
const CLIENT_ID = randomUUID();
const CART_ENDPOINT = `${URLS.app}/clients/${CLIENT_ID}/shopping-carts/current/product-items`;
const CONFIRM_ENDPOINT = `${URLS.app}/clients/${CLIENT_ID}/shopping-carts/current/confirm`;

// Matches the .http file — unitPrice is resolved server-side.
const ADD_PRODUCT_BODY = JSON.stringify({ productId: randomUUID(), quantity: 10 });


before(async () => {
  console.log(`\n▶ client ID for this run: ${CLIENT_ID}\n`);

  if (NO_START) {
    console.log('▶ --no-start: skipping docker compose and app startup');
    return;
  }

  if (CLEANUP) {
    console.log('▶ --cleanup: killing port 3000 and tearing down stack (down -v)…');
    await execa('bash', ['-c', 'fuser -k 3000/tcp 2>/dev/null || true']).catch(() => {});
    await new Promise((r) => setTimeout(r, 500));
    await execa('docker', [...COMPOSE, 'down', '-v', '--remove-orphans'], {
      stdio: 'inherit',
    });
  }

  const stackReady = await fetch(`${URLS.prometheus}/-/ready`)
    .then((r) => r.ok)
    .catch(() => false);

  if (stackReady) {
    console.log('▶ observability stack already up — skipping docker compose up');
  } else {
    console.log('▶ starting observability stack…');
    await execa('docker', [...COMPOSE, 'up', '-d'], { stdio: 'inherit' });
  }

  console.log('▶ waiting for backends…');
  await waitFor(() => checkUrl('Prometheus', `${URLS.prometheus}/-/ready`), {
    timeout: 90_000, label: 'Prometheus',
  });
  await waitFor(() => checkUrl('Grafana', `${URLS.grafana}/api/health`), {
    timeout: 90_000, label: 'Grafana',
  });
  await waitFor(() => checkUrl('Tempo', `${URLS.tempo}/ready`), {
    timeout: 90_000, label: 'Tempo',
  });
  await waitFor(() => checkUrl('Loki', `${URLS.loki}/ready`), {
    timeout: 90_000, label: 'Loki',
  });

  // /health returns { status: 'ok', service: 'expressjs-with-postgresql' } —
  // checking service name lets us distinguish our app from other processes on :3000.
  const checkOurApp = () =>
    checkUrl('app /health', `${URLS.app}/health`, async (res) => {
      const json = (await res.json().catch(() => ({}))) as { service?: string };
      if (json.service !== SERVICE_NAME) {
        console.log(
          `    app /health: service="${json.service ?? '(missing)'}", expected="${SERVICE_NAME}"`,
        );
        return false;
      }
      return true;
    });

  const appIsOurs = stackReady && (await checkOurApp());

  if (appIsOurs) {
    console.log('▶ app already running and healthy — skipping npm start');
  } else {
    const portTaken = await fetch(URLS.app).then(() => true).catch(() => false);
    if (portTaken) {
      // Port is occupied but not by our app — stale process or unrelated service.
      console.error(
        '\n  ✗ Port 3000 is occupied by a process that is not this app.\n' +
          '  It may be a stale version of this app (connected to a wiped database)\n' +
          '  or a completely different service.\n' +
          '  Fix: run  npm run verify:observability:cleanup  to kill it and restart,\n' +
          '  or manually free port 3000.\n',
      );
      process.exit(1);
    }

    console.log('▶ starting app…');
    app = execa('npm', ['start'], { stdio: 'inherit' });

    await waitFor(checkOurApp, { timeout: 60_000, label: 'app /health' });
  }

  console.log('▶ setup complete\n');
});

As you see, nothing fancy, the cleanup is even simpler

after(async () => {
  if (app) {
    console.log('\n▶ stopping app…');
    app.kill('SIGTERM');
    await app.catch(() => {});
    console.log('▶ app stopped');
  }

  if (CLEANUP_AFTER) {
    console.log('▶ tearing down stack (down -v)…');
    await execa('docker', [...COMPOSE, 'down', '-v', '--remove-orphans'], {
      stdio: 'inherit',
    });
    console.log('▶ stack torn down');
  } else {
    console.log('▶ stack is still running');
    console.log('▶ to clean up: npm run verify:observability:cleanup');
  }
});

Having that we can run tests:


test('successful command returns x-trace-id header', async () => {
  const res = await fetchWithDiag('POST add product', CART_ENDPOINT, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: ADD_PRODUCT_BODY,
  });

  assert.equal(res.status, 204, `Expected 204 — body logged above`);

  const header = res.headers.get('x-trace-id');
  if (!header) {
    console.error(
      '  ✗ x-trace-id missing — verify the wrapper app in src/index.ts ' +
        'adds it via @opentelemetry/api before mounting the emmett app',
    );
  }
  assert.ok(header, 'x-trace-id header missing');
  assert.match(header, /^[0-9a-f]{32}$/, `"${header}" is not a 32-hex trace ID`);

  traceId = header;
  console.log(`  trace ID: ${traceId}`);
});

test('OTel collector exposes Emmett metrics on port 8889', async () => {
  // Send a few more requests so metrics are definitely recorded.
  for (let i = 0; i < 5; i++) {
    await fetch(CART_ENDPOINT, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: ADD_PRODUCT_BODY,
    });
  }

  try {
    await waitFor(
      async () => {
        let text: string;
        try {
          const res = await fetch(URLS.otelCollectorMetrics);
          text = await res.text();
        } catch {
          console.log('    collector :8889: connection refused');
          return false;
        }
        const emmettLines = text.split('\n').filter((l) => l.startsWith('emmett_') && !l.startsWith('#'));
        if (emmettLines.length === 0) {
          const allFamilies = [...new Set(text.split('\n').filter((l) => !l.startsWith('#') && l).map((l) => l.split('{')[0]))].slice(0, 5);
          console.log(`    collector :8889: no emmett_* metrics yet. Present: ${allFamilies.join(', ') || '(none)'}`);
          return false;
        }
        return true;
      },
      { timeout: 90_000, interval: 5_000, label: 'emmett metrics on collector :8889' },
    );
  } catch (err) {
    await diagnoseCollector();
    await diagnoseDockerLogs('otel-collector');
    throw err;
  }
});

I put it into a single file that can be run as a regular Node.js script.

It already showed me (and Claude) that what they initially did wasn’t working if you try to run it multiple times. It also showed that doing a full cleanup and rebuild, and making it reproducible, needs more work.

Is it done? Not yet; it takes too much time and resources to run it continuously throughout the pipeline. The code is a bit messy, so it needs to be organised. It’s segmented into blocks, includes basic automation and tests, and has already gone through some failures to get it done.

Could I do it better? Sure, and I will improve it, but that’s not the point. I wanted to show you my findings during weekend vibing (without beer tho), the real, not polished iteration, before I run the next one.

The main idea behind OODA loops is not to be perfect, but to iterate quickly, gather feedback as soon as possible, learn from it, develop another theory, and verify it through action.

It’s not about vibing, but it’s also not about analysis paralysis.

I hope you’re now better equipped to think about when vibing, with beer or without, with LLMs or without, actually helps, and when it doesn’t.

Vibe coding is just high-frequency steering. It only works if you have a Harness: a mechanical way to observe and orient, so you don’t steer the whole project into a wall.

Act takes seconds now. Observe takes as long as it always did. Without a harness, you’re not going faster; you’re just making more stuff you haven’t checked.

Harness is not magic, a new discipline, or the next buzzword; I hope I showed you that a bit in this article on what it may look like.

So iterate fast, but wisely remembering to do the full loop. It’s great that LLMs can help us make Acting faster, but we should not skip other steps. We should aim for a fast feedback loop to iterate in the right direction and achieve continuous improvement, to deliver proper value.

Just like Vibing isn’t new, we shouldn’t abandon “old” engineering practices. We should also not replace collaboration with solitary self-high fives.

Check also:

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

Yoda Principle for better integrations

Oskar Dudycz — Mon, 20 Apr 2026 13:17:33 GMT

Try not. Do. Or do not. There is no try!

I’m calling this the Yoda Principle.

Master Yoda said that to Luke Skywalker a long time ago in a galaxy far, far away. He was teaching Luke how to name commands properly while trying to untangle some legacy enterprise mess.

I’m sure you’ve also seen a death star of weirdly-named stuff. Some of them have already tripped you hours of thinking whether someone named this thing badly, or there’s some hidden truth behind it.

Let’s discuss that by the example: E-Commerce order fulfilment.

The order is placed automatically once the customer confirms the items in the shopping cart. We’re not making any product reservations before the shopping cart is confirmed, as this would lock it for other customers, and, as you know, they tend to drop items from their carts.

Once we receive the event notification that the cart has been confirmed, we’ll start the order fulfilment process. It starts (as mentioned) with the order initiation, which acknowledges and initiates the multi-step fulfilment process.

The first step is checking product availability before confirming the order. We need to determine whether we can proceed with completing the shipment and initiating product payment. If the product is unavailable, we need to either ask the customer to wait until we have it again or cancel the order.

We have dedicated modules for order fulfilment and for inventory. Fulfilment is the orchestrator, and inventory is responsible for tracking the state in warehouses.

The ordering module would need to call the inventory module to verify product availability. We could send a command (through the messaging system or web api). How would we name it?

What about VerifyProductExists? We’d send the product id and quantity from the order information, and return true if we have enough products, false otherwise. Sounds fair, right?

Well, it may seem nice at first glance, but what happens if more than one order verifies the same product availability, and we’re running short?

Then we’re vulnerable to race conditions I described in Tell, don’t ask! Or, how to keep an eye on boiling milk. The information we get is only valid at the time of querying. If we don’t lock the product quantity, it can change before we get a response (think: Black Friday-like demand).

Naming our command like VerifyProductExists is a mistake.

VerifyProductExists is not even a command; it’s a query. Command is a request (intention) to run business logic. Query is a request to return data.

Of course, pragmatically, our command can return status information about the result of our operation. But the intention is different.

What’s our real intention here?

The real intention is that we’d like to reserve products so we can ship them and get payment for them, not to verify if they exist. It’d be better to name our command as ReserveProducts or LockProducts.

Why does it matter?

If we’re naming our commands with Verify/Validate/Check prefixes, we’re putting ourselves into the wrong mindset. We’re not focused on actions and integrations, but just brief checking. If we’re in such a mode, it’s easy to handwave the integration complexity.

Locking for shipment may require sending someone to double-check that the product is in the warehouse and hasn’t been stolen, damaged, or other steps. It may be an async process on its own. Still from the ordering module, we should not care, as we’re telling what our intention is and expect to get events informing us whether the reservation succeeded, failed, or timed out (we don’t want to lock products forever in case of order fulfilment issues, but only for some time).

Prefixes like Verify/Validate/Check, etc., are just synonyms for trying. And well, commands are always a form of trying. The handling module can reject the command, as its business rules and state are the source of truth.

We should always assume that the command processing can fail. We should not be discouraged by that, and we should double-check everything. We should not be intimidated by the potential failure, but prepared for it.

We should try not. Do or do not. There is no try.

What if we have both? So VerifyProductExists and LockProducts? It can work if the first one is a query used by the Shopping Cart module, without any guarantee that the data isn’t stale, on a best-effort basis.

If we’re always requiring VerifyProductExists from the handling module before LockProducts, we’re making our communication chatty. I described that in What does Mr Bean opening the car have to do with programming? that this is not only a bad developer experience, but also just redundancy. Locking should already verify whether the product exists, so why require someone to memorise those scenarios instead of checking it internally?

The same goes for cases like:

verify payment has been made,
check if the order wasn’t already fulfilled,
validate if the shipment has been completed,
etc.

All of them either hide a missing business concept or should be a business rule verified within the specific action (e.g., confirming an order).

I recognise this may seem nitpicky, but big things are built on small details.

If we don’t think about such things, we’ll not only end up with misnamed integrations but also fight race conditions and incorrect boundaries.

Then we’ll be doomed.

So better think twice and do or do not.

May the force be with you!

Oskar

Anti-patterns in event modelling - Passive-Aggressive Events

Oskar Dudycz — Mon, 13 Apr 2026 11:00:45 GMT

Have you ever heard phrases like.

Just an update, the milk ran out. Someone finished it and put the empty carton back.

So everyone is aware, the meeting started 15 minutes ago.

Heads up: the coffee machine is empty again.

I’m sure you either heard or used such phrases.

We all know that there’s some hidden intention behind it.

The intention is not to inform, but to trigger a certain action.

Formally, we’re reporting on events to announce the facts, but in practice, we’re using passive-aggressive words. The real intention is to command someone.

We don’t want to inform that the trash bin is full, but we want someone to take it out. We don’t want to inform that the coffee machine requires coffee beans refill, but we want someone to do it.

Passive-aggressive tone is the worst. It’s toxic for both sides of the communication. Usually, it’s just better to ask someone to do it.

The same rule applies in event-driven modelling. We should avoid passive-aggressive communication at all costs.

We should watch out for Passive-Agressive Events. So events that should be commands.

I already warned you in past not to let Event-Driven Architecture buzzwords fool us.

Event-Driven Architecture is an integration architecture style. We’re trying to model our business processes to run smoothly. To achieve that, we prefer a non-blocking communication flow, with things happening in parallel at their own pace. The goal is to achieve autonomous components, reducing the time needed to understand them. That helps maintain, or even replace them as your business evolves.

And events are enablers for that. They notify of what has happened, allowing other components to interpret facts and take the next steps.

But… Let me show one more photo.

It’s parliament, per the official definition: a room full of angry, shouting people.

If we model our communication only in terms of events, our system will look just like that. We’d just announce new facts in a passive-aggressive style and not be interested in what happens next. Oh, wait, are we really not interested? Actually, we are. If someone won’t do what we expect with our information, we’ll be even angrier.

What’s the difference between a command and an event? Both are messages. They convey specific information: a command indicating intent to do something, an event describes what has happened. From the computer’s point of view, they are no different. Only the business logic and the interpretation of the message can distinguish between an event and a command.

And that’s the main difference: commands can be rejected by the command handler. Events can only be ignored.

If we publish an event, we expect one or more consumers to be interested in it. Yet, we don’t know which components will do it. We just broadcast information.

This can easily change into passive-aggressive:

I did my work, now it’s your turn.

And here’s the crucial part. If we always have a single consumer for an event that needs to run the specific logic and expect to get the particular event back, then it should be a command. It’s not an event, we don’t inform. We want some component to take the next specific step and let us know when it’s finished.

Aren’t we making our communication synchronous?

What does it even mean, synchronous or asynchronous?

That’s what Sam Newman discussed in his great talk. The main conclusion is that synchronous vs asynchronous discussion is actually about blocking or non-blocking processing. And that’s much broader topic than the technical solution (so whether we call something in-process via an HTTP endpoint or a messaging system).

It’s a common misconception that events are published asynchronously through a messaging system (e.g. Kafka, RabbitMQ, SQS, WhateverQueue) and commands are sent through synchronous WebAPI. That can be true for a specific solution, but not as the general rule. As said, both events and commands are messages; we can send them through a messaging system or via HTTP (e.g. events via webhooks).

This misleading split came out from our expectation about handling. We expected the command handler to give us the result. For event handler, we don’t expect a specific result. At least in theory.

If we publish a specific event to the messaging system and expect a specific critical path of follow-up events, then we’re not making our communication non-blocking. It’s still sequential. We cannot proceed until the expected sequence occurs.

Whether something is blocking or not is not established by the tools we use, but by how our business process looks.

Speaking about it.

Let’s get back to our favourite E-Commerce Order scenario (read more in Predictable Identifiers: Enabling True Module Autonomy in Distributed Systems).

We could model it so we just publish the OrderConfirmed event and passively-aggressively expect that others will take it from there. So:

The payment module will initiate the payment.
Inventory will start completing shipments.
The notification module will send a confirmation e-mail.
Fraud detection module will check if the order is not rigged.

Once we receive information about a successful shipment or payment registration, we can complete the order.

You may notice two paths for order processing:

Blocking - We need to wait for information about payments and shipments. This is our critical path.
Non-blocking- Order process shouldn’t stop if the notification wasn’t sent or the data warehouse wasn’t able to process events. We’d like that to happen, but it’s expected rather than critical.

Now, both payments may fail (if our customer doesn’t have enough money), and the shipment may not be completed (if it’s Black Friday, and multiple people are competing for the same product).

If that happens, ordering module needs to take action, for instance, do reimbursement if the shipment wasn’t completed, and eventually cancel the order.

If we don’t foresee that and stay in passive-aggressive mode, we tend to forget about “negative” scenarios. it’s too easy to stay in I-Alread-Did-My-Job mode. This will have severe consequences: blocked orders, missed communication, and dissatisfied customers.

We may lear too late that another module can actually say no:

Payment module can say: Man, that’s not going to happen, you’ve already run out of money.
Shipment module can say: Man, I’m sorry, but you weren’t fast enough and we’ve run out of product.

And both of those scenarios will block successful order completion.

How to find such cases? Doing Example Mapping during modelling can be a good option for that.

Most importantly, we need to embrace the fact that some processes require direct, blocking communication, and others don’t. Just like in real life, sometimes it’s just more effective to tell someone to do something. We should avoid micromanagement and aim for autonomy, but not end up with anarchy.

In our case, it’d be better to have a coordinator (workflow, saga, process manager, To-Do List etc.) that publishes the OrderConfirmed event for modules not on the critical path and sends commands like RecordPayment and InitiateShipment.

By that, we’re separating responsibilities and making explicit what should be explicit. This also helps in understanding the business process, as you have a central place to see the critical flow and get proper observability.

Lacking tracing and observability of the business process is one of the most common issues I see in my clients’ projects. As said, if we don’t want to end up with parliament instead of proper communication in our system, we need to be explicit about our intention.

Is that all? Not quite, there’s one more message type we model as events that should not be events.

Gregor Hohpe, in “Enterprise Integration Patterns”, besides Event and Command defines one more message type: Document.

What’s the Document? It’s a state. Or to be precise: self-contained data we have at a certain point in time. We can store it, but we can also publish information about its new value.

That’s probably why Martin Fowler frames it as Event-Carried State Transfer, and I don’t like that term. For me, it’s extremely misleading as, it doesn’t tell what has happened, but what has changed. It just gathers the new version of the state (or the diff).

In my opinion, it’s a variation of State Obsession anti-pattern. Many people fell into that and believe it’s fine to connect the messaging system to the database, use tools like Change Data Capture, and publish it automatically to others. They end up with passive-aggressive communication style:

You have all you need. The whole state is in the events, just interpret it.

How can you reason about what has happened if instead of OrderConfirmed you get OrderCreated, OrderUpdated, OrderDeleted? You’d need to do the diff, compare with previous values, and do the guess about the reason of the specific change.

You deal with Clickbait Events and have a leaking business abstraction. All consumers need to understand the internals of your processing to detect a specific type of change. I wrote about it in detail in Internal and external events, or how to design event-driven API.

Again, the loose coupling of the event-driven processing is only loose for producers; consumers need to adapt. This can lead to hidden coupling, where a change in the producer breaks consumer flows. And that’s the worst type of coupling you can get.

If we’re making commands explicit, we’re also making an explicit relationship between components. It’s no longer flattened to producer <=> consumer, where the producer always shapes the communication. Now, if the other component exposes a command, that’s the driving force behind its behaviour. This helps to shape autonomy. In our case, we could make the Payment Module a generic module with a stable public API for registering payments, and an ordering module that requests them, in accordance with the Shipment Module. Fraud Detection could continue subscribing to events, as it already does. Context Mapping can greatly help in finding those relationships.

TLDR

We tend to be all about events these days, but they’re not the only message types. In our systems, messages take various forms: Events, Commands, and Documents, each serving distinct purposes:

Documents are all about state transitions, which are essential for syncing data across services but missing deeper business insights.
Commands represent a clear intent to act, directed with an expectation of execution, and can be accepted or rejected.
Events are immutable facts, announced without waiting for a response. They’re like broadcasting news, hoping it catches the right ears.

Event-Driven Architectures enable loose coupling, but only for producers. To make consumers loosely coupled, we need to take extra steps, embrace different message types, and have them participate in modelling business processes.

If we go too far with an event-all-the-things communication style, we’ll make our system a room full of shouting people, with a passive-aggressive communication style. Or just aggressive.

In consequence, we won’t know what’s happening in our system, will see only noise, and will have a hard time making it reliable, observable and predictable. We should treat our messages as a communication contract, API and model their flow in a way that shapes our regular communication.

So next time, ask yourself if your event shouldn’t be a command. If it has always had a single consumer and you expect a specific event back, then it’s probably so. It’s all about being clear about the intention, not lying to yourself and others.

I hope this article will equip you with the knowledge to fix that.

If you’re dealing with such issues, I’m happy to help you through consulting, training or mentoring. Contact me and we’ll find a way to unblock you!

See also more in series about event modelling anti-patterns:

Check also more general considerations:

Cheers!

Oskar

The one where Oskar explains Example Mapping

Oskar Dudycz — Mon, 30 Mar 2026 16:02:25 GMT

One of the first indications of getting old(er) is when people stop getting your movie or music references. Of course, based on this rule, some people are always old. That happens.

Recently, I realised during my workshops that referencing Friends is not so cool anymore. It started to happen when I was explaining the Example Mapping technique.

It always starts with “The one where”.
Just like in Friends.

I started to notice a bit slower head nodding and a bit more awkward smiles from the attendees. I repeated

You know, like in titles of Friends episodes.

And head-nodding stopped; only totally awkwardly polite smiles remained. Definitely, it wasn’t The One Where Everybody Finds Out. So I finally asked:

You weren’t watching Friends, didn’t you?

Obviously, the answer was:

Emmm. No…

Okay, then, if you don’t know Friends or the Example Mapping technique, this will be The One Where You Find Out.

Let’s say that we’re working on the guest checkout feature for a hotel management system.

We could start by asking the business how it works. We could get an answer that:

The guest approaches the desk and requests checkout. The clerk inquires about the quality of the products and services, and after receiving an answer, requests the room key. After gathering, the key clerk checks whether the balance is settled. If it’s settled, then proceed with the checkout. Marking the stay as completed.

Sounds straightforward, but we should already have several questions popping up, e.g. what does it mean that “balance is settled”? We could get quick feedback that:

This means that the difference between the sums of all charges and payments is equal to zero.

Then we could try to come up with an example:

Ah, so for instance, when guests haven’t paid upfront for their stay, right?

Right.

Oh, then we need to charge them, right?

Right.

We could visualise what we discovered in the following way:

Now, this generated another flow for us. We have a new feature we weren’t aware of: the guest’s stay payment registration. Let’s try to start this time from the visualisation.

It’s the one Oskar pays for his stay, because he wants to check out but didn’t pay upfront. The payment is registered, and we can try checking out again. Sounds fine, but we should ask whether there are any rules for payments. It may appear that:

Yes, there are some, for instance:
Only guests with a valid credit card can pay with it for their stay,
Guests paying in cash need to hand it over before accepting the payment.

And hey, we just found out new business rules, let’s put them on the board and update our flow to be more precise and reflect our scenario by adding a note that this scenario represents a guest paying with a credit card.

Now, what if the payment fails? Can it fail? Let’s ask the business!

Payment may fail if the payment gateway is unavailable or the issuer rejects it.

What if our internet connection is down for the whole day?

The clerk should ask the guest for cash.

What if the guest doesn’t have cash? 🤔

Then the shift manager can authorise unsettled balance checkout and register a charge with a delayed due date.

Here’s the updated flow. The one where Oskar pays for his entire stay with a credit card, but the Internet is down, and he doesn’t have cash.

Now, we found out:

A new outcome, failed payment,
A new rule, that we need Internet access to authorise credit card payment,
A new feature, the shift manager can authorise unsettled balance checkout and register a charge with a delayed due date.

How would the authorisation look? How should we register a delayed charge?

We should follow double-entry bookkeeping and register the authorisation (type of payment) with the unbalanced amount to settle the balance for today, and register an additional delayed charge for the same amount

The flow will look like:

And that’s precisely how the Example Mapping session looks like. It’s a structured conversation format created by Matt Wynne. You take a user story, gather a small group (usually a developer, tester, and someone from the business side), and spend around 25-30 minutes breaking it down together.

You don’t need a big setup, a huge ceremon, you don’t need sticky notes, you can just use plain text like:

Given: Example

When: We use specific features

Then: Based on business rules, we get a specific outcome.

Business people don’t need to give them to you in such form. You can use the interview as I showed above and note it on your own, while you’re discussing stuff. It’s also a nice way to collaborate and visualise your discussions.

You don’t even need to start with an interview; you can use the Example Mapping as a brainstorming tool to generate as many examples of your (part of the) system. Then, try to model it as you see fit and ask the business for clarifications in the preferred form. It can help facilitate discussion with your team, not only with business stakeholders.

It’s super helpful, as misunderstandings are expensive. When multiple people try to describe the same rule using real examples, you’ll quickly see where assumptions don’t match. Better to find that out in a 30-minute chat than after two weeks of coding the wrong thing.

It also works as a readiness check. Too many red cards? The story isn’t ready. Too many blue cards? The story is probably too big. If examples come easily and everyone nods along - you’re good to go.

There’s also more, as you can try to distil business rules based on the examples and outcomes the business describes.

It’s worth noting that I cheated you in colours. The original looks like this:

Why did I change them?

Example Mapping plays nicely with other collaboration techniques like Event Storming. And if you’re familiar with the Event Storming colour scheme, that’s also the reason why I used it. I aligned them. See:

I’m typically using it during modelling sessions to:

brainstorm (read more in Start Alone, Then Together: Why Software Modelling Needs Solitary Brainstorming),
challenging existing models with real-world examples,
expanding the model with uncovered (through examples) use cases,
finding business rules,
helping with facilitation by looking at the model from a different perspective.

And hot spots and notes, as known from EventStorming, are super helpful here. Read also more in The Underestimated Power of Hot Spots and Notes in EventStorming.

What’s more, if you look at the Given/When/Then pattern, you may notice that it works nicely with Behaviour-Driven Design. I already wrote that Behaviour-Driven Design is more than tests. How to do it? Check here.

I’ll also expand on it in the next articles. I’m doing the extreme Example Mapping with events, so stay tuned, the more will come.

For now, check also those materials:

And most importantly, try it. Take one feature from your system, try to crunch it, or start your design session with this technique.

Most of the teams I’m working with are enjoying this technique, as it’s a fun way to get quick, actionable outcomes.

Cheers!

Oskar

Interactive Rubber Ducking with GenAI

Oskar Dudycz — Mon, 16 Mar 2026 13:26:11 GMT

You may already know that I’m a GenAI sceptic. And a general sceptic.

Do you know that scepticism comes from the Greek σκέπτομαι (skeptomai), meaning ‘to search, to think about, or look for’? So my intention is not to say no to everything new, but more to think about it first, and understand before I say yes.

There’s a lot of stuff about GenAI that makes me smile, but I still understand that my way is my way, and I won’t stop the world. I won’t even try. Thus, I want to research and consider how those tools can help me. I already wrote that I don’t feel like 10x Dev, but I’m finding more ways to get help from it.

One of the ways that helped me is something I call “Interactive Rubber-Ducking”.

Initially, I called it just brainstorming, but that wouldn’t be precise, as I’m not using it to brainstorm ideas, more to challenge and clarify them.

Most of the code I write nowadays is done in my OSS projects. I’m grateful to have a great community with people actively contributing in different ways; still, the canonical design and code work is on my side. As I work in an event-driven niche, I’m often alone with my own thoughts. I try to use the RFC process and discuss it with other fellow humans, but they are not always available. Even if they do, to avoid wasting their time, I need to know what to tell or ask them. I need to give some proposals (with alternatives) to have an effective discussion. I may seem organised, but that’s not always the thing. Sitting in your own head is not a great place to be in general. If you’re a technical leader or an architect, I’m sure that you know that solitude too well.

GenAI tools are not great sparing partners. They’re Yes men. If they read this article, they’d for sure confirm it. They’d probably do it even without reading it. Of course, you can ask them not to be sycophant. You can ask numerous MUSTS with capital letters and bolded NEVER here and there, and it can help, but it won’t fully beat the way they were trained.

And talking to yourself is a similar experience: you’ll find numerous ways to justify your own decisions, and looking at the same place for too long will make you miss obvious blind spots.

Ok, so why would we take those two blind “people” and try to make them help each other?

That’s kinda what “Interactive Rubber-Ducking” is. It takes a blind human with an idea, and another blind not-so-human asking questions. It starts with such a prompt:

Ask me one question at a time so we can develop a thorough, step-by-step spec for this idea. Each question should build on my previous answers, and our end goal is to have a detailed specification I can hand off to a developer. Let’s do this iteratively and dig into every relevant detail. Remember, only one question at a time.
Once we are done, save the spec as spec.md
Before asking another question, store the previous one with the answer in qa.md. Write literally the question and answer, not just a summary.
Here’s the idea:

Don’t treat it as “one magical prompt that will change your life”. Most important is why we’re doing it, what happens next, and who’s actually doing the work. Spoiler alert: it’s not LLM.

I’m using it as a command in Claude Code, and, most importantly, with beefier models like Opus, which can better reason and ask better questions. Doing it with lower-level models always gave me much worse results.

I’m using it with Claude Code, not Claude Chat, because I want the model to scan my codebase. I can ask it to look in certain areas or to reference my answers. I can even ask to search the web or MCPs like Context 7 to check documentation and APIs for popular libraries. Then it’s getting more into brainstorming sometimes, than rubber-ducking, but that’s fine.

As a result, we’ll get two artefacts:

qa.md - with the log of the back-and-forth discussion,
spec.md - in theory spec built by LLM, but imho it’s more of a concise summary.

It may look like Specification-Driven Design, but it’s not.

My goal for this exercise is not to get an actionable specification.

The goal is to get our LLM-based Rubber Duck to ask us hard questions and make us think, not to make the LLM think for us. Find blind spots, and challenge our thinking.

But we’re drivers, we need to know what we want to do, we need to know all the WHYs, and we also need to know HOW. LLM is here to help, but not to do creative work for us. It just pulls it out from our heads.

It also helps to see how our design may be seen by others, especially such mediocre thinkers as LLMs.

I don’t expect the Agent to be able to start implementing the spec. I expect it to reflect all considerations and summarise findings. I’m always double-checking to make sure it includes all the important points. If not, I’ll keep doing Q&A until I’m satisfied.

Having both of those files will allow us to keep a full discussion without losing important details, and a shorter version. We can feed that to another model for review, or try to work on tasks and develop a more detailed plan. Sometimes I plan on my own; for simpler tasks, I may ask the LLM to do it fully. Usually, I’m driving the LLM step by step, passing just specific asks.

The example? Why not.

I recently did such an exercise, trying to narrow down how to introduce the Second-level cache to Pongo and Emmett. What’s Second-level cache? A second-level cache is a local store of data managed by the persistence provider to improve application performance.

Why do I want to introduce it? Because I got an issue from the user that rebuilding projections with a lot events can take too long. One of the reasons is that applying an event on a projection takes:

loading the current state,
updating it,
storing the result.

Those are two operations per event. If we have a batch of 100 events, that’d mean 200 operations; for 1000 events, this would be 2000, so the classical N+1 problem. We could do it differently, and within a batch:

group events by the target documents,
load all of them in one operation by finding all documents within an array of ids (taken from events),
caching them,
applying events in memory,
storing updated documents in one operation.

Then we’ll get at worst 102 operations for 100 events and 1002 for 1000 events, so a linear increase.

Still, I decided to add batching and introduce 2nd-level cache as a solution. I and my experience were a driving force behind that. I could, of course, rubber duck it, but I had that already planned, not just detailed. I also did some initial research before doing the session.

Let me show you the result below. Not too bad if you ask me. Of course, the task wasn’t very innovative; it’s a standard way for mature solutions. Still questions were surprisingly good and helped me.

I hope this log will show you (besides how bad I’m at typos) some insight into how I think, talk, and consider the design.

Tell me your thoughts, if that matches your way, if you haven’t done it yet, try it and tell me how it went!

Interactive Rubber Ducking example

My idea

I’d like to provide 2nd-level cache to Pongo. I want to either use keyv (https://www.npmjs.com/package/keyv) or lru-cache (https://www.npmjs.com/package/lru-cache) packages.

For now, I’d like to do it by id.

I probably need to either extend pongoCollection src/packages/pongo/src/core/collection/pongoCollection.ts with findeOneById and findManyByIds methods, or check if _id is passed in and then try to access the cache.

Cache should be set up either on the pongo collection, pongo db or pongoClient.

Optimistic Concurrency errors should invalidate cache (as that means someone else updated the record in db, e.g. through a different node).

InsertMany and insertOne should put it into the cache.

I’d also like to expand the handle method to allow passing either an id or an array of IDs to support batch loading of entities and improve performance, and to support batch update/insert/delete.

Thoughts?

Which defaults should I set? (Of course, allowing users to override it).

Check the related code I pasted. I want to keep it aligned and as non-invasive as possible.

Q1: Cache library choice — lru-cache, keyv, or custom interface?

Question: You mentioned both keyv and lru-cache. They pull in different directions. Which trade-off do you prefer?

A) Start with lru-cache, optionally let users swap adapters later.
B) Start with keyv so cross-node sharing works out of the box.
C) Define a thin cache interface (get/set/delete/clear) and ship lru-cache as the default adapter.

Answer: Custom cache interface (option C). No dependency on external APIs — keep it pluggable. Start with an in-memory provider only. Considered writing a custom LRU with me, but wants to ship fast. Key point: don’t stick to any third-party API surface, allow pluggability.

Q2: Where should the cache live in Pongo’s hierarchy?

Question: Three attachment points: A) PongoClient level (shared across all), B) PongoDb level (per database), C) PongoCollection level (per collection). Or configurable at multiple levels?

Answer: Allow defining it at different levels — cascading configuration. Set a default higher up, override per collection if needed.

Q3: Inheritance/override behavior for cascading cache config?

Question: For the cascading config — what should the inheritance/override behavior be?

Two sub-questions:

Should a lower level be able to disable caching even if a parent enables it?
Should cache settings (like max size, TTL) also cascade, or only the cache provider instance itself?

Answer:

Yes. Cache should be an object with type, general params and specific for this type, or 'disabled'. That’s a proper devex.
Yes, as above. But if one selects a different cache type then general params should be inherited but type-specific should not be inherited and should fallback to defaults.

Claude’s thoughts: The CacheConfig | 'disabled' pattern is clean — discriminated unions play nicely with TypeScript. The inheritance rule makes sense: general params (TTL, max size) cascade, but type-specific params reset to defaults when you switch providers, avoiding leaking nonsensical config (e.g., a Redis connection string bleeding into an in-memory cache). Also worth distinguishing between “I didn’t specify anything” (inherit from parent via undefined) and “I explicitly want defaults” (reset) — undefined vs 'disabled' vs CacheConfig gives a clean three-state: inherit / off / explicit.

Q4: Cache interface shape — async, batch ops, scoping?

Question: Settling the cache interface details:

Async or sync? Making the interface Promise-based from the start keeps it pluggable. Sound right?
Batch methods (getMany, setMany, deleteMany) — should the cache interface support batch ops natively, or loop internally?
Should clear() be scoped per collection, or always clear the entire cache?

Answer:

Maybe we could return either promise or no promise, then if we await it, it would support both? Also ensure that settings are close to what lru-cache and keyv are having.
Having them (even as syntactic sugar) would be a good devex.
We could pass down not only settings, but also the cache provider instance, then if settings are the same we can reuse cache without recreating it. If set had ttl provided, then we could do it by overriding those TTLs on item level. ClearAll probably would be something we should expose only for advanced, typically user should not be aware of cache.

Claude’s thoughts: MaybePromise (i.e. T | PromiseLike) works — await on a non-Promise just resolves immediately. No overhead for sync providers, no breaking change for async ones. Passing a cache instance is smart: config becomes either settings (Pongo creates the cache) or a pre-built instance (user brings their own). Multiple collections can explicitly share one cache instance. clear() should be internal/advanced — scope by collection prefix internally so a collection-level clear doesn’t nuke another collection’s entries when sharing an instance. For aligning with lru-cache/keyv settings: both support max (max entries) and ttl (TTL in ms). Start with max and ttl as the general params.

Q5: Cache key strategy

Question: Since cache instances might be shared across collections, how to avoid key collisions?

A) Pongo manages key prefixing internally ({dbName}:{collectionName}:{id}). The cache interface works with plain string keys, namespacing is Pongo’s concern.
B) Key is just the document _id. Simpler but only safe with per-collection instances.

Answer: A, definitely.

Q6: Cache invalidation and consistency with optimistic concurrency

Question: How should cache interact with different operations?

Successful writes → update/evict cache. Clear.
Concurrency errors → evict stale entry. Clear.
Reads with _id filter → cache hit or miss-then-populate. Clear.
handle method — should the read inside handle use the cache (risking more concurrency errors on stale data) or always go to DB?

Answer: Definitely use cache. We could consider disabling cache through handle options.

Claude’s thoughts: Use cache by default — optimistic concurrency is the safety net. If cached version is stale, write fails, we evict, caller can retry. Adding skipCache?: boolean to HandleOptions is a clean escape hatch for high-contention cases.

Q7: Should `skipCache` be available on read methods too?

Question: Should skipCache also be available on findOne and other read methods for symmetry, or keep it handle-only?

Answer: Good point, having skipCache on find methods (also findMany etc.) would be good.

Claude’s thoughts: Consistent API surface. Add skipCache?: boolean on the shared options type (e.g. CollectionOperationOptions) so it’s available everywhere without repeating yourself.

Q8: Batch operations on `handle` — signature and handler shape

Question: You want handle to accept an id or array of ids. Two design choices:

Overloaded signature (handle(id: string | string[], ...)) vs separate method (handleMany)?
Handler shape for batch: A) still receives/returns one document at a time (Pongo batches DB ops internally) or B) receives/returns full array?

Also noted: since PongoDocuments always have _version, if someone provides expectedVersion we could compare it against the cached version and decide if it’s stale without hitting the DB.

Answer: Option A for handler shape. handleMany would be consistent with other APIs, but single id or array (string | string[]) is clean enough.

Claude’s thoughts: Version comparison in cache is a cheap short-circuit — if cached _version doesn’t match expectedVersion, we know it’s stale without a round trip. On signature: TypeScript overloads can narrow the return type (string → single result, string[] → array). handle is a higher-level orchestration method, not direct CRUD, so a union input feels natural. Go with handle(id: string | string[], handler, options?).

Q9: Defaults — max entries, TTL, enabled by default?

Question: What defaults to set? Proposed: max: 1000, ttl: 60_000 (1 min), and should cache be enabled by default or opt-in?

Answer: Stick to recommended lru-cache defaults for now. Enable cache by default.

Claude’s thoughts: Enabling by default works because: (1) optimistic concurrency is always the safety net, (2) it’s document-by-id only so invalidation is straightforward, (3) users can set 'disabled'. lru-cache requires max to be set (no default) and has no default TTL. Suggested: max: 1000, no TTL (rely on LRU eviction + invalidation on writes/concurrency errors) — most lru-cache-aligned.

Q10: TTL or no TTL as staleness backstop?

Question: No TTL means cached entries live until LRU-evicted or invalidated by a Pongo write. If something updates the DB outside this Pongo instance (another service, direct SQL), the cache won’t know. Options:

A) No TTL, accept the trade-off — OC catches it on next write.
B) Set a conservative default TTL (e.g., 5 minutes) as a staleness bound.

Answer: Fine with using TTL and sometimes hitting OC issue. OC assumes rare conflicts.

Q11: Where does cache integration hook into existing code?

Question: Two approaches for minimal invasiveness:

Wrap at the pongoCollection level — intercept operations inside the factory function, driver/SQL layers untouched.
Decorator/wrapper pattern — cachedPongoCollection wraps PongoCollection, adds cache logic around delegation.

Answer: Leaning more to option 1. Thinks it’ll be easier to make observability precise eventually.

Claude’s assessment: Agreed with option 1. Cache decisions depend on internal state (whether filter targets _id, document _version, write success vs concurrency error). A wrapper would need to re-inspect all of that from the outside. For observability, cache hit/miss logic next to the actual DB call enables precise metrics in context. Pongo’s factory functions already compose behavior internally (transactions, schema migration) — cache is another concern at the same level. A decorator would duplicate filter-inspection logic and add indirection that doesn’t carry its weight.

Q12: Cache interaction with transactions

Question: How should cache interact with transactions?

Should writes within a transaction update the cache immediately, or only after commit?
Should reads within a transaction check the cache?

Answer:

After commit.
We should use cache, but allow skipping it on transaction or session options. We can also allow setting cache settings per session.

Claude’s thoughts: Reads within a transaction benefiting from cache is more aggressive but logical — most transactions are short-lived. The narrow case: insertOne then findOne same _id within a transaction — cache won’t have the uncommitted insert, falls through to DB, gets the right answer anyway. No correctness issue, just no perf benefit for that case. Per-session cache settings fit the cascading pattern (e.g., startSession({ cache: 'disabled' }) for batch migrations). Proposed cascade priority: client → db → collection → session → per-operation skipCache.

Q13: Cascade priority — does session override collection?

Question: Does the cascade priority client → db → collection → session → per-operation skipCache feel right? Session overriding collection is the non-obvious part — the argument being that session is the most immediate runtime context, while collection is structural config. Or would you prefer collection always wins over session?

Answer: That’s fine. Session should precede collection, as normally you create session to e.g. share connection, and join the number of operations logically, so it’d make sense actually to also do it to e.g. override settings like caching.

Claude’s thoughts: Session as logical grouping of operations with shared context (connection, transaction) — overriding cache settings at that level is extending the same concept. Bulk import in a session? Set cache disabled once on the session instead of skipCache on every call. Final cascade: client → db → collection → session → per-operation.

Q14: Cache provider error handling

Question: If the cache provider throws (e.g., Redis connection drops), should Pongo:

A) Treat it as a cache miss — fall through to DB silently, maybe log a warning. Cache failures never break the application.
B) Propagate the error — let the caller deal with it.

Answer: A — yes, when we add observability to Pongo, we need to make it also a metric and trace event/attribute/log entry.

Claude’s thoughts: Cache is an optimization, not a correctness requirement. Silent degradation now, observable degradation when the observability layer lands.

Q15: Cache warming, events/hooks, and delete operations

Question: Three remaining areas:

Cache warming — should Pongo support pre-populating the cache on startup?
Cache events/hooks — should users be able to listen to cache hits/misses/evictions?
Delete operations — deleteOne/deleteMany should evict from cache. Anything non-obvious there?

Answer:

For now keep it as a future concept, or define pluggable strategy for users.
You can add the basic hook to the Pongo Cache wrapper. Shouldn’t be that hard to just call some function.
Yes, they should evict if they can, we can delete (for now) when deleteMany deletes by ids.

Claude’s thoughts: Cache warming deferred — can add a strategy hook later. Basic event callbacks (onHit, onMiss, onEvict, onError) on the cache wrapper are low-cost and useful for debugging even before full observability. Delete eviction scoped to id-based deletes for now — filter-based deleteMany doesn’t evict (no way to know which ids were affected without querying first).

Interactive Rubber Ducking result

Pongo 2nd Level Cache — Specification

Overview

Add a document-level (by _id) 2nd level cache to Pongo with a pluggable provider interface. Ships with an in-memory provider backed by lru-cache. Enabled by default. Designed to be minimally invasive to existing code.

Cache Interface

`PongoCacheProvider`

A thin, Pongo-owned interface. No coupling to any third-party API.

type MaybePromise = T | PromiseLike;

interface PongoCacheProvider {
  get(key: string): MaybePromise;
  set(key: string, value: PongoDocument, options?: { ttl?: number }): MaybePromise;
  delete(key: string): MaybePromise;
  getMany(keys: string[]): MaybePromise<(PongoDocument | null | undefined)[]>;
  setMany(entries: { key: string; value: PongoDocument; ttl?: number }[]): MaybePromise;
  deleteMany(keys: string[]): MaybePromise;
  clear(): MaybePromise;
}

MaybePromise return types: sync providers (in-memory) return values directly, async providers (Redis) return Promises. await handles both transparently.
Batch methods (getMany, setMany, deleteMany) are first-class. Default in-memory implementation may loop internally, but the interface allows optimized batch ops for external providers.
clear() is internal/advanced — not exposed to typical users. When sharing a cache instance across collections, scoping is handled via key prefixing by Pongo, not by the provider.

Cache key strategy

Pongo manages key prefixing internally: {dbName}:{collectionName}:{documentId}.

The cache provider works with plain string keys — namespacing is Pongo’s concern, not the provider’s.

Event hooks

The Pongo cache wrapper supports basic callbacks:

onHit?(key: string): void
onMiss?(key: string): void
onEvict?(key: string): void
onError?(error: unknown, operation: string): void

These are optional and intended for debugging and future observability integration.

Configuration

`CacheConfig`

type CacheConfig = {
 type: string;               // e.g., 'in-memory', 'redis', etc.
 max?: number;               // max entries (general param, cascades)
 ttl?: number;               // TTL in ms (general param, cascades)
  // type-specific options live here too, keyed by type
 [key: string]: unknown;
} | 'disabled';

Three states:

undefined — inherit from parent level
'disabled' — explicitly turn off caching at this level
CacheConfig object — explicit configuration

Cascading configuration

Cache config can be set at multiple levels. Each level inherits from its parent unless explicitly overridden:

client → db → collection → session → per-operation

Inheritance rules:

General params (max, ttl) cascade down.
If a lower level switches type, type-specific params reset to defaults (not inherited from parent).
Session overrides collection — session is a logical grouping of operations, natural place to override runtime behavior (e.g., disable cache for a bulk import).
Per-operation skipCache?: boolean is the most granular escape hatch.

Passing a cache instance

Users can provide either:

Settings — Pongo creates and manages the cache provider.
A pre-built cache provider instance — Pongo uses it directly.

If settings are the same across multiple collections, Pongo can reuse the same provider instance internally. When a user passes an instance, multiple collections can explicitly share one cache.

Defaults

Enabled by default
max: follow lru-cache recommended defaults (1000)
ttl: follow lru-cache recommended defaults
Default provider: in-memory (lru-cache)

Integration points

Where: `pongoCollection` factory function

Cache logic is added directly inside pongoCollection, not as an external decorator/wrapper. This gives cache operations access to internal state (filter inspection, _version, write outcomes) and keeps observability precise.

Read operations

findOne:

If the filter targets _id, check cache first.
Cache hit → return cached document.
Cache miss → query DB, populate cache, return.
skipCache?: boolean option available.

findMany / other query methods:

If the filter is a list of _id values, check cache for each.
Return cached hits, query DB for misses, populate cache with DB results.
Non-_id filters bypass cache entirely (cache is by-id only).

Write operations

insertOne:

After successful insert, put the document into cache.

insertMany:

After successful insert, put all documents into cache.

updateOne / updateMany / replaceOne:

After successful write, update the cache entry with the new document state.

deleteOne:

After successful delete, evict from cache.

deleteMany:

If deleting by ids, evict those ids from cache.
Filter-based deleteMany does not evict (no way to know affected ids without extra query). Future improvement possible.

Optimistic concurrency

On concurrency error (version mismatch), evict the stale entry from cache. This is critical — a concurrency error means someone else updated the record (e.g., from another node), so the cached version is stale.
If a caller provides expectedVersion and the cached document has _version, compare them in memory. If they don’t match, we know it’s stale without hitting the DB — cheap short-circuit.

`handle` method

Accepts id: string | string[] (overloaded: string returns single result, string[] returns array).
Read phase uses cache by default.
Write phase updates/evicts cache based on outcome.
skipCache?: boolean available in handle options.
For batch: loads all ids from cache, fetches misses from DB in one query, processes handler per document, batch writes to DB, updates cache after commit.

Transactions

Writes update cache only after commit. Not during the transaction — uncommitted data should not enter the cache.
Reads within a transaction check cache by default (most transactions are short-lived, cached data is probably valid). If the transaction modified a document and then reads it, cache miss falls through to DB which returns the correct uncommitted state.
Cache can be disabled per-session: startSession({ cache: 'disabled' }).
Per-session cache settings follow the cascade (session overrides collection).

Error handling

Cache provider errors are swallowed — treated as cache misses. The application never fails because of a cache failure.

Cache is an optimization, not a correctness requirement.
Errors trigger onError hook for debugging.
Future: errors will become metrics and trace events when Pongo’s observability layer lands.

Future considerations (out of scope for v1)

Cache warming: pluggable strategy for pre-populating cache on startup. Interface hook defined but no default implementation shipped.
Query-result caching: caching results of non-_id queries. Complex invalidation — deferred.
Cross-node cache sharing: Redis/Memcached providers. The interface supports it; no provider shipped in v1.
Filter-based deleteMany eviction: would require querying affected ids before delete.
Observability integration: metrics (hit ratio, latency saved, eviction counts), trace attributes, log entries for cache operations.

Implementation approach

Define PongoCacheProvider interface and CacheConfig types.
Implement in-memory provider using lru-cache.
Build Pongo cache wrapper (key prefixing, event hooks, error swallowing).
Wire cascading config resolution (client → db → collection → session → per-op).
Integrate into pongoCollection: intercept _id-based reads, update cache on writes, evict on deletes and concurrency errors.
Extend handle to accept string | string[] with batch support.
Add skipCache to operation options.
Tests: unit tests for cache provider, integration tests for cache + collection operations, e2e tests for concurrency scenarios.

Cheers!

Oskar

p.s, and yes, Nano Banana also did pretty well generating the cover of this post!

p.s.2. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

The End of Coding? Wrong Question

Oskar Dudycz — Mon, 09 Mar 2026 10:54:24 GMT

Be careful what you wish for, because your wish may come true.

What LLMs revealed is how many people in our industry don't like to code.

It's intriguing that now they claim and showcase what they "built with Claude", whereas usually that means they generated a PoC.

It's funny, as people still focus on how they're building, so it's all about the code. And if that's the message sent outside, together with the thought that LLMs are already better than "average coder Joe", then the logical follow-up question is: why do we need those humans in the loop?

I think that most people look at the forest and see trees. The current way of working with LLMs is not scalable. It's transition phase. I can't imagine calling myself an engineer and doing ONLY stringly-typed development with chat or markdown.

What I can imagine is getting help from it, and using those tools as a help for research, for generating the OUTPUT, still keeping me responsible for the OUTCOME.

Why am I saying that we're in the transition phase? As prompting in our natural language is not precise, it's verbose, and adding a translation layer between our freeform prompts and programming languages is a waste of time (and tokens, which LLM vendors love).

I think that we'll still be coding, but with some other layer, as LLMs are good with structured input, like programming languages. So we might need other programming languages than we have atm. Might we need different tools to evaluate LLMs' output to make it deterministic? Might we need a different approach for engineering to make it scalable? Might we need more?

Still, I don't see those discussions.

I mostly see: noise and celebrities who don't code showing their beautiful PoCs. And doing a mic drop about the end of coding. Just like PoC represents the whole Software Development Life Cycle.

So, will we code or not?

If the answer is yes, let's talk about what's next, then let's discuss how and when.

If the answer is not, then let's talk about how our job will change, ask if it's still engineering, etc.

Let's try to make our discussions more precise, more focused on the essence, and avoid them from becoming just a bunch of anecdotal evidence.

Noone outside our industry cares how we code unless it changes the cost, quality, or delivery time.

Let's discuss the impact that matters, rather than just the amount of code we produce (or not).

Now, guess who I am quoting:

Imagine you’re a software application developer. Your programming language of choice (or the language that’s been foisted on you) is Java or Typescript . You’ve been at this for quite a while and your job doesn’t seem to be getting any easier.
These past few years you’ve seen the growth of multiple incompatible architectures. Now you’re supposed to cope with all this and make your applications work in a distributed client-server environment. The growth of the Internet, the World-Wide Web, and “electronic commerce” haveintroduced new dimensions of complexity into the development process.
The tools you use to develop applications don’t seem to help you much. You’re still coping with the same old problems; the fashionable new object-oriented techniques seem to have added new problems without solving the old ones.
You say to yourself and your friends, “There has to be a better way”!
The Better Way is Here Now
Now there is a better way—it’s our new model. Imagine, if you will, this development world…
• It’s still dead simple.
• Your development cycle is much faster.
• Your applications can be created across multiple platforms. Write your spec once, and you never need to port them—they will be recreated if you without your hand-rolled modification on multiple operating systems and hardware architectures.
• Your applications are adaptable to changing environments.
• Your end users can trust that your applications are secure, and you can use protection against viruses and tampering through security scans.
You don’t need to dream about these features. They’re here now.

And also this from another source:

When I started interviewing programmers in 2005, I would generally let them use any language or tool they wanted to solve the coding problems I gave them. 99% of the time, they chose Java.
Nowadays, they tend to choose LLMs.
Now, don’t get me wrong: there’s nothing wrong with LLM as an implementation tool.
Wait a minute, I want to modify that statement. I’m not claiming, in this particular article, that there’s anything wrong with LLM as an implementation tool. There are lots of things wrong with it but those will have to wait for a different article.
Instead what I’d like to claim is that LLM is not, generally, a hard enough programming tool that it can be used to discriminate between great programmers and mediocre programmers. It may be a fine tool to work in, but that’s not today’s topic. I would even go so far as to say that the fact that LLMs aere not hard enough is a feature, not a bug, but it does have this one problem.

Well, I cheated you, but only a bit. I changed “Java” to “LLM ” and cut some phrases.

The first one is from “The Java Language Environment” by Sun Microsystems, introducing Java in 1995.

The second one was from Joel Spolsky’s “The Perils of JavaSchools” article, written in 2001.

Let me be clear. I’m not trying to do grandpa talk on the old days and claim that it’s the same old thing.

What I’m trying to say is that we were continuously introducing new abstractions into our development cycle to scale it. By scale, I mean: getting more people to deliver more code. Even Java was invented for precisely this goal. Yes, the one that’s together with craftsmanship madness presented as “the enterprisy complex environment”. In the early days, it was just said that it’ll make us dumber.

The goal of abstraction is not to gatekeep but to allow us to reduce cognitive load. We invented new languages to help us, but then added more components like a distributed environment, multiregion, because of globalisation. For the same reasons, we use cloud-native tooling so we don't have to deal with it.

Some of the architecture and security tools were commoditised by the cloud. We don’t need to think about much stuff we had to do before.

Will Claude-native do the same?

We don’t need to learn Assembler, C++, Lisp anymore; we have lost a lot of mechanical sympathy. We deal with higher abstractions, but we still engineer solution, we still code, is it a different coding? It is. Is it better or worse? Well, it is how it is. As Gerald Weinberg said:

Things are the way they are because they got that way

And now the question is whether we’re fine with the way we’re doing stuff, and where we're getting to.

Personally, I don’t think that stringly-typed markdown or chat-based design will be the thing in the future.

Knowing how skilled we were always with:
- breaking down tasks into smaller chunks,
- writing precisely what we had in mind,
- thinking before doing,
- waging tradeoffs,

I'm optimistic about the Spec-Driven Design idea. It's going to be great.

Not.

If we want to call ourselves engineers, we need to put more structure and determinism.

I agree that reviewing all code generated by the GenAI is not sustainable.

But I also don’t think that generating tons of code is, in general, sustainable.

With the current state of the art we have, sure, that’s some solution to just generate based on the tools we have.

But let’s start to think what’s next.

Simon Wardley brought an interesting point:

That said, it is fine for an entire culture to decide that producing outputs matters more than understanding mechanisms. You only have to compare the practical engineering of the Roman Empire and the loss of inquiry from science in the Hellenistic age to see this. When the Roman Empire collapsed, the practical knowledge embedded in those institutions (how to maintain aqueducts, how to produce certain grades of concrete) was lost remarkably quickly. Not because people decided to forget it, but because the knowledge was procedural, embedded in chains of practice rather than recorded as transferable understanding. When the chains of practice broke, that embedded knowledge went with them. We had to rediscover the art of inquiry (i.e. Science) to bring them back.

So if we don’t code, how do we hone our skills? How will newcomers from LLMSchools (using Joel’s terms) be able to decide whether something is wrong or right? I don’t think that you can be good on something without doing it.

The paradox with code is that it’s not an asset; it’s a liability. Some say that code doesn’t matter, only proper design. But how do you define “proper design”? Yes, code style doesn’t matter as long as it works as expected. But…

But code eventually matters, as that’s the source of truth for what’s on production. As Alberto Brandolini said:

It's developers' (mis)understanding, not domain experts' knowledge, that gets released in production.

Now, it’s the developers’ and LLMs’ misunderstandings that are deployed to production, not the expert’s knowledge. Neither the markdown spec.

And coding is just one danger.

Outsourcing thinking is an even more dangerous path, as:

If LLMs are doing everything, then again, what are humans for? Aren’t we cutting the branch on which we’re sitting?
LLMs are statistical parrots. They repeat the most possible answer. Which means mediocre. This can still be fine enough for many cases, but for those we want to make a difference for? Definitely not.
Just like we’re losing our coding skills by not doing them, we’re losing design skills by not practising them.

Of course, whatever happens, LLMs will stay with us. How and where it’s hard to say. Unless you have a Magic 8 Ball of 100% correct predictions. I don’t.

That’s why I’d like our industry to finally start mature discussions on the real impact. I would like us to stop acting like children, bragging about generating code and then claiming that code doesn’t matter.

I’d like to think about how to reshape our SDLC process and make it sustainable.

I’d like us to think about what tools we need, and how to change what we have.

If we won’t finally start to do it, then things will be the way they are because they got that way.

And it might not be what we wished for.

Cheers

Oskar

p.s. to kinda prove that I’m more sceptic and pragmatic than hater, I recently started playing with building an Agent with Emmett to better understand those tools. If you’d like to read about the findings and honest thoughts I have while doing it, please comment!

Parse, Don't Guess

Oskar Dudycz — Mon, 02 Mar 2026 17:18:15 GMT

Last time, I shared with you how sneaky I was on transaction handling.. Today, the opposite: I’ll tell you how I fixed the issue when I tried to be too sneaky. I already told you that Sneaky Code Bites Back. The moral? Do as I tell, not how I do.

In some environments, we’re spoiled. We’re getting a lot from a Base Class Library or standard frameworks, so we stop thinking that those issues can exist. For instance, serialisation. Do you know how many data types JSON has? 6. Six. Sześć.

Exactly those:

string,
number,
boolean,
object,
array,
and (TADA!) null.

What about number precision and size? It is. That’s what I can tell you, but it’s not enough, e.g., to keep big int/long, etc. What about Dates? Also, there are none. I wrote about it longer here or how much fun that brings.

If you use statically typed languages and runtimes like C#, Java, etc., your serialiser can, in addition to parsing, perform mapping and, sometimes, validation. And it can also be tricky, as nicely Alexis King put in her “Parse, don’t validate”.

If you’re in a dynamic environment, like JavaScript, then you’re left with parsing and explicit mapping afterwards. What about TypeScript? Same case, types are only used during compilation, then erased and not visible at runtime. So, the place where we do parsing.

Because JSON was defined a long time ago, JavaScript moved on and now supports bigints (Big Integers) and Dates natively (what an achievement!), which created a gap I wanted to fill.

As you know from my previous articles (e.g. this one), big integers are quite important in distributed processing. You can represent the position in log with them. Since your log may be quite long, regular numbers aren’t enough. Or they’re long enough, until they overflow, then they’re not anymore.

I’m using those bigint types extensively in internals in Emmett and Pongo. And I store them in JSONs. I store them as alphanumeric strings, because strings don’t have a max length (or at least I don’t know such).

So, for instance, an event payload can look like:

{
  "type": "InvoiceIssued",
  "data": {
    "invoiceNumber": "123",
    "version": 1,
    "issuer": "John Doe",
    "issuedAt": "2026-02-23T14:07:20Z"
  },
  "metadata": {
    "streamPosition": "3",
    "globalPosition": "928391"
  }
}

As you can see in the metadata stream and global positions, the values are bigints (even if they’re smaller than the maximum value), and data can also use bigints if the user decides to (e.g., invoice number).

And encoding data is simple: you convert it to a string, call it a day. But how to get it back?

And here’s where my struggles started. How do you know that someone intentionally used bigint when they just wanted to store a number as a string?

There are several options. The first one is: encode value.

We could store it, for instance, as:

prefixed value: “_bigint:928391”. But then you need to find a prefix that will be unique enough not to cause conflicts,
nested object, e.g. { “_kind”: “bigint”, value: “928391” }.

Then, either based on the prefix or the object structure, we could automatically decode the value. Still, ther creates other issues, as the structure no longer matches the original value. If we’re just storing and retrieving, that shouldn’t be so bad, but… But remember that in Pongo I’m allowing the use of PostgreSQL and SQLite as document databases, supporting such queries:

const invoices = pongoDb.collection("invoices");

const invoiceNumber = 123n;
const invoice = await invoices.findOne({ invoiceNumber });

That gets translated into a fancy JSONB SQL query.

Of course, I could work around it by encoding the value, but… But I was lazy!

I decided to use a Get Out of Jail Free Card and just treat all strings with numbers as bigints. Sneaky. And it will get even sneakier.

In JavaScript, JSON.parse accepts a parameter that allows you to provide custom mapping logic. I decided to use it and check if the string is alphanumeric, and gulp, I’ve used a regular expression to parse it:

const bigIntReviver: JSONReviver = (_key, value) => {
  if (typeof value === 'string' && /^[+-]?\d+n?$/.test(value)) {
    return BigInt(value);
  }

  return value;
};

Yes, it’s either DNS or Regex. Or both.

I explained in another article that JavaScript runtime doesn’t like where you do CPU-heavy computations.

Small Regex isn’t CPU-heavy, but if you consider that ther will be done for each string in each document or event you try to deserialise, and multiply that by the number of concurrent requests? That can cause the JavaScript event loop to freeze.

What’s more, I plugged that automatically into node-postgres driver custom type handling, so each JSONB deserialization goes through it.

Again, not shit, Sherlock. I should have known it wasn’t the best choice, but I was busy trying to be sneaky at that moment.

Fortunately, a user, Dawid, benchmarked and noticed CPU freezes. It wasn’t catastrophic, but clearly needed a fix.

The Shift

And here I had several options. I could keep hacking on the same idea- maybe replace the Regex with a simpler string check, still globally. Or I could just ignore bigints during deserialisation entirely, let them stay strings, call it a day, move on. Or I could apply the encoding I mentioned earlier, prefixed values, and nested objects. All of those would fix the performance issue. And all of those would be the same kind of wrong choice I already made: trying to solve a schema problem without the schema.

Because that’s the actual mistake here, not the Regex. The pg driver has no idea what your schema looks like. It doesn’t know that “928391” is a bigint and “John Doe” is a name. It doesn’t know that “123” is an invoice number (bigint!) and “90210” is a zip code (string!). I asked it to guess, and it guessed wrong, because there is no right guess at that level.

Enough is enough. I had been planning to do ther properly for a while, and the performance issue gave me the push I needed.

Old rule says: “Make it work, make it right, make it pretty”. I had “make it work” covered for a long time. Now it was time for “make it right”.

And honestly? It wasn’t that hard. Maybe because “make it work” came first, I already understood the problem well enough to see the shape of the solution.

In Pongo, I dropped the automatic bigint parsing from the driver entirely. If you want bigint or date parsing, you say so at the client level:

const client = pongoClient({
  driver: databaseDriver,
  connectionString: postgresConnectionString,
  serialization: {
    options: {
      parseBigInts: true,
      parseDates: true,
    }
  },
});

By default, strings stay strings. You opt in. I didn’t want to break things for users who don’t care about bigint precision or don’t have performance-sensitive workloads. The serializer became an explicit parameter passed down to each query, each collection, each operation- instead of a global that silently changed everything.

That was the “make it right” part for Pongo. But disabling alone isn’t a solution, it’s a band-aid. Users who need bigints and dates still need a way to get them back after deserialisation. The question is: where does that conversion happen?

And that’s where upcasting comes in. Let me start with a simple example in Pongo, then build up.

Say you have a user document. In the database, dates are stored as ISO strings, and the version counter is a numeric string (because JSON). But in your application, you want proper Date objects and bigints:

type UserDocStored = {
  name: string;
  createdAt: string;
  lastLogin: string;
};

type UserDoc = {
  name: string;
  createdAt: Date;
  lastLogin: Date;
};

The upcast function does the conversion:

const upcast = (doc: UserDocStored): UserDoc => ({
  name: doc.name,
  createdAt: new Date(doc.createdAt),
  lastLogin: new Date(doc.lastLogin),
});

You wire it into the collection, and every read goes through it:

const collection = pongoDb.collection(
  'users',
  {
    schema: { versioning: { upcast } },
  },
);

What’s in the database:

{ "name": "Alice", "createdAt": "2024-01-15T10:30:00.000Z", ... }

What you get back:

{ name: 'Alice', createdAt: Date, ... }

That’s all. new Date(str) is cheap. Running a Regex against every string in the document is not. The CPU freeze Dawid spotted came from that check running millions of times at the driver level for every field on every concurrent request. With upcasting, the conversion runs only for the fields you declared, in a plain function, no Regex.

But ther is just type mapping - the simplest case. As I wrote about in my serialisation article, the explicit mapping pattern is useful for much more than just fixing types. It’s the same pattern you need for schema versioning. It defines the stored schema and the application schema separately together with function to transform one into the other.

Let’s say business requirements changed. You now need to group user data differently: a profile object for identity data, and a timestamps object for temporal data. The V1 documents are flat. The new V2 shape is nested:

type UserDocV1 = {
  name: string;
  createdAt: string;
  lastLogin: string;
};

type UserDocV2 = {
  profile: {
    name: string;
  };
  timestamps: {
    createdAt: Date;
    lastLogin: Date;
  };
};

Ther isn’t just a type change like string-to-Date anymore. The structure itself is different. Flat fields became nested objects, and field names moved into sub-objects. And you have thousands of V1 documents already stored. You can’t migrate them all at once (or don’t want to, because it’s risky, and some consumers might still expect V1). But your application now expects V2.

Compatibility FTW

Ther is where backward and forward compatibility come in.

Backward compatibility means: old data still works. V1 documents stored months ago need to be readable by the V2 code. The upcast handles ther. It reads the document in whatever shape it has and transforms it into V2.

Forward compatibility means: new data doesn’t break old consumers. If you have another service or an older deployment that still reads the V1 format, it needs to keep working. The downcast handles ther. When storing V2 documents, it writes the V1 fields alongside the V2 fields, so older readers can still find what they expect.

Together:

type StoredPayload = UserDocV1 & UserDocV2;

const upcast = (doc: StoredPayload): UserDocV2 => ({
  profile: doc.profile ?? { name: doc.name },
  timestamps: {
    createdAt: new Date(doc.timestamps?.createdAt ?? doc.createdAt),
    lastLogin: new Date(doc.timestamps?.lastLogin ?? doc.lastLogin),
  },
});

const downcast = (doc: UserDocV2): StoredPayload => ({
  name: doc.profile.name,
  createdAt: doc.timestamps.createdAt.toISOString(),
  lastLogin: doc.timestamps.lastLogin.toISOString(),
  profile: doc.profile,
  timestamps: doc.timestamps,
});

Look at the upcast: if the nested profile or timestamps fields exist (document was written by V2 code), it uses them. If they don’t exist (for the old V1 document), it falls back to the flat fields. One function handles both old and new documents: that’s backward compatibility.

And look at the downcast: it writes name, createdAt, lastLogin as flat string fields (V1 shape) alongside profile and timestamps (V2 shape). A service still reading V1 sees the flat fields and works fine. A service reading V2 sees the nested ones. That’s forward compatibility.

You wire both into the collection:

const collection = pongoDb.collection(
  'users',
  {
    schema: { versioning: { upcast, downcast } },
  },
);

From here, your application code only deals with V2. The collection handles the translation in both directions:

const v2Doc: UserDocV2 = {
  profile: { name: 'Alice' },
  timestamps: {
    createdAt: new Date('2024-01-15T10:30:00.000Z'),
    lastLogin: new Date('2024-06-20T14:45:00.000Z'),
  },
};

await collection.insertOne(v2Doc);

What’s stored (downcasted, both shapes for compatibility):

{
  "name": "Alice", 
  "createdAt": "2024-01-15T10:30:00.000Z",
  "lastLogin": "2024-06-20T14:45:00.000Z",
  "profile": { 
    "name": "Alice" 
  },
  "timestamps": { 
    "createdAt": "2024-01-15T10:30:00.000Z",
    "lastLogin": "2024-06-20T14:45:00.000Z" 
  } 
}

Then you can read it back with:

const doc = await collection.findOne({ ... });

And get upcasted to V2 data in your application code:

{
  "profile": { 
    "name": "Alice" 
  },
  "timestamps": { 
    "createdAt": "2024-01-15T10:30:00.000Z",
    "lastLogin": "2024-06-20T14:45:00.000Z" 
  } 
}

Same collection, V1 and V2 documents coexisting. insertMany, replaceOne, findOne: all go through the upcast/downcast. No batch migration needed. You roll out the new code, and old documents are handled transparently.

There’s another thing the downcast gives you: querying remains backwards-compatible. Because the downcast writes the flat V1 fields alongside the nested V2 ones, a query like collection.findOne({ name: ‘Alice’ }) still works even though V2 code doesn’t use name directly anymore. The V1 field is there in the stored document. That matters if you have queries or indexes built against the old shape. They don’t break.

Now, for events, ther matters even more. In event sourcing, stored events are immutable- the log is append-only, and you don’t modify what was already written. I wrote about versioning patterns in more detail. The core idea is: your business evolves, your code evolves, your event schemas evolve, but the events in the store stay as they were. You can’t go back and rewrite them (well, you can, but you really shouldn’t). Upcasting is how you bridge the gap.

For Emmett, the same pattern works at the event store level. You define the stored shape (what JSON gives you from the database) and the application shape (what your code works with):

type ShoppingCartOpenedFromDB = Event<
  'ShoppingCartOpened',
  { openedAt: string; loyaltyPoints: string }
>;

type ShoppingCartOpened = Event<
  'ShoppingCartOpened',
  { openedAt: Date; loyaltyPoints: bigint }
>;

And an upcast that handles each event type:

const upcast = (event: Event): ShoppingCartEventWithDatesAndBigInt => {
  switch (event.type) {
    case 'ShoppingCartOpened': {
      const e = event as ShoppingCartOpenedFromDB;
      return {
        ...e,
        data: {
          openedAt: new Date(e.data.openedAt),
          loyaltyPoints: BigInt(e.data.loyaltyPoints),
        },
      };
    }
    case 'ShoppingCartConfirmed': {
      const e = event as ShoppingCartConfirmedFromDB;
      return {
        ...e,
        data: {
          confirmedAt: new Date(e.data.confirmedAt),
          totalCents: BigInt(e.data.totalCents),
        },
      };
    }
    default:
      return event as ShoppingCartEventWithDatesAndBigInt;
  }
};

You pass it when reading a stream:

const { state } = await eventStore.aggregateStream<
  ShoppingCartState,
  ShoppingCartEventWithDatesAndBigInt
>(shoppingCartId, {
  evolve: evolveState,
  initialState,
  read: { schema: { versioning: { upcast } } },
});

Or in a command handler:

const handle = CommandHandler({
  evolve,
  initialState: () => ({ ... }),
  schema: { versioning: { upcast: upcastDatesAndBigInt } },
});

The difference with events is that you can’t update them in place. For documents, you have both directions: upcast on read, downcast on write. For events, upcasting is the main tool because the event store is append-only. Old events stay as they were written. But downcasting has its place too.

Consider ther: you have a projection or a subscriber that was built months ago against the old event schema. Maybe it’s a read model that listens to ShoppingCartOpened and expects clientId as a flat string. But your current code evolved. Now ShoppingCartOpened carries a client object with id and name:

// What old subscribers expect
type ShoppingCartOpenedV1 = Event<
  'ShoppingCartOpened',
  { clientId: string; openedAt: string }
>;

// What current code produces
type ShoppingCartOpenedV2 = Event<
  'ShoppingCartOpened',
  { client: { id: string; name: string }; openedAt: Date }
>;

Upcasting enables the current code to read older events. The ones stored with just clientId. Downcasting helps old subscribers consume new events. It transforms the new client object back into the flat clientId they expect. Same principle as with documents, but especially important here because event subscribers often live in separate services or deployments that you can’t update all at once.

And the same upcast function that started as simple type mapping string → Date, string → bigint handles ther structural change too. You just add another case to the switch:

case 'ShoppingCartOpened': {
  const e = event as ShoppingCartOpenedV1;
  return {
    ...e,
    data: {
      client: { id: e.data.clientId, name: 'Unknown' },
      openedAt: new Date(e.data.openedAt),
    },
  };
}

Old events get the client object synthesised from the flat clientId. New events already have it. The evolve function only deals with the V2 shape.

And here’s where things started to click for me. I added upcasting to fix the bigint problem: explicit type mapping instead of a Regex. But the same mechanism, without any changes, also handles structural versioning. The simple string → Date mapping from the first example is the same code path as the clientId → client migration above. It’s one function, one place, one pattern for all of it: type coercion, field restructuring, schema migration.

Right decisions stack

Right decisions stack. The Regex hack was blocking the slot where upcasting should have been all along. Once I removed it, the performance got fixed, and I got schema versioning on top. One fix created room for the next one, which created room for the next. That doesn’t happen when you keep patching around the same bad decision.

Looking back, maybe the Regex wasn’t the wrong first move. The rule is “make it work, make it right, make it pretty”, in that order. The Regex made it work. It had performance problems, but it still let me ship and learn where the real problem was. If I had tried to design the upcast/downcast system from scratch, without having lived with the Regex for a while, I might have over-engineered it or missed the connection to schema versioning entirely. The understanding came from living with the shortcut.

Dawid raised a performance issue with Pongo projections, but the same Regex was running in Emmett as well. I could have fixed it in one place and called it a day. Instead, I used it as a push to do the thing I’d been planning anyway and applied it to both Pongo and Emmett to keep things consistent. Because I already understood the problem well enough, “make it right” turned out easier than I expected.

You can recover from shortcuts. You should. But you also shouldn’t be afraid to take them in the first place, as long as you come back and do it properly.

Full changes:

Cheers!

Oskar

How I cheated on transactions

Oskar Dudycz — Mon, 16 Feb 2026 16:53:30 GMT

We’re being told that software design is the art of making tradeoffs. But... Are we taught how to make them?

Not that it’s easy to teach tradeoffs, it’s a subtle art of explanation. You need to provide enough context and be precise so others don’t treat tradeoffs as a general best practice. Because they’re usually not such, they typically come from the muddy banks of the Wishkah River.

I think that a decent way is to tell the story. Not the fairy tale, but the specific tradeoff applied in practice. That’s what I’m going to do today: tell you how I cheated on database transactions.

Dumbo

Do you know Dumbo? Of course, flying elephant, such a sweetie. It’s also a codename of my Open Source project. I didn’t tell you about it so far, as it’s a shared dependency for Pongo and Emmett, responsible for:

connection pooling,
safe connection lifetime management,
handling transactions,
migrations,
SQL execution.

Not that small a scope for a Dumb tool, aye? Still, the intention is to make usage dumb, hiding the weirdness of a specific SQL database, so I solve it once and don’t need to be constantly distracted thinking about it, but focus on:

How to append and process events the best way in Emmett,
How to make the best JSON handling and translation into specific SQL dialects in Pongo.

Dumbo usage is quite simple:

import { dumbo } from '@event-driven-io/dumbo';
import { pgDumboDriver } from '@event-driven-io/dumbo/pg';

const pool = dumbo({ connectionString, driver: pgDatabaseDriver });

You need to set up a specific database driver (e.g. pg for PostgreSQL or sqlite3 for SQLite), as Dumbo now supports multiple relational databases.

Having that, you can do stuff like:

import { SQL } from '@event-driven-io/dumbo';

await pool.execute.batchCommand([
  SQL`CREATE TABLE test_users (id SERIAL PRIMARY KEY, name TEXT)`,
  SQL`INSERT INTO test_users (name) VALUES ('Alice'), ('Bob')`,
]);

And also do queries:

const count = pool.execute.query<{count: number}>(
  SQL`SELECT COUNT(*) as count 
      FROM test_users 
      WHERE ${SQL.in('id', userIds)}`,
);

It’ll handle query parameterisation, data escaping, etc.

It can also handle transactions:

const users = await pool.withTransaction(async (tx) => {
  await tx.execute.command(
    SQL`INSERT INTO test_users (name) 
        VALUES (${firstUserName}), (${secondUserName})`,
  );

  return execute.query(
    SQL`SELECT *
        FROM test_users 
        WHERE ${SQL.in('id', userIds)}`,
   );
});

As simple as this looks, you should already have the question about the first tradeoff I made in your head.

Why on earth I think that’s a sane move to write my own multi-database driver?!

Well, I agree, that’s not the best move at the first glance, but let me explain to you why in MY context this actually makes more sense:

Yes, there are tools like Knex, Kysely, Drizzle, etc. in Node.js land that handle similar stuff. They’re nice, I really like them, I really do, but… But they are all big and are bringing a lot of their conventions, and when I’m building storage tools like Emmett and Pongo, I need to have more control. I don’t want those tools and their limitations drive my architecture decisions. I also don’t want to be surprised when the creator decides to drop working on it, or when I become a victim of a supply chain attack. Still, when I want a new kitchen table, I don’t start by going to the forest with a saw. I’m still using existing database drivers, what’s more I’m allowing people to choose the one they prefer. I just don’t want to be driven (and also my users be driven) by some other higher abstraction tool.
Even with that, this decision can seem like a bold “how hard can it be?!” statement. And it is, but… But I’ve built, or was co-authoring, such tools in the past. Some were proprietary, some were Open Sourced (see Weasel). And they served me well.
And last but not least, the last point. Well, I only had those two above. OK, I can add that I didn’t plan to make it a general-usage tool, just a small one for my needs.

So I did.

And here we’re at the moment when I had to cheat on transactions because of those decisions.

Cloudflare D1

All relational databases seem similar, but only until you start using them extensively, or until you need to write your own storage library. Then you learn stuff that not necessarily the stuff you’d like to spend time on. For instance:

What’s the difference between databases like PostgreSQL and single-threaded databases like SQLite or DuckDB, and how concurrent processing can be surprising
or that sqlite3 only calls the first query, but the next silently ignores,
etc.

Still, well, abstractions like Dumbo give the possibility to massage such cases behind the scenes.

And then Cloud Databases and SaaS Databases came into play, like Cloudlfare D1.

I was motivated by the generous sponsorship from Sam Hatoum to make supporting Emmett and Pongo a top priority. Thanks Sam, appreciate that! And I made it, but I had to cheat a bit.

Databases like Cloudflare D1 and Supabase expose databases as pay-as-you-go services. They optimise deployment, making it highly scalable, so you don’t need to care about it. It’s cost-effective for start-ups and scenarios when you’re not under a huge load. If you are, then you’ll need to pay more, but they also give you autoscaling at least.

They do it by exposing the database API through HTTP API, for instance PostgREST. This gives them easier management around throughput, security, etc., as each call is made as an HTTP request through the exposed API.

Yet, that kills some options like: ekhm, transactions. The challenge with transactions is that they don’t scale (that’s why MongoDB is WebScale). They don’t scale, as you’d need to open a transaction, do some freehand operations, and then commit or rollback. That means (typically) you need to keep a connection open during that time. If you’re building SaaS, it’s a no-go, because sneaky users would open it for a few hours, do crazy stuff, and kill your SaaS resources’ utilisation.

But, boy, aren’t transactions one of the selling points of relational databases? They do, so how to proceed?

Then you need to cheat, as I did.

Repeatable reads, batches, etc.

Cloudflare D1 doesn’t provide transaction data for the reasons I outlined, but it doesn’t leave us without other tools. Those two tools to let me do the smoke and mirrors trick are:

sessions,
SQL batches.

What are sessions? Per Cloudflare Docs:

A session encapsulates all the queries from one logical session for your application. For example, a session may correspond to all queries coming from a particular web browser session. All queries within a session read from a database instance which is as up-to-date as your query needs it to be. Sessions API ensures sequential consistency for all queries in a session.

Essentially, that means that we’re getting repeatable reads. So when we’re starting specific sessions, they will be handled sequentially.

And now, the second ingredient: Batches. Per Cloudflare docs

Sends multiple SQL statements inside a single call to the database. This can have a huge performance impact as it reduces latency from network round trips to D1. D1 operates in auto-commit. Our implementation guarantees that each statement in the list will execute and commit, sequentially, non-concurrently.
Batched statements are SQL transactions. If a statement in the sequence fails, then an error is returned for that specific statement, and it aborts or rolls back the entire sequence.
To send batch statements, provide D1Database::batch a list of prepared statements and get the results in the same order.

So Cloudflare didn’t allow us to do the full, freehand transaction, but they allowed us to send multiple statements that internally will be executed as a SQLite Transaction. When the request is handled, it’ll open a transaction, run the statements, and return the results.

Cool, let’s mix this soup together, as having that, I decided to:

Fail automatically if someone tries to create a transaction on Cloudflare D1 with an error:

D1 does not support SQL transactions (BEGIN/COMMIT/ROLLBACK/SAVEPOINT). Use { mode: “session_based” } to opt-in to session+batch semantics, or use ‘connection.execute.batchCommand() for atomic multi-statement execution.

Then they get clear information on the first try.

Allow users to explicitly open a session-based transaction by providing mode:

const users = await pool.withTransaction(
  async (tx) => {
    await tx.execute.command(
      SQL`INSERT INTO test_users (name) 
      VALUES (${firstUserName}), (${secondUserName})`,
    );

    return tx.execute.query(
      SQL`SELECT *
              FROM test_users 
              WHERE ${SQL.in('id', userIds)}`,
    );
  },  
  { mode: 'session_based' }
);

When they do it, they will need to be aware of the limitations of the tool they have. So that this will internally create a D1 session, and only handle a single batch of operations properly. We’re mimicking the sequential processing by the session-based repeatable reads capability. Still, we need to remember that we won’t be able to roll back changes across multiple statements. Only a single command or batch command is an atomic operation.

We can’t, for instance, run a batch of updates, and fail the whole batch if one update didn’t change any record. The batch will only fail if the database throws an exception. An exception can be in SQLite only called by a table constraint or trigger.

By making this choice to require explicit mode and naming it explicitly, I didn’t manage to cover all cases, but at least made it safe, so people need to learn about this non-default behaviour and be more careful about it. When designing an API, it’s usually better to start with a more strict option and do a bit of “scarification” sometimes.

Still, when I implemented that in Emmett and Pongo, I intentionally used it to enable event appends, but document operations, etc.

If you’d like to try it, you can check Emmett’s or Pongo’s beta versions.

For Pongo, you can install it with:

npm install @event-driven-io/pongo@0.17.0-beta.21

And use it as:

import { d1PongoDriver } from '@event-driven-io/pongo/cloudflare';

const client = pongoClient({
  driver: d1PongoDriver,
  database,
  transactionOptions: { mode: 'session_based' },
});

Or in Emmett by installing:

npm install @event-driven-io/emmett-sqlite@0.43.0-beta.1

And use it as:

import { getSQLiteEventStore } from '@event-driven-io/emmett-sqlite';
import { d1EventStoreDriver } from '@event-driven-io/emmett-sqlite/cloudflare';

const eventStore = getSQLiteEventStore({
  driver: d1EventStoreDriver,
  database,
});

Still, even if you don’t care about Emmett, Pongo, and my Open Source project, I hope that this will give you decent inspiration for your own tradeoffs analysis.

I hope that you learned a bit about how design APIs work, how to check the guarantees of your tools and learn to walkaround them when you have to.

Cheers!

Oskar

On rebuilding read models, Dead-Letter Queues and Why Letting Go is Sometimes the Answer

Oskar Dudycz — Mon, 19 Jan 2026 18:01:47 GMT

In the last article, I explained how to rebuild Event-Driven Read Models in a safe and resilient way. I asked readers to let me know if they find any blind spots in my design.

Well, I found one myself.

This article is about that edge case, but more importantly, it’s about the rabbit hole I went down thinking about how to “fix” it. At the end of this hole, there is a nice learning about dealing with distributed systems. Sometimes the best engineering isn’t about preventing all failures. It’s about recognising your blind spots and making sure they don’t catch you off guard.

Everyone has a plan until they get punched in the mouth

Let me recap the situation described in the previous article. We’re using PostgreSQL to store events and read models. We want to rebuild an inline projection, one that’s applied in the same transaction as the appended event.

In the design from my previous article, during a rebuild, we:

Mark the projection as “rebuilding”.
Skip inline projections (the rebuild process will catch up anyway).
Process all historical events.
Mark the projection as “active” again.

The hybrid locking strategy with advisory locks and status checks ensures that inline projections know when to skip and that only one rebuild runs at a time.

Sounds solid. Here’s where it breaks:

The rebuild process checks for new events, sees none (because event 1000 is still in an uncommitted transaction), declares victory, and sets the projection to active. Meanwhile, the append transaction has already decided to skip the inline projection. When it finally commits, the event exists but was never projected. The rebuild process is done, and only new events will update the projection during the next update to the new event append.

We could handwave it with it won’t ever happen!. But, well, under load, this will happen eventually. You might not even notice it, but it will. The timing window is small, but with enough throughput, it becomes a certainty.

The Rabbit Hole of “Fixes”

My first instinct was to engineer my way out of this. Surely with enough clever coordination, we can close this gap?

Let me walk you through the rabbit hole.

Attempt 1: Wait for In-Flight Transactions

PostgreSQL provides pg_snapshot_xmin(pg_current_snapshot()) which tells us the oldest transaction that’s still running. The idea: after processing events, wait until all potentially-skipping transactions have completed before marking the projection as active.

Why it fails: We don’t know what we’re waiting for. While we wait for existing transactions to complete, new transactions start and make their own skip decisions. The target keeps moving. We can never “catch up” because new skips happen while we’re waiting for old ones to become visible.

Attempt 2: Use Transaction IDs as Boundaries

PostgreSQL assigns monotonic transaction IDs to each transaction when it starts. This seems useful—what if we record a “sealing” transaction ID when we’re ready to complete the rebuild, and use it to make decisions?

The logic would be:

When rebuild is ready to complete, record sealing_txid = current_transaction_id

Any event from a transaction with ID lower than the sealing point was “in flight” during rebuild, so it should be handled by async Events from transactions with higher IDs started after sealing, so they can safely use inline projections (the read model will be ready)

Sounds reasonable?

Here’s why it doesn’t work.

The core problem: transaction ID order ≠ commit order.

When a transaction starts, PostgreSQL assigns it the next available transaction ID. But transactions don’t commit in the order they started. A transaction that started earlier (lower ID) might take longer to complete and commit after a transaction that started later (higher ID).

Let’s trace through a concrete scenario:

Here’s what happened:

Tx 99 started first and got the lowest transaction ID. It inserted an event and decided to skip the inline projection (status was ‘rebuilding’). But then it got slow—maybe network latency, maybe the application did other work.
Tx 100 (the rebuild) started second, recorded sealing_txid = 100, and prepared to complete.
Tx 101 started third. It checked: “Is my transaction ID (101) >= sealing_txid (100)?” Yes, so it assumed the read model was ready and processed its inline projection. It committed successfully.
Tx 100 marked the projection as active and committed.
Tx 99 finally committed. But it had already decided to skip the projection back when status was ‘rebuilding’. That decision was made, the skip happened, and the event is now missing from the read model.

The fundamental issue: we can’t see uncommitted transactions. When Tx 100 set the sealing point, it had no way to know that Tx 99 was still out there, holding an event that would skip projection. Transaction 99 is invisible until it commits, but by then it’s too late.

You might think: “Just wait until all transactions before the sealing point have committed!” PostgreSQL even provides pg_snapshot_xmin(pg_current_snapshot()) which tells you the oldest active transaction. But this leads us back to Attempt 1—while we wait, new transactions start and make their own skip decisions. The target keeps moving.

I wrote about this exact problem in How Postgres sequences issues can impact your messaging guarantees. The same visibility challenges that affect outbox ordering apply here. Transaction IDs are useful for ordering within committed data, but they can’t help us coordinate with transactions that haven’t committed yet.

Attempt 3: Lock Appends During Transition

What if we use advisory locks more aggressively? The idea: when rebuild is ready to complete, acquire an exclusive lock that blocks the entire append path. While holding this lock, flip the status to ‘active’. No appends can be in progress during the flip, so no race condition, right?

Here’s the proposed flow:

Rebuild finishes processing historical events
Acquire exclusive advisory lock (all new appends must wait)
Set status = ‘active’
Release lock
Waiting appends resume and sees ‘active’, processes inline normally

It should work. In theory. We’re creating a synchronisation point where nothing can slip through. But there’s a subtle problem: the skip decision was already made inside the append transaction.

An append transaction doesn’t just check the lock and status at one instant. It does several things:

BEGIN transaction
INSERT the event
Try to acquire a shared advisory lock
Check projection status
Decide: process inline or skip?
Execute that decision
COMMIT

The decision in step 5 happens inside the transaction. If the transaction saw ‘rebuilding’ status, it was skipped. That decision is now part of the transaction’s pending work. The transaction might be waiting to commit or doing other work, but the skip decision has already been made.

Our exclusive lock in the completion flow blocks step 3 - new transactions can’t acquire the shared lock. But what about transactions that already passed step 5 and are just waiting to commit?

The timing here is tricky. Advisory locks in PostgreSQL can be either:

Transaction-scoped (pg_advisory_xact_lock): released automatically when transaction commits
Session-scoped (pg_advisory_lock): held until explicitly released or session ends

If we use transaction-scoped locks for inline projections (which makes sense—you want the lock tied to the transaction lifetime), then the append transaction might have already released its shared lock by the time we try to acquire an exclusive lock. The lock protected the status check, but the transaction is still running with its skip decision already made.

Even if we could perfectly synchronise the lock acquisition, there’s another problem: advisory locks are session-scoped, not durable.

If the connection dies while holding the exclusive lock:

The lock releases immediately (that’s how advisory locks work)
We might have partially updated the status
The system is now in an unknown state
Other transactions resume with potentially inconsistent data

We can’t build reliable coordination on something that vanishes when connections fail. This is exactly why the original article combined advisory locks with persistent status checks—but that combination doesn’t solve this particular race condition.

TLDR on Attempts

Every “fix” follows the same pattern:

Identify a coordination point.
Discover there’s a window before that point we can’t see.
Try to close that window.
Create a new window somewhere else.
Repeat.

We’re not solving the problem. We’re relocating it.

So is it a bug in the initial design? Depending how you look on that. I see it as an example of fundamental property of concurrent systems. PostgreSQL’s isolation guarantees mean uncommitted transactions are invisible to other transactions. That’s a feature, not a bug - but it means there’s always a window we can’t see into.

Stop Fighting, Start Tracking

After chasing my tail through various “solutions,” I stepped back and asked a different question.

Instead of: “How do we prevent events from being skipped?” (which requires blocking appends or seeing into uncommitted transactions—both unacceptable because of performance, guarantees etc.)

I started to think on: “How do we know when an event was skipped?” and “How do we ensure skipped events get processed eventually?”

Both of these are solvable.

If we can’t prevent skips from happening, let’s make them visible. When an inline projection skips, it could record that it skipped in the same transaction as the event append.

If the append transaction rolls back, the skip record also rolls back (there’s no event to worry about). If it commits, we have a durable record of exactly what was skipped.

In Emmett, I’m using a dedicated emt_system_messages table rather than reusing the regular emt_messages table or creating a simple “skipped events” table. This might seem like over-engineering—why not just create a simple table with projection ID and event position? Or why not just reuse the existing messages table?

Why a dedicated system messages table?

The regular emt_messages table is for business events—the actual domain events that drive your application. Mixing system-level concerns (like “this projection skipped this event during rebuild”) with business events pollutes the event log and makes it harder to reason about. A simple “skipped_events” table with just (projection_id, event_position) could be enough, but in the long term, it may be hard to maintain. As systems evolve, we’ll have more cases where we might want to skip events. Plus, having system events recorded with all metadata gives us full observability of the internals of our system. We could even make tables for projections and processors’ status, built as read models from system events! Still, let’s hold our horses and get back to our use case.

The system messages table could mirror the structure of regular messages with:

Global position sequencing (with all the transaction visibility handling from the outbox ordering article)
Transaction ID tracking for proper visibility checks
Archiving support via is_archived flag
Partitioning for performance

If we created a throwaway “skipped events” table, we’d need to solve all these problems again. We’d essentially be building a second event log with the same guarantees.

The skip could be stored as a system message where:

Stream ID = the projection identifier (name + version), so we can query all skips for a specific projection
Message data = reference to the original event (sequence ID from the event log)
Message metadata = processor ID, reason for skip, timestamp
Message type = indicates this is a “skip during rebuild” system event

Won’t this table grow too much? It may, but we can archive skip events when they’re no longer needed.

When the rebuild processor handles a skipped event, we could archive it. In Emmett, this means setting is_archived = true on the record. The table is partitioned by this flag. Archived records automatically move to a separate partition, which can even be on a different disk drive.

Why archive instead of delete?

Audit trail: You can see what was skipped and when it was processed
Debugging: If something goes wrong, you have history to investigate
Idempotency: If a processor crashes and restarts, it won’t reprocess already - archived skips

Later, we could define retention policies and clean up old archived records. We could do it more aggressively than for business events, since these are operational records, not business history. You might keep business events forever, but archived skip records only need to stick around long enough for debugging (days or weeks, not years).

The same processor that handles regular events for the projection could also process its skipped records. When it reads an event from the main event log and applies the projection, it also checks for and archives any corresponding skip record for that event position. This keeps everything in sync.

As mentioned, this also opens the door for other system events in the future - not just rebuild skips, but potentially failed projections, poison messages, or audit records. The infrastructure would already be in place, separated from the business event log.

The Final Flow

Let me walk through how this works in practice, bringing together all the pieces.

During rebuild

The projection status is ‘rebuilding’. The async rebuild processor works through historical events from the beginning. Meanwhile, normal operations continue. Events are appended, and inline projections check the status. When they see ‘rebuilding’, they skip the projection but record a skip message in emt_system_messages within the same transaction.

When rebuild catches up

The rebuild processor eventually reaches the current position where no new events are visible. At this point, it sets the status to ‘active’. From now on, new appends will process their inline projections normally.

But what about those skip records? The events they reference exist, but their projections were never applied.

Draining the skipped events

The rebuild processor (or a dedicated processor, or a manual trigger—your choice) now queries emt_system_messages for skip records belonging to this projection. Using the same transaction visibility rules from the outbox pattern (transaction_id < pg_snapshot_xmin), it only sees skip records from committed transactions.

For each skip record:

Find the referenced event in the event log (or use its data if stored in skipped event)
Apply the projection for that event
Archive the skip record (set is_archived = true)

The skip record and the event are committed in the same transaction. This gives us a simple invariant: if an event exists, either its projection was applied inline (status was ‘active’) or a skip record exists (status was ‘rebuilding’). There’s no third option where an event exists without a trace.

The drain process might find new skip records appearing. Transactions that were in flight during the status change, committed after we started draining. That’s fine. We keep querying until no more visible skip records exist. They will stop appearing as we alread stopped rebuilding processor, so no more inline projections should be skipped.

The “drain skipped events” phase can be automatically triggered by finishing projection rebuild. It could be handled as the 2nd phase of rebuild processor, or triggering a dedicated one. It could be also just handled by a human initiating the drain.

This flexibility lets user choose the approach based on the specific operational requirements.

The Dead Letter Queue Pattern

What we’ve built is essentially a Dead Letter Queue (DLQ)—a place where messages that couldn’t be processed normally are stored for later handling.

This pattern exists in every serious messaging system:

Apache Kafka: Dead Letter Topics for messages that fail consumer processing
RabbitMQ: Dead letter exchanges for rejected or expired messages
AWS SQS: Redrive policies that move messages to a DLQ after N failures
Azure Service Bus: Built-in dead-letter sub-queues for each queue

The pattern is universal because all messaging systems face the same fundamental problem: sometimes messages can’t be processed immediately, and you need a place to store them without blocking the rest of the system.

It’s a topic in its own right, as it’s not perfect and should be used cautiously. Many teams fail to apply them correctly. The DLQ becomes like a car alarm in a parking lot, technically signalling a problem, practically ignored by everyone. And that’s what I see in many systems: teams set up DLQs and do nothing about them.

It starts innocently. You configure the DLQ “just in case.” A few messages end up there during a deployment. Someone says, “We’ll look at it later.”

More messages accumulate. The DLQ becomes background noise—a number on a dashboard that nobody checks. Eventually, something critical lands there, and nobody notices until a customer complains.

That’s why in the discussed design, skip records aren’t meant to accumulate indefinitely. The rebuild processor drains them during completion.

Retention policies clean up archived records after a reasonable period. If skip records exist for too long, that’s a signal that something is wrong with the rebuild process - and users should know about it.

A DLQ is only helpful if it’s monitored, processed, and understood why messages end up there. Otherwise, it’s just a fancy way to lose data slowly rather than immediately.

The Broader Lesson

This article isn’t really about PostgreSQL advisory locks or projection rebuilding. It’s about how we approach problems in distributed systems.

When we find a race condition, the instinct is to fix it. Add a lock. Add a check. Add a coordination phase. But each “fix” often just moves the race condition somewhere else. We went through three attempts—waiting for transactions, using transaction ID boundaries, locking appends—and each one failed for a different reason. We weren’t solving the problem; we were relocating it.

At some point, I had to ask myself: am I making this system more reliable, or just more complicated?

The answer came when I changed the question. Instead of asking “how do I prevent events from being skipped?” I asked, “How do I know when an event was skipped, and how do I make sure it gets processed eventually?”

The first question has no good answer, not without blocking appends, which defeats the purpose. The second question is straightforward: record the skip in the same transaction as the event, and process it later.

A system isn’t trustworthy because it never fails. That’s impossible for anything sufficiently complex. A system is trustworthy because you know when it can fail, how it will fail, and how to recover. The skip tracking approach doesn’t prevent failures during the transition period. It makes them visible and recoverable. That’s a stronger guarantee than complex coordination machinery with hidden edge cases.

Sometimes the best engineering decision is to accept what you can’t control and focus on what you can. We can’t control PostgreSQL’s transaction visibility rules. We can’t see into uncommitted transactions. We can’t make the append decision and the rebuild completion atomic without stopping the world.

What we can control is recording skipped items, ensuring those skips are processed, and making the whole thing observable. That’s enough.

The blind spot I found wasn’t really about the specific race condition. It was a reminder that distributed systems have fundamental constraints we can’t engineer around, at least not without trade-offs worse than the original problem.

Transaction isolation and concurrent systems mean we can’t have zero-downtime rebuilds with perfect inline projection consistency and no coordination overhead, all at the same time.

Something has to give.

What we get instead is explicit tracking of what was skipped, guaranteed eventual processing via the system messages table, observability into the transition period, and crash recovery that doesn’t lose data. The read model might be briefly inconsistent during the transition, but we know exactly what’s missing, and we have a clear path to fix it.

That’s the kind of system I can reason about and trust.

If you’re dealing with similar challenges, I’m happy to help through consulting or mentoring. You can also join the discussion in our Emmett Discord—we have a nice community working through these exact problems.

Or check also other related resources:

Cheers!

Oskar

Rebuilding Event-Driven Read Models in a safe and resilient way

Oskar Dudycz — Mon, 05 Jan 2026 19:01:10 GMT

Let’s make a soup today: a blog soup. We’ll mix multiple ingredients like:

events (obviously),
read models,
inline and async projections,
rebuilding read models,
backfilling new ones with data from existing events,
scaling async processing horizontally,
distributed locking,
PostgreSQL and its Advisory Locks.

Sounds a lot? Well, the soup should be nutritious.

In an event-driven way, after handling business logic, we record new facts and call them events. They gather information about what has happened. That brings many benefits, such as business observability by keeping a log of them. Especially if we’re doing Event Sourcing, we can make the next decision based on them.

Typically, we’re using events in two ways:

reacting to them, triggering and integrating the steps of our business workflow,
projecting them, and getting the flattened interpretation of our system state inside read models.

Such processing can happen asynchronously, but doesn’t have to. If we’re using a transactional database (like PostgreSQL, SQLite, or even MongoDB), we can update our read models in the same atomic transaction that stores new events. Such a process is typically called inline projection. Event stores like Emmett and Marten allows that. Still, you can do the same if you’re using outbox pattern, then you can achieve the same without Event Sourcing.

Inline processing is tempting because we’re getting immediate consistency. Yet, there’s no free lunch here. We’re slowing down our event append as we need to process more, and our transactions will be open longer, which can cause deadlocks, etc. We also can’t take advantage of batching event processing.

Also, as Bastian Waidelich rightfully pointed out to me, using inline projections also increases the coupling and fragility of our business logic. If the inline projection fails (due to a bug in its projection logic, a database constraint or other random issue), then we won’t be able to append our event, which is counterintuitive, as why would read model block our business logic (e.g. confirming shopping cart).

My thumb rule is that for single stream, simple projections, I prefer inline projections, but for more complex or workflow processing, I’d go with async.

The big benefit of a durable event log is that we can correct past mistakes and gain more insights from existing data.

How does it look in practice? Let’s say we initially had a basic read model that showed a summary of the specific shopping cart. In TypeScript this could look as follows:

type ShoppingCartSummary = {
  _id?: string;
  productItemsCount: number;
  totalAmount: number;
};

Besides the data, we also need a method that represents how we apply events on top of the existing state to get the next, evolved state.

const evolve = (
  document: ShoppingCartSummary | null,
  event: ProductItemAdded | ProductItemRemoved,
): ShoppingCartSummary => {
  document = document ?? { totalAmount: 0, productItemsCount: 0 };

  switch (type) {
    case 'ProductItemAdded':
      return withAdjustedTotals({
        document,
        productItem: event.data.productItem,
        by: 'adding',
      });
    case 'ProductItemRemoved':
      return withAdjustedTotals({
        document,
        productItem: event.data.productItem,
        by: 'removing',
      });
  }
};

const withAdjustedTotals = (options: {
  document: ShoppingCartSummary;
  productItem: PricedProductItem;
  by: 'adding' | 'removing';
}) => {
  const { document, productItem, by } = options;
  const plusOrMinus = by === 'adding' ? 1 : -1;

  return {
    ...document,
    totalAmount:
      document.totalAmount +
      productItem.unitPrice * productItem.quantity * plusOrMinus,
    productItemsCount:
      document.productItemsCount + productItem.quantity * plusOrMinus,
  };
};

Sounds fine, but well, it may appear that either we, implementing it, or our business, through requirements, forgot about some requirements. Or we didn’t forget anything, but just requirements evolved as they tend to always do.

What if we need to handle now also:

Show the shopping cart status and show whether the shopping cart is open, confirmed or cancelled.
not only show totals, but also a list of product items with their details.

Our new read model would look like:

type ShoppingCartSummary = {
  _id?: string;
  productItemsCount: number;
  totalAmount: number;
  status: 'Opened' | 'Confirmed' | 'Cancelled';
  productItems: ProductItem[];
};

type ProductItem = {
  productId: string;
  name: string;
  quantity: number;
  unitPrice: number;
}

And here we should ask ourselves the following questions:

Is it really the same read model or another one that’ll be used in another place in our UI? Maybe the initial just shows basic data in the menu bar, and this will be used as the summary before confirmation?
If the read model is the same, then are you fine with downtime where you clean the old data, and reprocess events?
If it’s the new read model, do we need to backfill it with the old data?

There’s no best practice here; we need to do the drill and be prepared for multiple options.

Add new vs update existing.

In our case, my initial guess would be that this should be a new read model. We need to add significantly more data and new event handling for confirmation and cancellation.

You could say that:

Why add a new read model with similar data? Can’t we just do a subquery?

Of course, you can. I wrote about that more in How to create projections of events for nested object structures?. This can make sense if those read models will always evolve together.

If you need to run multiple views on the same read models, you’re increasing coupling. As such, when you’re rebuilding the read model, then potential downtime will impact both. Also, each time you adjust one view, ensure you haven’t broken the others.

You’re getting a smaller storage size, and potentially don’t need to remember multiple read models, but are doing it just one.

My experience shows that optimising for the end storage until we check that it’s too big isn’t a good driver. Nowadays, storage is cheap, so my default is to keep read models separated, and also not reuse the same evolve logic. A bit of code duplication won’t harm us, but we’ll see benefits as our models evolve. We’ll decrease the cognitive load.

Still, if those models are indeed the same, or we bet they’ll constantly evolve together, then it could be fine to reuse them.

In place update, blue/green rebuild and backfilling data

Ok, I was already using rebuilding word multiple times. But how do we actually do it?

If you’ve read articles on checkpointing or positions in event stores, you already know that each event in the event store/outbox can have its unique, monotonic position. We can subscribe to notifications about new events and process them one by one. That’s how consumers and processors work in Emmett. Once we process a specific event, we can store the checkpoint in the end storage. This enables resilient failover when our processor dies for some reason. When we restart, we’ll read the last processed checkpoint first and start listening for events from that point.

We could reuse this not only for failover but also for rebuilds. Instead of starting processing from the last known checkpoint, we could reset the checkpoint in our database to a specific point (e.g., the beginning of the log). Then we’ll get recorded events again.

And this is the moment we need to decide whether to do an in-place update or a blue/green rebuild.

In-place means that we’re the same storage target. We need to truncate it and then apply data from scratch, typically. The downside is that when we clear it, our read models won’t be up to date with the current state of our event store. We already have newer events recorded, but our read models are empty. We need to fill them in.

The same happens when we create a new read model based on the existing events. When we create a new read model, it won’t magically contain data from existing events.

Typically, we need to spin up a background worker (e.g., Emmett consumer with a projector plugged in) that pulls all events since the beginning and runs projection logic. Obviously, that means we need to run it asynchronously, and depending on the data size, we’ll need to wait until it catches up. During that time, querying for the data will return outdated information - eventual consistency in practice.

That’s why there’s another option. Instead of truncating existing data, we can keep it as it is, or even keep updating it in the old form. Having that, we can build a new read model in parallel. This read model could have a different name, or just a suffix, e.g. V2, V2.0.1, whatever we find helpful.

Then, once this new read model catches up (so it has processed all events, or is close enough to the latest event with a defined threshold), we can switch queries from the old to the new read model storage.

This is actually the preferred way, but it’s a bit more challenging when it comes to dynamically switching the query target. If you’re using Pongo, that’s not that hard, since you just switch the text-based collection name, which is just another table. But if you’re using an ORM, adding a new table dynamically and mapping it can be much more challenging.

Concurrency issues while (re)building read models

To make it harder, eventual consistency is not the only challenge we’ll face. We also need to deal with concurrency and parallel processing.

What should we do when a new event is appended while we’re rebuilding/backfilling the inline projection? If we try to update the end storage in the middle of the processing, we can end up with an inconsistent or erroneous state.

Another issue is what to do if we’re in a Kubernetes-like setup and don’t have full control over the number of instances of the same service? Or what happens if we accidentally spin up multiple rebuild workers?

Then we’re doomed, or at least the consistency of our read model data.

How do we solve it? Let me explain my plan for Emmett.

Distributed Locking and PostgreSQL advisory locks

I encourage you to check my other article: Distributed Locking: A Practical Guide. Yet, don’t worry, I won’t leave you with Read-The-Fucking-Manual type of answer.

Distributed locks are a fundamental tool for coordinating concurrency across systems. We can use a central place, typically scalable on its own, that’ll be used in multiple instances of our service, to ensure that exactly one can request a lock and run specific code.

In the mentioned article, I described multiple options and popular tools for handling that (e.g., Redis, Zookeeper, Kubernetes Replica Sets), as well as PostgreSQL and its Advisory Locks. Let me focus today on the last one and describe how I want to use it in Emmett’s PostgreSQL projection handling.

PostgreSQL gives us two options for coordination.

Row-level locks lock individual table rows. You do SELECT … FOR UPDATE, and anyone else trying to modify that row waits. The lock is tied to the specific row in a table.

BEGIN;
SELECT * FROM some_table WHERE id = @key FOR UPDATE;
-- make changes
COMMIT;

Such locks are straightforward, as we explicitly state what we want to lock. We could define the following table for projections:

CREATE TABLE IF NOT EXISTS emt_projections(
      version                       INT                    NOT NULL DEFAULT 1,  
      type                          VARCHAR(1)             NOT NULL,
      name                          TEXT                   NOT NULL,
      status                        TEXT                   NOT NULL,  
      PRIMARY KEY (name, version)
);

The background worker could lock the projection by a specific name and version and set the status to “rebuilding”. Then, during handling the inline projection, we could use the same lock and check whether the status is “active”; if not, skip processing. Using a lock-in inline projection would also prevent the rebuilding process from starting, as they wouldn’t acquire the lock.

And that’s a nuke option, as it would work but could create a performance problem. If every inline projection needs to grab a row lock on a coordination row, they will all be processed sequentially, one at a time. That’s a throughput killer when you’re appending thousands of events per second. Each of these append would require access to the lock for the specific projection type (e.g. our shopping cart summary), making not only updates but also event appends sequential. That’s not acceptable for most cases.

Advisory locks are different. They’re locks on arbitrary integers. PostgreSQL doesn’t care what the integer means—it just manages who holds the lock. No table rows involved. It also allows two modes: exclusive or shared.

-- Shared lock: many can hold simultaneously
SELECT pg_try_advisory_xact_lock_shared(12345);

-- Exclusive lock: only one holder, blocks shared locks
SELECT pg_advisory_lock(12345);

Those locks are either scoped to an open connection session or an opened transaction (with xact with name) and released automatically once they end.

Shared locks allow multiple sessions to access the lock with the same value. Exclusive lock blocks both other exclusive locks and new shared locks.

This allows a design where shared locks are used for readers, and exclusive locks for writers and maps directly to our problem:

Inline projections take shared locks as they run concurrently, and just need to check if there’s no async job updating these projections
Rebuilds take exclusive locks - they block inlines and other instances of async processing.

We just need to do one more thing: since advisory locks take integers, we need to map our projection name and version to them. We can do it by a consistent hash algorithm, either in the application code or in PostgreSQL.

PostgreSQL provides a built-in MD5 hash function. It’s not perfect, as it’s not a sophisticated hash, but it’s fast enough and predictable. In our case, we won’t have thousands of projections in our application, so the risk of a hash collision is negligible. If you’re still worried it’s too high, we could store id in our projections table and use it instead of hash-mapping. Still, if we used md5 function, it could look as follows:

// shared for inline projections
SELECT pg_try_advisory_xact_lock_shared(
        ('x' || substr(md5(?), 1, 16))::bit(64)::bigint
) AS acquired;

// exclusive for inline projections
SELECT pg_try_advisory_xact_lock(
        ('x' || substr(md5(?), 1, 16))::bit(64)::bigint
) AS acquired;

Where as query param, we’d pass the joined projection name and its version.

Thanks to that, multiple inline projections can access the lock if it’s not held exclusively by the async (re)building worker. Thanks to that, we’re not blocking event appends because of the lock on the inline projections.

An exclusive lock can be held only when there’s no single inline projection being applied at the moment.

What’s also cool is that advisory locks can be used with two strategies: fail fast when the lock is held, or wait for the lock to be released. The first option would be useful for inline projections, and the second for rebuilds.

Is that all? Well, not quite.

Why advisory locks alone aren’t enough

Advisory locks have a gap: they’re session-scoped. If the connection holding the lock dies, the lock releases automatically. For most cases, that’s fine and expected, but think through this scenario:

We have 1000 events in our event store.
Rebuild starts, acquires exclusive lock.
It truncates the projection and processes events 1-500.
Connection dies (network blip, out of memory kill, whatever).
Lock releases automatically.
Projection contains data only for events 1-500.
Inlines see no lock, start processing.
Inline applies event 1001, ending up with a potentially corrupted state (as there may be some events between 500 and 1001 that would impact the state).

Advisory locks can’t persist across connection failures. We need to add something that does.

The hybrid-locking approach

Emmett already maintains tables for tracking processors and projections. The relevant ones:

CREATE TABLE IF NOT EXISTS emt_projections(
    version         INT         NOT NULL DEFAULT 1,  
    type            VARCHAR(1)  NOT NULL,
    name            TEXT        NOT NULL,
    partition       TEXT        NOT NULL DEFAULT 'emt:default',
    kind            TEXT        NOT NULL, 
    status          TEXT        NOT NULL, 
    definition      JSONB       NOT NULL DEFAULT '{}'::jsonb, 
    PRIMARY KEY (name, partition, version)
) PARTITION BY LIST (partition);

CREATE TABLE IF NOT EXISTS emt_processors(
    last_processed_transaction_id XID8    NOT NULL,
    version                       INT     NOT NULL DEFAULT 1,
    processor_id                  TEXT    NOT NULL,
    partition                     TEXT    NOT NULL DEFAULT 'emt:default',
    status                        TEXT    NOT NULL DEFAULT 'stopped', 
    last_processed_checkpoint     TEXT    NOT NULL,    
    processor_instance_id         TEXT    DEFAULT 'emt:unknown',
    PRIMARY KEY (processor_id, partition, version)
) PARTITION BY LIST (partition);

The status column in the projections table can do what advisory locks can’t: persist status between connection crashes. Even if the connection dies, status = ‘rebuilding’ stays in the table. Inlines could check this and skip processing.

The processor’s table tracks checkpoint progress. When a rebuild runs, it updates last_processed_checkpoint as it goes. If it crashes and restarts, it can resume from where it left off rather than starting over.

Potentially, we could reuse the processor’s table, but having a dedicated projection table could also be useful for diagnostics (e.g., storing the projection definition) and for switching queries when a new projection version catches up. We could select the highest active projection version.

Having that, the query checking if the inline projection should be processed or skipped could look as follows:

WITH lock_check AS (
    SELECT pg_try_advisory_xact_lock_shared(
        ('x' || substr(md5($1), 1, 16))::bit(64)::bigint
    ) AS acquired
),
status_check AS (
    SELECT status = 'active' AS is_active
    FROM emt_projections
    WHERE partition = $2 AND name = $3 AND version = $4
)
SELECT
    COALESCE((SELECT acquired FROM lock_check), false) AS acquired,
    COALESCE((SELECT is_active FROM status_check), true) AS is_active;

Two checks, both must pass:

Can we get a shared advisory lock? Fails if a rebuild holds exclusive.
Is the projection status ‘active’? Fails if it’s ‘rebuilding’.

Then for async (re)building worker:

WITH lock_check AS (
    SELECT pg_try_advisory_lock(
        ('x' || substr(md5($1), 1, 16))::bit(64)::bigint
    ) AS acquired
),
ownership_check AS (
    INSERT INTO emt_processors (
        processor_id,
        partition,
        version,
        processor_instance_id,
        status,
        last_processed_checkpoint,
        last_processed_transaction_id
    )
    VALUES ($2, $3, $4, $5, 'running', '0', '0'::xid8)
    ON CONFLICT (processor_id, partition, version) DO UPDATE
    SET processor_instance_id = $5,
        status = 'running'
    WHERE 
       -- We already own it
       emt_processors.processor_instance_id = $5 
       -- Unclaimed
       OR emt_processors.processor_instance_id = 'emt:unknown' 
       -- Previous instance finished or crashed
       OR emt_processors.status = 'stopped
    RETURNING last_processed_checkpoint
)
SELECT
    COALESCE((SELECT acquired FROM lock_check), false) AS acquired,
    (SELECT last_processed_checkpoint FROM ownership_check) AS checkpoint;

Together Advisory locks prevent the race at the transition point where the connection was closed, and the rebuilding job is restarting. The status check handles crash recovery

The rebuild acquires the exclusive lock before updating the status. It waits for in-flight inlines (which hold shared locks) to finish.

If the checkpoint is null, the UPDATE matched no rows, then another instance owns the processor and is actively running. Back off.

If both succeed, you own the lock and the processor. The checkpoint tells you where to resume from (with a fallback to the beginning). Only then does it flip the status and start async processing.

If the rebuild crashes, the lock is released, but the status remains ‘rebuilding’. Inlines check status and skip.

The scenarios walkthrough

As a single good image speaks more than thousands of words, let me give you some diagrams to summarise how it works.

1. During inline projection while event appends:

Each inline projection grabs a shared lock before applying,
Multiple inlines can run concurrently (shared locks are compatible),
Projection updates happen normally.

2. When a rebuild starts:

Rebuild grabs an exclusive lock (waits for any in-flight inlines to finish),
Marks projection as “rebuilding”,
New inlines see the lock or status and skip,
Rebuild processes all historical events,
Marks the projection as “active” and releases the lock,
Inlines resume.

3. If rebuild crashes:

Lock releases automatically
Status stays “rebuilding”
Inlines keep skipping until another rebuild completes

4. Multiple async processors:

Each processor tries to acquire the exclusive lock,
First one wins, others wait or skip,
Guarantees that only one processor handles a projection at a time.

5. When you add a new projection:

No status row exists yet,
Inlines skip automatically,
First rebuild creates the row and backfills data.

TLDR

Just like making delicious soup, designing robust, fault-tolerant and performant distributed systems is not that easy. Building an event store is not that hard. But only if we exclude the async processing part.

I hope this walkthrough covers both the conceptual and practical aspects of handling projection rebuilds.

We used PostgreSQL and Advisory Locks, as PostgreSQL is cool and is a driving force in Emmett. But all the same principles apply to other tools and storage (with their specifics).

I explained why advisory locks and status columns complement each other:

Advisory locks handle the fast path (in-memory, no disk I/O for normal operations) and prevent races at transition points (rebuild can’t start until in-flight inlines finish)
Status column handles crash recovery (persists across connection failures) and new projection bootstrapping (inlines skip until first rebuild completes)

Neither alone is sufficient. Together, they provide the guarantees we need without external infrastructure. Just PostgreSQL doing what PostgreSQL does.

The cost on the hot path is microseconds: one in-memory lock check, one indexed read on a tiny cached table. For most systems, that’s an acceptable tradeoff.

I hope this article also shows you how to use distributed locking in practice.

Please tell me your thoughts and concerns, especially if you see any blind spots in this design! You can do that in our Emmett Discord, come on in, we have a nice community!

If you’re dealing with such issues, I’m happy to help you through consulting or mentoring. Contact me and we’ll find a way to unblock you!

Or check also other related resources:

Cheers!

Oskar

Architecture Weekly is 5 years old! 🎉

Oskar Dudycz — Mon, 29 Dec 2025 10:40:08 GMT

Welcome to the new week!

Who said that LinkedIn notifications are useless? I did, several times. Yet! Today, I opened my computer for the first time since last Tuesday and just saw a notification that Architecture Weekly is 5 years old now!

At this point, you probably know that I’m not the anniversary-type-of-guy (my wife loves that part of me…), but well, that’s a nice milestone!

Architecture Weekly started as a way to organise my readings. I’m a curious guy, and I like to read a lot about different topics. During my career, I worked on a lot of weird projects, which exposed me to many technologies and approaches. It was not always pleasant, but it taught me that technology is a tool, and there’s a big diversity of tools that can lead you to a proper solution.

So, I had always had hundreds of open browser tabs until my Firefox crashed from time to time (yes, I’m a Firefox-type-of-guy). I’m sure you know that too. I tried multiple ways to organise that: tools like Notion, e-mail threads, etc. None of them was perfect. At some point, my friends treated me like a librarian and asked me for interesting links on a topic or what interesting things I’d read recently. I decided to put my readings publicly into a Git repository and use Markdown, so I can easily share them. This repository still exists: https://github.com/oskardudycz/ArchitectureWeekly. At first, just links organised in groups.

I started sharing it with my friends, and they told me it was useful, so I started sharing it with their friends. It seemed so useful to others that I shared it on the other channels, socials, and groups I was part of. Feedback was still positive, and some people started pushing me to create the newsletter, even though they could subscribe to GitHub release e-mail notifications; they still landed in the same notification swamp, as usual, with other notifications. So they asked if I could set up “a proper newsletter”. I was reluctant to do it, but well, as you already know, I did. I used Substack because it helped me avoid the usual accidental complexity by writing my own blogging engine from scratch.

At some point, I decided to try making a paid version of the newsletter. I decided to build a Discord community for paid subscribers and my GitHub Sponsors. I started to also run webinars every month, also inviting guests.

Complete list of webinars we had so far:

And now, yes, all of them are available for free. You can watch them. It’s around 50 hours of free knowledge. I believe that it is of much better quality than some paid courses. Why did I make them public?

At some point, I realised that I don’t have much fun with organising those links; it was more of a burden, as curating such lists takes a lot of time. I’m also a perfectionist-type-of-guy, so I wanted to ensure that what I recommend is worth the read. It triggers the movement of brain cells. So if you read the article, you might disagree with it, but at least it’s sparking some thought. I also wanted to synthesise them and put my comments. It may seem like a simple thing to provide such articles. But try to keep it at the proper level week after week, and you’ll see it might not be as easy as it seems.

There were not enough paid subscribers to justify the time spent on it, even though I was named a Substack Bestseller. Yes, running a newsletter is not a great business plan. There are some exceptions to that, but even if you’re in the top few per cent, then it might not be enough to even justify the time spent on it.

I decided to try a different angle and made Architecture Weekly a fully paid newsletter. Starting with a deep dive into Database Connection Pooling. I tried to see if I could grind for a while, providing high-quality content that’s not available elsewhere. I think I managed to deliver a lot of articles like that, and the paid subscriber count grew, but it was still not enough to justify my work. I enjoyed writing such articles a lot, and I still do, but it has put more pressure on me. And preparing such content takes time and effort. The effort I could spend elsewhere, e.g., with my family, in better-paid work like my workshops and consulting, or in my OSS projects.

And now, here we are, 10 months later and well, I still delivered a lot of deep dives, but shifted closer to the stuff I do daily, so helping other humans build systems and getting benefit from the Event-Driven approach. As you see, recently I cross-posted articles to my blog and here. I also used the angle of my work on tools like Emmett to explain general concepts.

What’s next? I’ll probably continue doing it as is; I might also consider merging the blog and newsletter into one. I even own domain event-driven.news, so maybe some rebranding will happen. But it’s not yet decided.

I’ll definitely continue sharing what I learned, blogging, but the exact form? We’ll see. I’m open to suggestions! Comment under this post and tell me your thoughts.

What type of content would you like to see here? Or is there some that you’d be even open to paying for?

Can’t promise that I’ll deliver all of them, but it’ll give me a food for thought.

Nevertheless, it’s time for a small celebration!

Thanks for being here with me through this journey!

Cheers!

Oskar

Multi-tenancy and dynamic messaging workload distribution

Oskar Dudycz — Mon, 15 Dec 2025 15:50:13 GMT

There are several reasons why I’m blogging.

The first one is that I forget, and writing helps me to remember and organise my findings.

The other is that I like to share my journey with the intention of sparing you, my dear reader, some of my struggles.

Last but not least, it gives me the chance to learn from discussions inspired by them. It’s always a chance to meet new perspectives, correct what I did or just trigger some recollection.

After the article about consumer, processors and all that messaging jazz, there was a great discussion on Emmett Discord, there were a lot of interesting threads there, but the one that inspired what you read today came from the Ismael Celis question. Ismael was curious about:

Distributing the workload dynamically between processors

And as that’s something I haven’t planned to deliver initially, but it’s a great question, let me put down some notes on my plan around it.

Why would we want to distribute the workload dynamically? What does that even mean?

Let’s start from the classical approach. We can define projections that will build read models based on the upcoming events. In Emmett, it can look like that when using Pongo as the storage tool:

const cartsSummaryProjection = pongoSingleStreamProjection({
  collectionName: shoppingCartsSummaryCollectionName,
  getDocumentId: (event) => event.data.shoppingCartId,
  evolve,
  canHandle: [’ProductItemAdded’, ‘ShoppingCartConfirmed’],
  initialState: () => ({
    status: ‘pending’,
    productItemsCount: 0,
 }),
})

This means that when the projection handles only ProductItemAdded and ShoppingCartConfirmed event types. It’ll insert or update rows in the table based on the shopping cart id from the event data.

We can plug this projection into a projector (the specific type of processor responsible for running projections).

const cartsSummaryProjector = postgreSQLProjector({ 
  processorId: ‘shoppingCartSummary’,
  projection: cartsSummaryProjection,
 });

And plug it into the consumer:

const consumer = postgreSQLEventStoreConsumer({
  connectionString,
  processors: [
    cartsSummaryProjector
  ],
});

Now, consumers will poll the PostgreSQL event store and pipe filtered events by type into our projector.

That’s simple, we know the filtering criteria, as consumers will know which event types their processor(s) can handle.

Things get harder when we don’t know those criteria upfront. When can that happen? What if we were building an e-commerce SaaS product that lets shop owners buy a subscription to run their shops? Then we won’t know all of our tenants upfront. They will register as they go.

What options do we have for such a multi-tenant setup?

The simplest option would be to add a custom filter that allows filtering events on the consumer by the tenant.

Emmett doesn’t support such a feature yet, but it could (and will at some point). We could extend our event metadata to contain the tenant and spin up a consumer for a dedicated tenant.

We could even wrap it into a dedicated function.

function tenantedConsumer(
  connectionString: string,
  tenant: string,
  processors: MessageProcessor[]) {
  return postgreSQLEventStoreConsumer({
    connectionString,
    processors,
    filterBy: (event) => event.metadata.tenant === tenant1,
    projection,
  });
}

Of course, this wouldn’t be fully dynamic, but we could make it dynamic at the infrastructure level, e.g., by passing the tenant ID from an environment variable and spinning up a new Container (e.g., a Kubernetes Pod).

const consumer = tenantedConsumer(
  process.env.PG_CONNECTION_STRING,
  process.env.TENANT_ID,
  [ cartsSummaryProjector ],
);

We could also start a new job from the API endpoint, running it in the existing deployment; there are plenty of options. You can go wild and think about other ways.

Still, logically, they would look more or less like that. Either you’re scaling horizontally and sharding physically, tenants

Or you’re scaling vertically and running multiple tenants inside your box. Then, instead of spinning up new containers, you’re spinning up new jobs (processes, threads, virtual threads, etc.).

That’s nothing special for Emmett; other tools also do this, e.g., Spring Boot Kafka concurrent listeners.

And hey, btw. there’s one more reason why I’m writing about my Emmett design: to let you benchmark your design against it. Even if you’re not planning to use it, then those are considerations for you either as:

internal tooling creator,
user of other OSS tooling, to check how they solved it.

Separating tenants through sharding give you:

better option to scale and align the needs to the specific tenant workload,
makes possible full separation in terms of networking, storage, etc.
can be more costly.

Separating tenants through partitioning the load inside one box:

is usually cheaper,
easier to manage,
doesn’t give you a full separation and can fall into Noisy Neighbour issue.

So there’s no golden rule to choose, it depends on your tooling, needs, etc.

You can, of course, have a mixed solution, so running most of the small tenants in the same box, and the one with bigger security needs gets a special setup. Especially that dynamic split doesn’t have to be only per tenant. You can, e.g., have per region, per product range, per chain, etc.

Also, such a dedicated setup can work for a set number of dynamic options. If you have 1000 tenants, would you spin up 1000 containers or jobs? You can, but just because you can doesn’t mean that you should. The more containers or threads you have, the more you pay for the coordination costs. At some point, it’s better to group processing into a manageable range of containers or threads.

Ok, but how do we know which messages to process where? We could use a consistent hash. I wrote about it in detail in Understanding Kafka’s Consumer Protocol: A Deep Dive into How Consumers Talk to Brokers.

Kafka, by default, partitions its data on the producer side. The topic represents a logical split (e.g. all messages from the E-Commerce module), and the partition represents physical layout. Consumer groups receive messages. Kafka guarantees that precisely one consumer from the consumer group will receive messages from the specific partition. If we have fewer consumers than partitions, then of course, we can get more partitions to handle.

The pseudo code for distributing load to consumers could look as follows:

const partitionId = message.headers.partitionId;
const hash = consistentHashFunction(partitionId);
const consumerId = hash % totalNumberOfConsumersWithinGroup;

In our case, we could use the tenant id as the partition id.

Kafka’s strategy is also simpler than we might need. The partitioning is done on the producer side. The producer sends a message to the topic that already has a certain number of partitions:

const hash =  consistentHashFunction(message.header.recordId);
const partition = hash % totalNumberOfPartitionsWithinTopic;

In our case, if we’d like to allow distribution by any property, we’d need to load the message, read the field we want to partition/shard on, and send it to the appropriate container or thread.

The mechanism can get pretty hefty. You’d need a more sophisticated mechanism to know where to spin up what, how to distribute the load between containers or threads, and to make it resilient. We’re getting into the area of distributed consensus algorithms such as Raft and Paxos.

Do I want to go into this area with Emmett? Definitely not for free! I don’t think that’d be even worth it, as we have mature solutions like Kafka, RabbitMQ, and other messaging systems that implement such algorithms and specialise in that. I’d prefer to make it easier to forward messages to them and let them do the work.

What is definitely on the plate is the second option, which allows partitioned producers. You could define it as such:

const cartsSummaryProjector = postgreSQLProjector({ 
  processorId: ‘shoppingCartSummary’,
  projection: cartsSummaryProjection,
  partitionBy: (event) => event.metadata.tenant,
 });

Then the consumer would poll messages from all tenants, forward them to projectors, and the projectors would internally spin up worker threads per tenant or group them by consistent hashing.

If we add to that distributed locking or detecting conflicting checkpointing to ensure that there’s only one worker instance handling messages for the processor and partition id, then this should be good enough for the majority of cases.

Keeping in mind the flexibility in which you can group (or not) projectors within consumers, you can define your own topologies.

What are your thoughts? How do you deal with such cases? Would you like me to expand more on some cases?

Or maybe you’d like to help me and sponsor my work in this area in Emmett? Then your project could also benefit faster from it!

Cheers!

Oskar

Checkpointing the message processing

Oskar Dudycz — Mon, 08 Dec 2025 15:15:49 GMT

Let’s start by asking you two questions.

What Super Frog has to do with messaging?
When was the last time you wrote if statements in SQL? If it’s been a long time, have you at least seen them? If not, (don’t) worry, you’ll see them today.

Will it be a post about weird SQL usage? Not necessarily.

We’ll talk today about checkpointing our processing.

I’ve started my relationship with computers with games. I still have my Amiga 500. In those days, computers didn’t always have a hard disk. You’ve got a bunch of diskettes with different chapters of the game. Not all of them were simple games; many were quite sophisticated, and it took some time to finish them.

Yet they were dealing with limited diskette space, so it was best if they didn’t have to use any of it. How can you then allow you to stop playing and return to the previous state? Or how to not force you to start from the beginning of the game when you fell from the platform, and well, you died? You died in the game, ofc, that at least you should be able to recover, right?

As mentioned earlier, the limited space on diskettes and the additional complexity that came with it led many game makers to adopt a simple solution: checkpoints with codes.

After you passed a level, you got a code you could type when you started the game, and instead of starting from the beginning, you could go directly to the place where you left off. That worked pretty well for the platform and race car games, since your game’s storyline was always the same, immutable. If you had to go to level 27, the starting point and your character would always look the same. Of course, for RPG and strategy games, that’s a different story.

Surprisingly, this parallel also matches the recovery from a business process failure.

Let’s say we’re using message-based communication to streamline and make it more resilient. We don’t want to make it vulnerable to scenarios where we store information in one system, our process dies, and we don’t manage to notify the other parts.

We’re using Outbox pattern to enable that technically. We’re storing messages in the relational table within the same transaction, updating the state after running business logic. Thanks to that, either both states are updated, and the message is scheduled, or none of it is. We’re getting (eventual) consistency thanks to that.

Now we’re on the receiving end, so where we were in the previous article with the explanation of Consumers, Processors and all that jazz.

Let’s say that we’re using PostgreSQL and our Outbox structure looks as explained in the other article:

CREATE TABLE outbox(
   -- the autoincremented position of the message to respect the order
   position        BIGSERIAL                 PRIMARY KEY,
   -- used to detect gaps in numbering
   transaction_id    xid8 NOT NULL,
   -- unique message id, which can be used for deduplication or idempotency
   message_id       VARCHAR(250)             NOT NULL,
   -- the message type, e.g. `TransactionRecorded`
   message_type     VARCHAR(250)             NOT NULL,
   -- serialised message data, e.g. to JSON
   data             JSONB                    NOT NULL,
   -- diagnostic information on when the message was scheduled
   scheduled       TIMESTAMP WITH TIME ZONE  NOT NULL    default (now())
);

As you can see, besides the message ID, type, and data, we’re also storing the (global) position number and transaction ID (to ensure we don’t skip in-flight transactions that have requested the global position number, read more here for reasoning).

Now we can be polling it with the query like:

SELECT 
     position, message_id, message_type, data
FROM
     outbox
WHERE
     (
          (transaction_id = last_processed_transaction_id
               AND position > last_processed_position)
          OR
          (transaction_id > last_processed_transaction_id)
     )
     AND transaction_id < pg_snapshot_xmin(pg_current_snapshot())
ORDER BY
    transaction_id ASC,
    position ASC
LIMIT 100;

Now, thanks to that, we can have the global ordering guarantee on the receiving end. We’re trading a bit of performance for greater correctness. Not always acceptable, but for internal module communication or forwarding to the messaging system, that’s usually more than enough.

That’s also the place where we’re back to our checkpointing. How do we know the last processed position?

By default, that’s simple, we could either say:

start from the beginning and use -1 or some other hardcoded position,
start from the end and use the position of the last message in the table plus one.

Those cases can be fine when we need to handle all messages (e.g., adding a new process or reading a model), or when we don’t care about the past and need to process the newest notifications.

Still, the reality is just like in the old games, we’d like to start somewhere in the middle, precisely where we left off.

We need one more table for storing our checkpoints. It can look as follows:

CREATE TABLE processor_checkpoints
(
   -- subscription name
   processor_id          VARCHAR(100) PRIMARY KEY,
   -- information about the position of the last processed message
   last_processed_position  INTEGER      NULL,
   -- used to detect gaps in numbering
   last_processed_transaction_id    xid8 NOT NULL
);

Simple stuff: the processor ID should be unique and reflect the processor’s logical name and the last processed position(s).

If that looks simple, then let me follow up with the next set of potentially simple questions:

How to store it?
When to store it?

If your answer is: “just do upsert statement”, then you’re kinda right, but that wouldn’t be simple, it’d be over-simplification. At least if you’d like to run it in production.

Let’s start with how. And, for that, let me bring you now the promised stored procedure with if statements, it’s a bit simplified version from Emmett:

CREATE OR REPLACE FUNCTION store_processor_checkpoint(
  p_processor_id VARCHAR(100),
  p_position BIGINT,
  p_expected_position BIGINT,
  p_transaction_id xid8
) RETURNS INT AS $$
DECLARE
  current_position BIGINT;
BEGIN
  -- Handle the case when p_check_position is provided
  IF p_expected_position IS NOT NULL THEN
      -- Try to update if the position matches p_check_position
      UPDATE processor_checkpoints
      SET 
        “last_processed_position” = p_position, 
        “last_processed_transaction_id” = p_transaction_id
      WHERE “processor_id” = p_processor_id AND “last_processed_position” = p_check_position;

      IF FOUND THEN
          RETURN 1;  -- Successfully updated
      END IF;

      -- Retrieve the current position
      SELECT “last_processed_position” INTO current_position
      FROM processor_checkpoints
      WHERE “processor_id” = p_processor_id;

      -- Return appropriate codes based on current position
      IF current_position = p_position THEN
          RETURN 0;  -- Idempotent check: position already set
      ELSIF current_position > p_expected_position THEN
          RETURN 2;  -- Failure: current position is greater
      ELSE
          RETURN 3;  -- Default failure case for mismatched positions
      END IF;
  END IF;

  -- Handle the case when p_check_position is NULL: Insert if not exists
  BEGIN
      INSERT INTO processor_checkpoints (”processor_id”, “last_processed_position”, “last_processed_transaction_id”)
      VALUES (p_processor_id, p_position, p_transaction_id);
      RETURN 1;  -- Successfully inserted
  EXCEPTION WHEN unique_violation THEN
      -- If insertion failed, it means the row already exists
      SELECT “last_processed_position” INTO current_position
      FROM processor_checkpoints
      WHERE “processor_id” = p_processor_id;

      IF current_position = p_position THEN
          RETURN 0;  -- Idempotent check: position already set
      ELSIF current_position > p_expected_position THEN
          RETURN 2;  -- Insertion failed, row already exists with a greater position
      ELSE
          RETURN 3;  -- Default failure case for mismatched positions
      END IF;
  END;
END;
$$ LANGUAGE plpgsql;

Oooh, even by copy and pasting, I’m already tired; there’s a fair reason why we’re not doing that too often nowadays.

Let me untangle that for you:

We’re trying to update the existing position or insert it if we’re storing it for the first time.
If all went fine, we’re returning 1 as a result to denote complete success.
If we saw that the checkpoint in the database had the same value, we’re returning 0.
If we saw that the checkpoint is different from what was expected and further away from it, then we’re returning 2.
Otherwise, we’re returning 3, which means that the checkpoint is different from the expected and older.

Essentially, by passing the expected position, we can detect whether we:

already handled the specific position,
have some competing instance of our processor handling our data.

That’s why we’re doing this fancy dance with IF statements and a stored procedure.

Detection assumes that we have a global ordering processing guarantee (thus, tricky bits with transaction ID).

It also shows why global ordering is useful.

By detecting that we’ve already handled a specific position, we can skip processing handling idempotency on the processor level.

By detecting that there’s another processor with the same id processing messages, we can make it more resilient and detect the noisy neighbour issue.

How would that look in the code?

async function handleBatch(messageBatch: RecordedMessage[], context: ProcessorContext): Promise {
  const { checkpoint } = messageBatch[messageBatch.length - 1].metadata;

  return context.pool.withTransaction(async (transaction) => {
    for (const message of messageBatch) {
      await context.onMessage(message);      
    };

    // No error was thrown: proceed to store checkpoint of the last processed message
    const result = await storeProcessorCheckpoint(transaction.execute, {
      processorId: context.processorId,
      newCheckpoint: checkpoint,
      lastProcessedCheckpoint: context.lastProcessedCheckpoint,
    });

    if(result.success) {
      await transaction.commit();
    } else {
      // no need to do here, either we already handled it
      // or we have a mismatch of expected and existing checkpoints
      await transaction.rollback();
    }

    return result;
  });
}

type ProcessorContext = {
  processorId: string;
  lastProcessedCheckpoint: bigint | null;
  onMessage: (message: AnyMessage) => Promise;
  pool: ConnectionPool;
}

type BatchHandlingResult =
  | {
      success: true;
      newCheckpoint: bigint | null;
 }
  | { success: false; reason: ‘IGNORED’ | ‘FURTHER’ | ‘OLDER’ };

async function storeProcessorCheckpoint(
  execute: SQLExecutor,
  options: {
    processorId: string;
    newCheckpoint: bigint | null
    lastProcessedCheckpoint: bigint | null;
    partition?: string;
 },
): Promise {
  const { result } = await single(
      execute.command<{ result: 0 | 1 | 2 | 3}>(
        SQL`SELECT store_processor_checkpoint(
            ${options.processorId}, 
            ${options.newCheckpoint}, 
            ${options.lastProcessedCheckpoint}, 
            pg_current_xact_id()
        ) as result;`,
      ),
    );

    return result === 1
      ? { success: true, newCheckpoint: options.newCheckpoint! }
      : { success: false, reason: result === 0 ? ‘IGNORED’ : result === 2 ? ‘FURTHER’: ‘OLDER’ };
};

As you can see, thanks to:

global ordering,
checkpoint detection,
storing checkpoint where our side effects will be stored,
transactional capabilities of our end storage,

We can ensure that the entire batch is processed or not. We could even optimise it by storing the checkpoint first and not processing the business logic if there’s a mismatch, then committing only if the logic succeeds.

We’re getting by that generic idempotence check and detection of the noisy neighbour.

Of course, I still believe that idempotence check should happen on the business logic side. But why not both?

Being able to detect a noisy neighbour can help you automatically stop (or pause) one of the competing consumers and avoid inconsistency conflicts.

What are the tradeoffs of this approach?

This will work if we have a global ordering guarantee. Not many messaging solution gives us such. If we have a subscription-based outbox as explained, event store like Emmett, Marten or KurrentDB, Kafka, this will work, but not necessarily for solutions like RabbitMQ, SQS, Google Pub Sub, etc.
This works best if you have transaction capabilities. Batching updates generally improves performance, but sometimes can lead to long-lived transactions; beware of that. The subscription-based solution with a transaction ID also works best if your transactions are short. If they’re open for a long time, it can cause delays.
Still, even without transactions, it will work fine, as long as you’re fine with having retries more often. If the business logic fails and the checkpoint is not committed, it’ll reprocess the already-handled messages from the previously stored messages. Which means that they can be handled more than once, but you should not lose any messages. You may also not fully benefit from the idempotence check for skipping already handled messages.

As always, the choice is yours.

Still, I hope that this article will show you why:

having a global ordering guarantee can be useful,
why and how to checkpoint your processing, how they relate to level codes from old games like Super Frog,
What are the tradeoffs, and how to consider them,
…and that SQL IF statements are sometimes justified. But don’t go wild with them!

And hey, I also hope that’s not something that you’d like to maintain on your own. There are mature tools to deal with such stuff, like Emmett, which implements this for you.

What are your thoughts? Questions? Concerns?

Cheers!

Oskar

Consumers, projectors, reactors and all that messaging jazz in Emmett

Oskar Dudycz — Mon, 01 Dec 2025 16:02:26 GMT

Did you know that you can build an event store in one hour? I even did it a few times on the conference stage. Actually, it took me usually around 25 minutes; the rest was mistyping, lame jokes and a bit of explanation. See:

Yet, my final thought was: Kids don’t do it at home.

It’s a fun coding exercise, but using the outcome in production? Not as much fun running and maintaining it. Why though? How hard can it be?

Event Sourcing systems have two phases: appending events and processing them afterwards. The write side gets most of the attention in tutorials and talks - commands, deciders, event stores, optimistic concurrency, as you saw, I’m also one to blame.

Providing the guarantees on the write side is relatively simple, especially if you use a database like PostgreSQL as a storage. You need to provide features like:

appending an event at the end of the stream,
reading all events from the stream,
a guarantee of the ordering within the stream,
being able to read your writes,
strong-consistent, atomic writes and optimistic concurrency.

That can be solved with knowledge about transactions, database design, etc. So again, why so hard?

The processing side is where systems often struggle as they grow. This is where the Event Sourcing solution becomes an Event-Driven Messaging tool. And if you’ve read my previous articles, you know that this can be tricky at times.

How do you reliably process events to build read models? How do you trigger side effects without losing messages? How do you scale processing independently from writes? How do you make it performant and run multiple handlers in parallel?

I’ve been working on the message processing architecture in Emmett for a while now. I’ll try to explain how I designed the split between Consumers and Processors, the problems it solves, and the tradeoffs involved.

Why Split Consumers and Processors?

When processing messages, we already know that someone produced them. We’re on the receiving end. Facts are already known; now we need to do something about them.

When we process them, do we care about the source? Typically, we take the information it gathers and reason about it. For instance, when we received an event indicating that a room reservation was made, we may need to send an email with details to the consumer, update the reservations dashboard, and generate a pro forma invoice. We may have specific logic, whether it came from our internal reservation platform or Booking.com, but we know the source from the message payload.

That seems obvious, but it was an important realisation for me. When we’re building a read model in MongoDB, we don’t care if events come from PostgreSQL event store, EventStoreDB, RabbitMQ queue or Kafka topic.

It needs events and the projection logic. Of course, it needs to know the guarantees around: delivery, ordering, idempotency, etc., but besides that? The message’s source doesn’t matter to its logic.

Similarly, a component polling PostgreSQL for messages to publish them doesn’t care what happens to those events - whether they update read models or trigger webhooks is irrelevant to polling logic.

These concerns are orthogonal.

I realised that much of the complexity comes from coupling those two together. We wouldn’t like to change our processing logic because of an internal change in how they’re produced, or vice versa. I concluded that separating them means each can evolve independently. And I came with the initial idea for the split: Consumers and Message Processors.

Consumers are responsible for getting messages from a source and forwarding them to processors. Think of them as the “delivery mechanism.” They handle the “where do events come from” concern. A consumer might connect to:

A PostgreSQL event store, polling the events table,
EventStoreDB, using push-based catch-up subscriptions,
Kafka, consuming from topics,
Any other message source you might have.

Processors are responsible for doing something meaningful with those messages. They handle the “what do we do with messages” concern. A processor might:

Update a read model in PostgreSQL or MongoDB,
Call an external API when certain events occur,
Publish events to Kafka or send webhooks,
Trigger workflow steps or saga operations.

This separation follows Unix philosophy: small, focused components connected by simple interfaces. Each piece does one thing well. You can plug any processor into any consumer. This gives you flexibility that matters in practice:

You can run the same projection logic against different event sources
Adding new processors doesn’t require changing consumer code
You can test processors in isolation with fake event streams
Consumers and processors can be scaled independently

Let me show you how this looks in practice.

How Consumers Work

I deliberately decided to keep Consumers as dumb as possible. A consumer’s entire job is:

Connect to a message source,
Poll messages in batches or subscribe to notifications (depending on the source specifics).
Forward them to all registered processors.
Go back to step 2.

That’s it. No business logic. No complex state management. No decision-making about what to do with messages. Consumers are essentially routers.

Why this simplicity? I’m a simple guy; I like clear boundaries for responsibility. They help me reason about both how to use the tool and how to handle it when things go wrong. When message delivery breaks, you want to know exactly where to look. With a simple consumer, the question is binary: did it deliver messages or didn’t it? There’s no complex interaction between delivery logic and processing logic to untangle at 3 AM.

For EventStoreDB, the consumer creates a single subscription and fans out messages to all registered processors. For PostgreSQL, it polls the message table in batches, handling ordering guarantees (we’ll get to why that later).

Here’s what a basic consumer setup looks like for PostgreSQL :

const consumer = postgreSQLEventStoreConsumer({
  connectionString,
  processors: [
    shoppingCartDetailsProjector,
    customerAnalyticsProjector,
    orderNotificationReactor
  ]
});

await consumer.start();

Set up looks accordingly for the other sources providing options specific for the source, e.g. fthe or EventStoreDB store, you may want to provide the category stream name:

const consumer = eventStoreDBEventStoreConsumer({
  connectionString,
  from: { stream: ‘$ce-roomRservations’, options: { resolveLinkTos: true } },
});

The consumer receives messages, batches them and forwards them to all processors. Each processor handles messages independently - they don’t know about each other, and they don’t need to.

Why Batching Belongs to Consumers

Who decides how many messages to fetch at once? That’s a tricky question. I think that batching can happen both at the consumer and processor levels. The consumer decides the batch size for the polling or receiving to tune the receiving throughput. The processor can either align with it, using those batches as a safe default or diverge to its specifics. Read more on Why you should batch message processing in my other article.

Different message sources have different optimal batch sizes. PostgreSQL might be efficient with 100-row fetches. EventStoreDB subscriptions don’t have built-in batching; they deliver events as they arrive. Kafka has its own batching semantics. These are all source-specific optimisations we should be able to apply without ending up with the lowest common denominator.

Processors, by default, can just receive batches and process them. Then they can decide whether to split batches into smaller chunks, group them into even larger chunks, or process them as single messages. For instance, PostgreSQL can handle random single updates pretty well, whereas Elastic prefers batching updates.

How Processors Work

If consumers are simple routers, processors are where the interesting work happens. They’re the smarter ones in this relationship. A processor is responsible for:

Processing logic: Actually doing something useful with events. Updating a read model, sending an email, calling an API.
Checkpointing: Tracking which messages have been processed. This is crucial - without it, you’d reprocess everything from the beginning every time you restart.
Error handling: Deciding what to do when processing fails. Retry? Skip? Stop everything?
Idempotency: Doing their best to ensure that reprocessing the same event doesn’t cause problems. Of course, still assuming that handlers should be idempotent, read more in my other article,
Backpressure: They need to be able to tell consumers that they cannot process more messages at the moment, and that the consumer needs to slow down delivery.

Those are general promises and common stuff for the message processing logic. Still, there are multiple reasons why you want to process incoming messages:

Read models (projections) transform events into queryable state. For instance, shopping cart events - ProductItemAdded, ProductItemRemoved, ShoppingCartConfirmed - need to become a document showing current items, quantities, and totals. Something your API can quickly return when a user opens their cart.
Reactions trigger side effects after a business fact has happened. When a shopping cart is confirmed, you may want to send a confirmation email, notify the shipment module, and register a new order. These things need to happen, but they’re not part of the core business logic.
Workflows coordinate multi-step processes across multiple streams. An order might involve payment processing, inventory reservation, and shipping coordination - each with its own state and events.
Integration means forwarding events to other systems. Other services in your systems might need to know about orders. External partners might need webhook notifications. You might publish to messaging systems for downstream consumers.

All of those processing needs a bit different ways to handle reliability, ordering, throughput, etc. Also, all tools we integrate with require a different approach: storing the read model in PostgreSQL will be quite different from forwarding a message to Kafka.

I wouldn’t like to handwave all of those specifics and end up with the lowest common denominator. That’s why I decided to group them into the following archetypes:

projectors,
reactors,
workflows,
allow custom message processors to allow people to tune it fully to their needs,
and in the future, stuff like forwarders, web hooks and others we find useful.

All of them should have a unified API that allows them to be plugged into different consumers, while also embracing differences in message processing and target API specifics.

That’s also why each message processing target (PostgreSQL, EventStoreDB, MongoDB, InMemory, Kafka, SQS, etc.) will have its own implementations.

I believe that this focused responsibility, different archetypes, and specific implementations for different tools will strike the right balance between reusability and avoiding the lowest common denominator. We’ll see if that’s not a famous last words.

Read also more in:

The example projector can look like that:

const projection = pongoSingleStreamProjection({
  collectionName: shoppingCartsSummaryCollectionName,
  evolve,
  canHandle: [’ProductItemAdded’, ‘ShoppingCartConfirmed’],
  initialState: () => ({
    status: ‘pending’,
    productItemsCount: 0,
 }),
});


const postgreSQLProjector = postgreSQLProjector({ projection });

const reactor = postgreSQLReactor({
    processorId: ‘order-notifications’,
    canHandle: [’ShoppingCartConfirmed’],
    eachMessage: (event) => 
      emailService.sendOrderConfirmation(event.data.customerId);    
  }

Native implementations of processors

Different storage requirements require different capabilities, and getting proper guarantees might involve deeper knowledge. For instance, Postgres sequences issues can impact your messaging guarantees. Those are cases where, when you’re starting, you might not anticipate. Test environments may not even catch it; you might realise you’re losing business data when you reach production. That’s why it’s, imho, better to have a tool that solves it rather than trying to maintain it on your own, making technical infrastructure something you need to keep working on instead of your business features. How does Emmett solve them? Let’s discuss them briefly. I’ll try to expand in the future posts about the details.

Resilience

What happens if the processor fails? By default, it stops processing. But only this one, the consumer keeps pushing events to the other processors that can continue. Consumer stops when all their processors are inactive.

Why? Consider this scenario: you have two processors, one updating MongoDB and another updating PostgreSQL. MongoDB becomes temporarily unavailable. Should that stop PostgreSQL updates?

Still, failure behaviour is configurable; your message handler can return:

void/ACK: message processed successfully, continue to the next one.
Skip: Skip this message, useful for poison messages that consistently fail
Stop: Stop this processor entirely.

Why have skip separate from ACK? Consider a poison message - a message that causes your processor to fail every time. Without skip, you have two bad options: fail forever (blocking all processing) or ACK it (pretending you processed it). With Skip, you can move it to a dead-letter queue for investigation while continuing to process other messages.

For now, Emmett doesn’t support Dead Letter/Poison Message Queues out of the box, but they will be supported in the future. You could already append those events to some specific stream.

In upcoming releases, we’ll also have configurable retry policies based on error type and other factors. Just like we already have for command handlers (e.g. to retry 3 times with exponential backoff for Optimistic Concurrency error).

There’s no easy answer to when to stop and when to skip poison messages. Neither choice is universally correct. A financial system might need all-or-nothing semantics. A social media feed can tolerate inconsistency between views.

That’s also why you can freely group processors within consumers. Best if they share similar resiliency and desired throughput characteristics. If they’re very different, you can always spin up another consumer for the same source and process it differently.

Checkpointing processing

In Emmett, processors own their checkpoints. Each processor independently tracks the last message it processed. The consumer doesn’t maintain any checkpoint state.

When a consumer starts up, it asks all registered processors for their last processed position and starts polling from the earliest one.

It has several benefits:

Independent progress: Processors can move at different speeds. If your MongoDB projector is fast and your analytics processor can get slow at times, they each track their own progress. The slow one doesn’t hold back the fast one.
Isolated failures: If one processor’s checkpoint storage fails, only that processor is affected. Others continue working.
Easy replay: To rebuild a single projection, you just reset that processor’s checkpoint. No need to coordinate with other processors or manage a global position.
Flexibility: Processors can store checkpoints wherever makes sense - in the same database as their read model, in a separate checkpoint table, or anywhere else.
Capability to redistribute the load. As mentioned in the previous points, if you observe that one of the processors is slower or demands more resources, you can freely deploy it separately in a different consumer, and it’ll start where it left off.

The tradeoff? When a consumer restarts, it might poll events that most processors have already seen. If one processor is significantly behind, all processors receive those events again (they just skip them based on their checkpoints). This is why you should group processors by their typical processing pace - don’t put a real-time dashboard projector and a monthly analytics batch processor on the same consumer.

Backpressure: When Processors Can’t Keep Up

Backpressure occurs when processors can’t process messages fast enough for the consumer to deliver them. This is a real operational concern that needs explicit handling.

Emmett doesn’t support it at the moment, but here’s what I’m thinking about it.

There are several strategies, each with tradeoffs:

1. Ignore backpressure: Consumer keeps polling and pushing regardless of processor state.

Pro: Simple, maximum throughput when processors can keep up
Con: Memory grows unbounded, possible OOM, cascading failures

2. Stop on any slowdown: If any processor signals it’s overwhelmed, stop polling.

Pro: Safe, no resource exhaustion
Con: Slowest processor determines overall throughput

3. Force synchronised pace: All processors must process each batch before the next is fetched.

Pro: All processors stay in sync, predictable memory usage
Con: The Slowest processor becomes the bottleneck for all

4. Slow down ingress: Adaptively reduce polling rate based on processor feedback.

Pro: Balances throughput and stability
Con: More complex, needs tuning

5. Rolling buffer: Buffer messages up to a limit, retry delivery to slow processors.

Pro: Absorbs temporary slowdowns, maximises throughput
Con: Needs memory limits, complex failure handling

Different systems need different strategies. Real-time dashboards might use strategy 1 (drop messages rather than lag). Financial transactions might use strategy 3 (consistency over throughput). Event forwarding to Kafka might use strategy 5 (buffer temporary network issues).

I’m leaning toward making this configurable per consumer, with sensible defaults. The default would be a bounded buffer with adaptive polling slowdown.

Scaling: Current State and Future Plans

For now, the big benefit of having dumb consumers is that you can scale them horizontally. Of course, this works for offset-based solutions like event stores and streaming tools like Kafka. It may not always work for systems that remove the message once it’s handled. Still, current consumers are using only event stores as sources; Kafka will likely come next.

You can group processors into consumers by that, reducing the number of polling jobs (one consumer polls/subscribes to one source).

I already mentioned batching, which should also increase the throughput.

Running multiple instances of the same processor causes conflicts. Both process the same events, update the same read models, and corrupt the state. Emmett already has the basic capability to do distributed locking, but it’s not fully plugged yet. This will come in future releases.

For now, checkpointing can detect whether a newer checkpoint is already stored (which can suggest another processor is running) and stop processing.

The recommended approach is to run the consumer as a separate service from the API. Then you can scale it separately. You can also set replicas=1 for the specific consumer to ensure one instance.

Rebuilding Projections

Event Sourcing enables rebuilding read models from events. Bug in projection? Fix code, rebuild. New read model? Populate from history.

With processor-owned checkpoints, you can either rebuild read model from scratch by:

Stopping the processor.
Delete read model data.
Reset the checkpoint to the beginning.
Restart processing.

Or doing blue greeen by:

Creating a new version of your storage (with Pongo, it’s just adding a suffix or prefix to your collection name).
Start consumer since the beginning.
Check if read models are close enough, and stopthe old processor
Start processing.

In Emmett you have even some syntactic sugar on top of consumers and processors to make this easier:

import { rebuildPostgreSQLProjections } from ‘@event-driven-io/emmett-postgresql’;

const rebuilder = rebuildPostgreSQLProjections({
  connectionString,
  projection: shoppingCartsSummaryProjectionV2
});

await rebuilder.start();

This will spin up a new consumer; other consumers and processors continue normally, with their checkpoints unaffected. You can specify the position from which you want to start, and also whether to truncate the end storage.

We’ll need more metrics like gap detection and distributed locking to make it more plug-and-play.

Wrapping Up

The consumer/processor architecture in Emmett is about making event processing concerns explicit and separable:

Consumers handle delivery - getting events from sources to processors. They’re simple by design. When delivery breaks, you know where to look.

Processors handle processing - doing useful things with events. They own their checkpoints, track their own progress, and handle their own failures.

This separation gives you:

Flexibility to mix and match consumers and processors,
Independent scaling of different processing workloads,
Isolated failure domains,
Easy projection rebuilds,
Testability at multiple levels.

The design makes tradeoffs explicit:

Partial progress over all-or-nothing (configurable soon),
Processor-owned checkpoints over global tracking,
Simplicity in consumers, complexity in processors,
Eventual consistency for async operations.

There’s more to build - distributed locking, partitioning, better backpressure handling. There’s still a lot to do, but I believe the foundation is there, and I know real applications are using it already.

I hope that this is a good food for thought, even if you’re not using Emmett. I’m curious about your thoughts and feedback. I’ll try to tackle those cases in more detail in dedicated articles.

If you have questions, feedback, or would like to help me speed up the planned stuff, come chat in the Emmett Discord. We have a small, but welcoming and awesome community.

Cheers!

Oskar

Requeuing Roulette in Event-Driven Architecture and Messaging

Oskar Dudycz — Mon, 17 Nov 2025 18:35:13 GMT

Welcome to the new week!

I’m always saying that there’s a thin line between good and bad practice, and this thin line is named “Context”.

That’s also true for the (anti-)pattern I’m calling “Requeuing Roulette”. Let’s discuss it today, continuing the “race condition series”:

What’s the Requeuing Roulette? As the name suggests, this technique involves putting a message back into the queue. It also (correctly) suggests that we’re hoping for the best. And sometimes we may be lucky to be true.

The basic primitive for a messaging system is a queue. The producer is putting messages into the queue, and the consumer is getting them on the other end. If everything goes well, the consumer receives them in the order the producer put them (thus, a queue, like a queue in a shop).

If the consumer is not available, the messaging system will try to deliver messages and handle retries for us.

We discussed it in detail in:

Ordering of processing works if we have a single consumer for a single queue. If we have more than one consumer, we lose the ordering guarantee. Why would we want to have more than one consumer? Obviously, to speed up processing. If messages in the queue are not causally correlated, then we can process them in parallel.

What does the smartass “causally correlated” even mean? For instance:

depositing money into a bank account is causally correlated to opening it, as we can’t deposit money if we don’t open it.
depositing money into a bank account is NOT causally correlated to other deposits, as we can deposit as much money as we have (of course, ignoring weird regulations),
money withdrawal is causally correlated to depositing money and other withdrawals, as we need to check the balance, and they may impact it.
Withdrawals and deposits are only causally correlated if they happen on the same bank account; other bank account operations can happen at any time.

You get the idea, aye?

So if we set up a queue to process money transfer events, then it could look as follows:

I assumed that we’re following the advice from the previous article. Besides the event type and payload, we’d also pass the record revision, which represents the logical order of events. It comes from the number incremented with each change. Assuming we’re publishing events after each successful business logic handling, it should be gapless.

You may notice that our queue actually has multiple timelines for each causally correlated message sequence. If we simplify our considerations and assume that all events from a certain account are causally correlated, then we could visualise them as:

It’s fine to process events that are not causally correlated, as by that we’re increasing throughput, not trading off correctness:

But it’s not fine to process messages from the same timeline in parallel, as we may get race conditions when the consumer processing the earlier message will be slower than the one processing the later message.

We actually have two orders in messaging systems:

Queue Order: The order messages are produced to the queue
Processing Order: The order messages are actually consumed and processed.

We already learned that we can detect the out-of-order processing by:

We can also apply techniques like the Phantom record, where we keep data as it comes and take the next steps only when defined conditions are met. And that’s also what I’d recommend in general, but…

But I promised you to talk today about the Requeuing Roulette (anti)pattern, aye? So let’s do it!

If we’re using tools like RabbitMQ, SQS and other classical messaging tooling (so not you Kafka! You’re a streaming or log solution!), then we can put the message back in the queue.

RabbitMQ message ordering documentation states:

Messages published in one channel, passing through one exchange and one queue and one outgoing channel will be received in the same order that they were sent. RabbitMQ offers stronger guarantees since release 2.7.0.
Messages can be returned to the queue using AMQP methods that feature a requeue parameter (basic.recover, basic.reject and basic.nack), or due to a channel closing while holding unacknowledged messages. Any of these scenarios caused messages to be requeued at the back of the queue for RabbitMQ releases earlier than 2.7.0. From RabbitMQ release 2.7.0, messages are always held in the queue in publication order, even in the presence of requeueing or channel closure.
With release 2.7.0 and later it is still possible for individual consumers to observe messages out of order if the queue has multiple subscribers. This is due to the actions of other subscribers who may requeue messages. From the perspective of the queue the messages are always held in the publication order.

The last paragraph seems promising, as it suggests the message will be put back before the next messages, since it was placed in the queue.

Unfortunately, it’s only a best effort. Another place in the documentation states:

When a message is requeued, it will be placed to its original position in its queue, if possible. If not (due to concurrent deliveries and acknowledgements from other consumers when multiple consumers share a queue), the message will be requeued to a position closer to queue head.

So in the worst case, it can even end up like that:

Then the fun will begin. Now we have two messages that we already see will be handled out of order, and we’ll need to requeue them, hoping they land after the message we need to process as the next one. What if we have multiple messages like that? What if our consumer randomly fails when processing our message with revision 12, and we need to requeue it again? How lucky will it be with Requeuing Roulette?

As you can see, the more correlated our messages are and the more we’d like to parallelise them, the more likely we are to face Requeuing Roulette. Typically, we put messages into the same queue so we can process them in order.

When Requeuing Roulette is helpful?

It can be useful if:

We want to get the best parallelism for message processing, and ordering best effort is good enough for us.
Our messages are most of the time not causally correlated or…
Events for the same records/processes are not quickly published one after another, so we could safely retry message before the next one will arrive,
Our consumers are stable, not failing too often.

As you see, those assumptions can be fragile and classical famous last words.

Of course, we can use one of the techniques like:

RabbitMQ routing key, correlation id,
AWS SQS message group id, visibility timeout,
Azure Service Bus sessions,
etc. see Ordering, Grouping and Consistency in Messaging systems for details.

Making this trade-off more in favour of the ordering guarantee or parallelism may reduce it to an acceptable level, but you need to be aware of the risk of an unexpected traffic spike or a different message distribution than you expected.

Dangers of Requeuing Roulette

Even when order doesn’t matter, requeueing has a hidden cost that becomes visible under load.

Suppose you reject the message with requeue set to true. In that case, it can be redelivered to your consumer almost instantly, resulting in a very high workload, since your consumer will reject it again.

Let’s say you have a message that fails because a downstream service is down. You requeue it. It immediately returns to the consumer (maybe even the same one), fails again, and is requeued. This can happen hundreds of times per second. It can also swamp the slow consumer, preventing it from even recovering.

In the worst case, your CPU can be spent processing and requeueing the same 10 messages over and over, while thousands of processable messages sit behind them in the queue (because RabbitMQ will try to put requeued messages before the next messages).

What about Kafka?

Well, in Kafka, this issue doesn’t exist as Messages with the same record key go to the same partition, maintaining order within that partition while allowing parallel processing across partitions.

So Kafka, for the win? Hold your horses!

Only one consumer from the consumer group can handle a specific partition. So, within a single partition, parallelisation isn’t possible. If we map the RabbitMQ queue to Kafka’s partition, then the conclusion can be that Kafka solved it by removing this feature.

Also, when we consume a message from the classical messaging system (like RabbitMQ), it will be removed from the queue. In a streaming solution like Kafka/Pulsar, etc., they will remain in the log until the retention policy kicks in and drops old messages from the partition.

Kafka maintains the offset of the last processed message in each topic partition. You don’t need to requeue messages; you can just rewind the offset to an older position when you want to reprocess messages.

TLDR

The “requeueing roulette” is a symptom of trying to solve a distributed systems problem with a technical solution.

The requeueing roulette is seductive because it promises something impossible: maintaining strict order in a distributed, concurrent system without sacrificing throughput. It’s trying to cheat the fundamental trade-offs of distributed systems.

Still, cheating can take us far enough, but there’s always a danger that we’ll be caught and handcuffed.

If you’re considering using Requeuing Roulette, then consider the other techniques I described in previous articles. I’d treat Requeuing Roulette as a temporary solution and a tradeoff.

In my opinion, if you decide to use it, then the question isn’t whether you’ll abandon it, but how much pain you’ll endure before you do.

The real skill isn’t in making requeueing work - it’s in understanding your actual ordering requirements and choosing the most straightforward solution that meets them. Often, that means accepting that perfect ordering is neither necessary nor worth its cost, especially in the long term.

Read also more in:

Cheers!

Oskar

Handling Events Coming in an Unknown Order

Oskar Dudycz — Mon, 03 Nov 2025 20:04:01 GMT

Welcome to the next week!

After the last article on Dealing with Race Conditions in Event-Driven Architecture with Read Models, I got such a question from Ben:

You described the scenario where you know what events you should receive, just not the order. But what if you don’t know that? For example, you get an ItemRemovedFromCart event, but the item doesn’t exist in your view of the current state of the cart. Is it an invalid event? Or is there an ItemAddedToCart event that hasn’t come through yet?

That’s a good question, and good questions usually require more depth to give a precise answer. That’s what we’re here for!

Let’s follow up and discuss how to determine whether we have complete information for our events!

What we learned so far

Communication in messaging systems works like a department store. It has multiple cash registers and separate queues for each of them. You can only guess which person will be handled in a specific queue: first in, first out. Between queues, you only know that the slowest will be the one you’re standing in.

Of course, you could put a single cash register and a single queue, and everything would be sequential. Such a setup can work for small groceries with few customers. Tho, for a supermarket, it’d end up with an extremely long waiting queue.

You can think about a single module as a single cash register in a department store or a small grocery store. Inside it, you can get strict ordering of processing, but not in the relationship with the outside world. For instance, you may know that on Monday morning, you’re getting the fresh fruits delivery, and at noon, you’re getting the dairy delivery. And it’s typically like that, but from time to time, because of the fruit delivery delay, you may get your fresh dairy first.

Is it an issue? Not a huge one, as you just want them to come asap so you have fresh stuff to sell. When would it be an issue? If you were running the Milk Shake Cafe and needed both to make your special strawberry shake recipe.

In many systems, ordering is not a key concern, especially when we partition the workload. The issue may arise when we need to correlate separate actions.

In the last article, we showed the payment verification workflow. To make the final decision, we needed to correlate data from the external payment gateway with our own modules, which calculate fraud scores, check limits, and assess risk. Only after receiving data on available merchant limits and the fraud assessment score could we make the final decision. Those pieces of information could return to us at different times and in a different order.

To resolve it, we were just gathering and aggregating data as they went. Then, after each step, we checked whether we now have all the data. If we had, we were making the final decision; if not, we were storing it as is, assuming that at some point, data would arrive.

If we model that as the workflow, then it’d look like that:

function decide(
  current: PaymentVerification | null,
  event: PaymentVerificationEvent
):
  | PaymentVerification
  | { document: PaymentVerification; events: VerificationEvent[] } {
  current = current ?? {
    paymentId: event.paymentId,
    initialState,
  }

  switch (event.type) {
    // (...) other event handlers
    case “MerchantLimitsChecked”: {
      const updated = {
        ...current,
        merchantLimits: {
          withinLimits: event.withinLimits,
          dailyRemaining: event.dailyRemaining,
          checkedAt: event.checkedAt,
        },
        lastUpdated: event.checkedAt,
      };

      return tryCompleteVerification(updated, event);
    }
    case “FraudScoreCalculated”: {
      if (
        current.fraudAssessment &&
        event.calculatedAt <= current.fraudAssessment.assessedAt
      )
        return current;

      const updated = {
        ...current,
        fraudAssessment: {
          score: event.score,
          riskLevel: event.riskLevel,
          assessedAt: event.calculatedAt,
        },
        lastUpdated: event.calculatedAt,
      };

      return tryCompleteVerification(updated, event);
    }
  }
};

And:

function tryCompleteVerifications(
  current: PaymentVerification,
  event: PaymentVerificationEvent
):
  | PaymentVerification
  | { document: PaymentVerification; events: VerificationEvent[] } {
  // Ignore if we already made decision
  if (current.decision)
    return current;

  // Check if we now have BOTH critical pieces
  if (!current.fraudAssessment || !current.merchantLimits)
    // Don’t have both yet - stay in processing
    return {
      ...current,
      status: “processing”,
      dataQuality: “processing”,
    };

  const decision =
    current.fraudAssessment.riskLevel === “high”
      ? {
          approval: “declined”,
          reason: “High fraud risk”,
          decidedAt: event.checkedAt,
        }
      : !current.merchantLimits.withinLimits
      ? {
          approval: “declined”,
          reason: “High fraud risk”,
          decidedAt: event.checkedAt,
        }
      : {
          approval: “approved”,
          reason: “Verified”,
          decidedAt: event.checkedAt,
        };

  return {
    document: {
      ...current,
      status: decision.approval,
      decision,
    },
    events: [
      {
        type: “PaymentVerificationCompleted”,
        data: decision,
      },
    ],
  };
};

This works fine, as we know precisely which steps need to happen, so we know what we’re waiting for. And that’s how we made the full loop to Ben’s question. What if we didn’t know which steps we’re waiting for?

How to know what we don’t know?

Let’s have a look at the case brought by Ben: the e-commerce flow. First, we complete the shopping cart by adding and removing items, then we confirm it. The example event flow could look as follows for the online food ordering:

ItemAddedToCart     (cartId: 1, name: Pizza Napoletana) 
ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
CartConfirmed       (cartId: 1, confirmedAt: 2025-11-03 11:44:27)

We see here that someone added the first Pizza, then maybe accidentally added it again, corrected their mistake, and confirmed the order.

Then, if that was an online ordering system and we had it integrated with the kitchen ordering, then we could get those events in a different order, for instance:

ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
CartConfirmed       (cartId:1, confirmedAt: 2025-11-03 11:44:27)
ItemAddedToCart     (cartId: 1, name: Pizza Napoletana) 
ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)

We see that someone removed one Pizza from their shopping cart, which suggests that some information is missing. When we get a confirmation event, we still know that there’s more to come, as an order with a removed item doesn’t make sense. The same goes for the information that one pizza was added; when we correlate it with the removal event having the same cart identifier, we still see zero items in the shopping cart. Once we get the next event, we will finally know that we have more than one item in our shopping cart.

Can we then proceed? Maybe yes and maybe no. For this particular order, it’d be correct, but what if our real order:

ItemAddedToCart     (cartId: 1, name: Pizza Napoletana) 
ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
ItemAddedToCart     (cartId: 1, name: Spaghetti Carbonara)
ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
CartConfirmed       (cartId:1, confirmedAt: 2025-11-03 11:44:27)

Also, since messaging systems retry to ensure delivery, how would we know that those “doubled” events for adding or removing are actually distinct events and not just retries?

For instance, in such a delivery case:

ItemAddedToCart     (cartId: 1, name: Pizza Napoletana) 
ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
CartConfirmed       (cartId:1, confirmedAt: 2025-11-03 11:44:27)
ItemAddedToCart     (cartId: 1, name: Spaghetti Carbonara)
ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)

Let’s discuss a few strategies to deal with that!

External vs Internal events

Doctor, it hurts when I bend my arm this way.
Then don’t bend it this way

One of the most common mistakes we learn too late is separating our events into internal and external (or private and public). Internal information can and should be more granular. We need it to be precise in capturing the business context and making our decision.

Yet, other parts of our system don’t need to know all of that. Is the kitchen interested in the details of all the changes procrastinating customer made to their shopping cart? No, they just want the final information on which meal they need to prepare.

So in our example, if we published to the outside world just:

CartConfirmed  {
    cartId: 1, 
    items: [
        { name: Pizza Napoletana }
    ]
    confirmedAt: 2025-11-03 11:44:27
}

Such a type of event is also called a Summary Event. We should not mistake it with the latest state. It’s still an event because it tells what has happened business-wise. It gathers all the information needed for other modules and summarises the changes. And no more than that. It should still be as small as possible and expose only the information that other modules need. It’s a contract made between different teams. I wrote about it in detail Internal and external events, or how to design event-driven API.

We can define such an internal event API as:

type ItemAddedToCart = {
  type: ‘sc:int:ItemAddedToCart’;
  data: {
    cartId: string;
    productItem: ProductItem;
  };
};

type ItemRemovedFromCart = {
  type: ‘sc:int:ItemRemovedFromCart’;
  data: {
    cartId: string;
    productItem: ProductItem;
  };
};

type CartConfirmed = {
  type: ‘sc:int:CartConfirmed’;
  data: {
    cartId: string;
    confirmedAt: Date;
  };
};

type ShoppingCartEvent =
  | ItemAddedToCart
  | ItemRemovedFromCart
  | CartConfirmed;

interface ProductItem {
  productId: string;
  quantity: number;
}

and public as:

type CartOpened = {
  type: ‘sc:ext:CartOpened’;
  data: {
    cartId: string;
    openedAt: Date;
  };
};

type CartConfirmed = {
  type: ‘sc:ext:CartConfirmed’;
  data: {
    cartId: string;
    productItems: { productId: string; quantity: number }[];
    confirmedAt: Date;
  };
};

type ShoppingCartExternalEvent = CartOpened | CartConfirmed;

As you see, we can even have more than one summary event, and not even be one-to-one with an internal event. Maybe we also have an analytics module that analyses how long it takes the user to make a final decision after adding the first product. Then we may decide to expose such an event, hiding the details of the internal flow. We’re also defending ourselves and minimising the need for versioning when flow changes.

Ok, but how to map internal events into external?

We can enrich them using such a function:

import type { ShoppingCartExternalEvent } from ‘./shoppingCart.external’;
import type { ShoppingCart, ShoppingCartEvent } from ‘./shoppingCart.internal’;

const enrich = (
  event: ShoppingCartEvent,
  state: ShoppingCart | null,
): ShoppingCartExternalEvent | [] => {
  switch (event.type) {
    case ‘sc:int:ItemAddedToCart’:
      return state == null
        ? {
            type: ‘sc:ext:CartOpened’,
            data: {
              cartId: event.data.cartId,
              openedAt: new Date(),
            },
          }
        : [];
    case ‘sc:int:CartConfirmed’:
      return {
        type: ‘sc:ext:CartConfirmed’,
        data: {
          cartId: event.data.cartId,
          productItems: state?.productItems ?? [],
          confirmedAt: event.data.confirmedAt,
        },
      };
    default:
      return [];
  }
};

We can then subscribe to internal events in our module, load the state (best to build it from events if we’re using Event Sourcing), and publish enriched events externally.

If we’re using messaging, this means also separating queues/topics. If you’re using Kafka both for internal and external communication, then you should separate topics and have two different topics for outgoing communication, e.g.:

‘carts:events:int’
‘carts:events:out’.

Similarly, for RabbitMQ or similar tools, you should have separate queues for internal and external communications.

This is important, as you can now:

Publish messages that other modules need, decreasing the number of issues with ordering,
Have enrichment as an anti-corruption layer for your internal process changes,
You can have different scaling capabilities for internal and external events. Maybe for internal, you don’t even need a messaging system, maybe outbox or event store subscriptions will be enough? Maybe you could cut costs using AWS SQS for internal communication and AWS Kinesis for cross-module?
You can now define different security for those topics and retention policies.

Sweet, right?

It’s not me, it’s them

Maybe it’s sweet enough for you, but you may also say:

But Oskar, it’s not me, it’s them. If I was responsible for that, I’d go this way, but I can’t change it how messages are published.

I could handwave it and say I pity you, but well, this actually can happen. Let’s see what else we could do about it.

The first idea could be: Let’s add timestamps!

Let’s see how it looks for our example:

11:40:10 - ItemAddedToCart     (cartId: 1, name: Pizza Napoletana) 
11:40:10 - ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
11:42:13 - ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
11:43:18 - ItemAddedToCart     (cartId: 1, name: Spaghetti Carbonara)
11:44:23 - ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
11:44:27 - CartConfirmed       (cartId: 1)

And the out of order delivery:

11:40:10 - ItemAddedToCart     (cartId: 1, name: Pizza Napoletana) 
11:40:10 - ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
11:44:23 - ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
11:44:27 - CartConfirmed       (cartId: 1, confirmedAt:2025-11-03 11:44)
11:43:18 - ItemAddedToCart     (cartId: 1, name: Spaghetti Carbonara)
11:42:13 - ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)

Would that help? No, because how would we know, based on timestamps, that there will be two more events after confirmation? Timestamps only tell us when a certain operation happens. They could help us order items that were delivered, but they won’t help us know what we’re missing. Because how do we know that there’s a gap in our knowledge? Within a minute, one could do nothing and order or remove a few more items. Also, timestamps might work if the data is coming from the same node, but we don’t have any guarantees across nodes. Read more about clock drift.

What we actually need is the logical clock. One that increments after each operation. So something like:

1 - ItemAddedToCart     (cartId: 1, name: Pizza Napoletana) 
2 - ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
3 - ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
4 - ItemAddedToCart     (cartId: 1, name: Spaghetti Carbonara)
5 - ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
6 - CartConfirmed       (cartId: 1)

If we had such, then our delivery would look as follows:

2 - ItemAddedToCart     (cartId: 1, name: Pizza Napoletana) 
1 - ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
5 - ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
6 - CartConfirmed       (cartId: 1, confirmedAt: 2025-11-03 11:44:27)
4 - ItemAddedToCart     (cartId: 1, name: Spaghetti Carbonara)
3 - ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)

If this number were monotonic and gapless, we’d know that if the event has number 3, then we have completeness of information if we received three events; if not, then we’re missing something.

We still need to define the completion criteria and determine, from a business perspective, where we can make a decision or proceed to the next step. Here we know that we can start preparing a meal when the shopping cart is confirmed.

In this case, we got the events in the following order: 2, 1, 5, 6.

We know we’re missing events 3 and 4, so we need to wait for them. Only when we receive them can we proceed. Ok, but how to do it?

What if we kept a list of pending events in our data model? Let’s try that!

Our kitchen order could look as follows:

type KitchenOrder = {
  orderId: string;
  productItems: ProductItem[];
  status: ‘Incomplete’ | ‘Ready’ | ‘InPreparation’;
};

type KitchenOrderCommand =
  | {
      type: ‘AddItem’ | ‘RemoveItem’;
      productId: string;
      quantity: number;
    }
  | {
      type: ‘Confirm’;
      orderId: string;
    };

type ProductItem = {
  productId: string;
  quantity: number;
};

When storing it, we could store it with additional metadata:

type DocumentWithPendingCommands = State & {
  metadata: {
    lastProcessedRevision: number;
    pendingCommands: PendingCommand[];
  };
};

As you see, besides the regular data, we have two other properties: pending commands and last processed revision.

You can think about pending commands as your git repository on your local disk. It contains the list of all operations that you’ll eventually commit. The rest of the data is like the remote git repository. They will be updated when you push your changes there. Then, the last processed revision will be updated with the revision of the last applied command.

The code for that workflow could look as follows:

function handle(
  event: ShoppingCartEvent,
  document: DocumentWithPendingCommands<
    KitchenOrder,
    KitchenOrderCommand
  > | null,
): DocumentWithPendingCommands {
  const { metadata, ...data } = document ?? {
    metadata: {
      lastProcessedRevision: 0,
      pendingCommands: [],
    },
  };

  const state: KitchenOrder = {
    orderId: event.data.cartId,
    productItems: [],
    status: ‘Incomplete’,
    ...data,
  };

  const updated = {
    ...state,
    metadata: {
      lastProcessedRevision: metadata.lastProcessedRevision,
      pendingCommands: [
        ...metadata.pendingCommands,
        mapToPendingCommand(event),
      ],
    },
  };

  return handlePendingCommands(updated, decide);
}

It takes the current state of the kitchen order. If it doesn’t exist, then we need to set it up with default data. We’re also appending a pending command that’s built from an event.

function mapToPendingCommand(
  event: ShoppingCartEvent,
): PendingCommand {
  switch (event.type) {
    case ‘sc:int:ItemAddedToCart’:
      return {
        type: ‘AddItem’,
        productId: event.data.productItem.productId,
        quantity: event.data.productItem.quantity,
        metadata: {
          revision: event.metadata?.revision,
        },
      };
    case ‘sc:int:ItemRemovedFromCart’:
      return {
        type: ‘RemoveItem’,
        productId: event.data.productItem.productId,
        quantity: event.data.productItem.quantity,
        metadata: {
          revision: event.metadata?.revision,
        },
      };
    case ‘sc:int:CartConfirmed’:
      return {
        type: ‘Confirm’,
        orderId: event.data.cartId,
        metadata: {
          revision: event.metadata?.revision,
        },
      };
  }
}

Why not use a regular event here? We could. The benefit is that then we’d have all the data stored as it came. This could make troubleshooting and correction easier. Downside? We’re coupling the event with our internal business logic. Also, keeping the whole event payload increases the size of our data. The choice is yours.

As you see, we’re taking revision from the event metadata. We’ll get to that later, how to fill it on the producer side. Now, let’s focus on the workflow.

Let’s see what processing pending items looks like:

function handlePendingCommands(
  document: DocumentWithPendingCommands,
  decide: (command: Command, state: State) => State,
): DocumentWithPendingCommands {
  const { metadata, ...data } = document;

  const commandsToHandle = getCommandsReadyToHandle(
    metadata.pendingCommands,
    metadata.lastProcessedRevision,
  );

  // Nothing to do see here, please disperse
  if (commandsToHandle.length === 0) return document;

  let state = data as State;
  for (const command of commandsToHandle) {
    state = decide(command, state);
  }

  const lastCommand = commandsToHandle[commandsToHandle.length - 1];

  return {
    ...state,
    metadata: {
      lastProcessedRevision: lastCommand.metadata.revision,
      pendingCommands: metadata.pendingCommands.filter(
        (a) => a.metadata.revision > lastCommand.metadata.revision,
      ),
    },
  };
}

We need to get the commands ready to be handled. For instance, if we already had commands with the following revisions 2, 6, 1, 4, and now we got an action with number 3, then that means:

We can process actions 1, 2, 3, 4
Action 6 remains, as we’re missing 5.

The filtering can look as follows:

function getCommandsReadyToHandle(
  pending: PendingCommand[],
  lastProcessedRevision: number,
): PendingCommand[] {
  return (
    [...pending]
      // filter out commands that have already been processed
      .filter((cmd) => cmd.metadata.revision > lastProcessedRevision)
      // sort by revision to ensure correct order
      .sort((a, b) => a.metadata.revision - b.metadata.revision)
      // only take commands that are consecutive in terms of revision
      .reduce[]>((acc, command) => {
        const lastRevision = acc[acc.length - 1]?.metadata.revision;

        return !lastRevision || command.metadata.revision === lastRevision + 1
          ? [...acc, command]
          : acc;
      }, [])
  );
}

When we filter them out and have actions to process, we need to run the actual logic for each of them. For our case this could look like that:


function decide(
  command: KitchenOrderCommand,
  order: KitchenOrder,
): KitchenOrder {
  switch (command.type) {
    case ‘AddItem’:
    case ‘RemoveItem’: {
      if (order.status !== ‘Incomplete’) return order;

      const updatedItems = new Map(
        order.productItems.map((item) => [item.productId, item.quantity]),
      );
      const current = updatedItems.get(command.productId) ?? 0;

      const multiplier = command.type === ‘AddItem’ ? 1 : -1;
      const updated = current + multiplier * command.quantity;

      if (updated > 0) {
        updatedItems.set(command.productId, updated);
      } else {
        updatedItems.delete(command.productId);
      }

      return {
        ...order,
        productItems: Array.from(updatedItems.entries()).map(
          ([productId, quantity]) => ({
            productId,
            quantity,
          }),
        ),
      };
    }
    case ‘Confirm’:
      if (order.status !== ‘Incomplete’) return order;

      return {
        ...order,
        status: ‘Ready’,
      };
  }
}

type KitchenOrderCommand =
  | {
      type: ‘AddItem’ | ‘RemoveItem’;
      productId: string;
      quantity: number;
    }
  | {
      type: ‘Confirm’;
      orderId: string;
    };

After processing each command, we’re returning the document with the updated state, filtered-out processed commands, and the last processed revision set to the revision of the last processed command.

Not that big a hassle as it may seem, but…

What’s revision and how to get it?

A revision needs to be created on the producer side. The best is to use Optimistic Concurrency for that. If you don’t know what optimistic concurrency or locking is, then you should. Check my intros:

Essentially, if you’re:

using a typical implementation of optimistic concurrency where each record/document change increments its version/revision,
publishing an event after each business operation.

Then you can use this incremented state revision and pass it in your events metadata. Having that, you’d know precisely on which revision it was recorded and get gapless monotonic numeration. It’d also ensure that each message is produced in a certain order, as operations on the specific record will be sequential. Read also more on how revision can help in Dealing with Eventual Consistency and Idempotency in projections.

But Oskar, what if I have more than one record?

Well, then you either need to store multiple revisions. But this won’t help if you need to correlate data between them, as revision is monotonic and gapless for the specific record.

What about global positions? Well, they’re useful for knowing the order of things, but they won’t help here, as they’re monotonic but may have gaps.

TLDR

Proper modelling in Event-Driven Architecture can spare you a lot of complicated implementation tricks.

If you

define essential events for your process,
ensure that they have completeness of information,
shape contracts and communication between modules, respecting the internal and external split.

Then, when things get easier to handle, we can define conditions that tell us when to take action.

Still, sometimes we may:

have strict ordering needs,
be using a queue that doesn’t give us an ordering guarantee,
need to adjust to the other teams.

Then, using revision can be a decent option to solve things in an organised way.

If you’re dealing with such issues, I’m happy to help you through consulting or mentoring. Contact me and we’ll find a way to unblock you!

Read also more in:

Cheers!

Oskar

Dealing with Race Conditions in Event-Driven Architecture with Read Models

Oskar Dudycz — Mon, 20 Oct 2025 18:02:36 GMT

My events came out of order! What should I do?!

Are you familiar with the term “phantom record” and its benefits? No? Let me explain it to you today.

Everyone has a plan until they get punched in the mouth. Design, architecture, and modelling are important, but it’s the actual code that reaches production. That’s also the place where we see all those nasty issues that we haven’t foreseen, like: race conditions, eventual inconsistency, idemNotency, etc.

We realise that our BBC Architecture is just a Box-Box Cylinder on paper; in reality, things get messy. We go from a 2D layout to a 3D or even a 4D view.

We see that our business processes are not so linear and predictable as we draw them. Of course, we have somewhere in the back of our mind that subprocesses can go in parallel, but somehow our tests go down the happy path. Then bugs and incidents come in.

We learn the hard way that real processes have delays; they run in parallel. For instance, when you process a payment, fraud checking, risk assessment, and merchant validation happen simultaneously. That’s efficiency, not a flaw.

And the sad part is that it’s our role to provide this efficiency. Yelling at event-driven clouds won’t help.

Race Conditions in EDA

When we distribute our systems, we’re getting better isolation of failure, we can deploy changes at a different pace, but… But we also get communication going at various paces. We can no longer get a unified view of the business actions in the order of appearance. Now each service has its own linearity.

To integrate services predictably, we’re using messaging tools. They provide us with durability, retries when the recipient is unavailable, and ensure that information flows and data are delivered effectively. Still, they’re not magical creatures; they have their limits. We can’t cheat physics.

Between one service and the other, we’re putting in a queue. That’s the place where we can achieve an ordering guarantee. Typically, one queue, one consumer means that we can achieve ordering. Still, not all tools are giving us that.

Out of order issues may happen when the RabbitMQ queue has multiple consumers racing for messages. When we add consumers, we get better throughput, but we’re risking ordering issues.

Out of order issues may also happen when you’re using tools like SQS or Google PubSub, which only guarantee best-effort ordering.

Or when your outbox pattern deletes processed messages and loses sequence.

Or when network delays shuffle carefully ordered streams.

It’s safe to say that you won’t get any ordering guarantee between different queues. Since queues represent communication flows between modules, we should not assume strict ordering in cross-module communication.

Events vs Rumours?

Continuing with the payment verification process example: Fraud module publishes to one queue. Risk assessment for another. The payment gateway to a third. Messages arrive at your module in the order your system received them, not the order they were created. Fraud detection service flags a high-risk payment at 10:00:01. The payment initiation event from the gateway, created at 10:00:00, arrives at 10:00:02. The fraud score arrives before the payment exists in your system. You might think sorting by timestamp solves this. It doesn’t. Clock skew between services means timestamps lie. Each service node has its own time. It may be similar, but if you have a system with high throughput, the skew may be significant enough, causing more issues than actual help.

But, well, that’s the same case when you’re getting information from the outside world. You can get multiple friends sending you some news, you can get them at different paces, and read them in a different order. You may get obsolete news and then newer, but it can go the other way round. You’re only sure of which you received those messages; to get the truth, you need to correlate that information and do fact-checking based on some rules (e.g. which source is more reliable, newer, etc). Then you can deduce the actual information.

We’re saying that in the event-driven world, events are facts. But that’s just half the truth.

Those events that we store/publish are facts for us, or at least represent the current state of our knowledge.

The events from external systems are rumours at best. We need to interpret them to make them (our) facts.

Ok, but what can we do? Sit and cry?

Of course, we can do more. We’ll discuss today a simple technique with Read Models that can take you far enough.

Let’s say that you’re using Event Sourcing in your system. You take all the messages coming from other systems, and you store them in your event store.

Event Sourcing assumes you can rebuild state by replaying events in sequence. When FraudScoreCalculated arrives before PaymentInitiated, the payment doesn’t exist. You can’t apply a fraud score to nothing. You realise that when your carefully designed domain model throws exceptions. This should never happen, aye?

This isn’t specific to Event Sourcing. Even if you don’t use it but just update read models directly from events, you have the same problem. Your read model update handler for FraudScoreCalculated looks for a payment document to update. No document exists. The update fails.

Such issues are a recurring theme in my consulting, as they are a common concern for my customers. My advice is usually: Don’t try to fight them.

Store data as it arrives and “denoise” on your side. Interpret them and save your own “facts”. Acknowledge the need for a split between internal and external events. Use an Anti-Corruption Layer (ACL) pattern to protect from external chaos.

Read models as Anti-Corruption Layer

Read models can be used as such an ACL. A read model document can have a partial state, with optional fields that are filled as events arrive. You can process each event, updating the fields it can, ignoring those it can’t, and making decisions based on the available data.

Let’s explore how to apply this in practice using our payment orchestration scenario. Let’s say we’re gathering information about the payment verification process, displaying its state, and performing follow-up operations once the process is complete.

When we trigger payment, we need to correlate data from the external payment gateway with our own modules, which calculate fraud scores, check limits, and assess risk.

We could define events using TypeScript, as:

type PaymentVerificationEvent =
  | PaymentInitiated
  | FraudScoreCalculated
  | RiskAssessmentCompleted
  | MerchantLimitsChecked
  | PaymentCompleted
  | PaymentDeclined;

Having

type PaymentInitiated = {
  type: “PaymentInitiated”;
  data: {
    paymentId: string;
    amount: number;
    currency: string;
    gatewayId: string;
    initiatedAt: Date;
  };
};

type FraudScoreCalculated = {
  type: “FraudScoreCalculated”;
  data: {
    paymentId: string;
    score: number;
    riskLevel: “low” | “medium” | “high”;
    calculatedAt: Date;
  };
};

type RiskAssessmentCompleted = {
  type: “RiskAssessmentCompleted”;
  data: {
    paymentId: string;
    riskScore: number;
    factors: string[];
    assessedAt: Date;
  };
};

type MerchantLimitsChecked = {
  type: “MerchantLimitsChecked”;
  data: {
    paymentId: string;
    withinLimits: boolean;
    dailyRemaining: number;
    checkedAt: Date;
  };
};

type PaymentCompleted = {
  type: “PaymentCompleted”;
  data: { paymentId: string; completedAt: Date };
};

type PaymentDeclined = {
  type: “PaymentDeclined”;
  data: { paymentId: string; reason: string; declinedAt: Date };
};

Let’s say the fraud system flagged the payment as high-risk before it even existed in your system. Approval happened before the risk assessment was completed. The example race condition can look as follows:

10:15:32.123 - FraudScoreCalculated (score: 85, high risk)
10:15:32.145 - PaymentInitiated (amount: $500)
10:15:32.167 - PaymentCompleted (approved by automated system)
10:15:32.201 - RiskAssessmentCompleted (risk: medium)
10:15:32.234 - MerchantLimitsChecked (within limits)

If we assume that things go as on the whiteboard, and handlers can’t process events for non-existent payments, then we’ll be the ones who get punched in the mouth by reality. They will fail to find documents to update.

The first step is to embrace that we have a problem. Actually, it’s not a problem, but rather a challenge - a scenario we just need to support.

We can start by defining a document that represents a payment’s verification state. The document should include optional fields since you can’t predict which event will arrive first. The projection function processes each event and updates this document, creating it if necessary.

It can look as follows:

type PaymentVerification = {
  paymentId: string;
  payment?: Payment;
  fraudAssessment?: FraudAssessment;
  riskEvaluation?: RiskEvaluation;
  merchantLimits?: MerchantLimits;
  decision?: Decision;
  status: “unknown” | “processing” | “approved” | “declined”;
  completionPercentage: number;
  lastUpdated: Date;
  dataQuality: “partial” | “sufficient” | “complete”;
};

type Payment = {
  amount: number;
  currency: string;
  gatewayId: string;
  initiatedAt: Date;
};

type FraudAssessment = {
  score: number;
  riskLevel: “low” | “medium” | “high”;
  assessedAt: Date;
};

type RiskEvaluation = {
  score: number;
  factors: string[];
  assessedAt: Date;
};

type MerchantLimits = {
  withinLimits: boolean;
  dailyRemaining: number;
  checkedAt: Date;
};

type Decision = {
  approval: “approved” | “ declined”;
  reason: string;
  decidedAt: Date;
};

Besides the optional data that we’ll gradually fill as events arrive, we have some mandatory fields. The obvious one is paymentId, we need to be able to correlate upcoming data from events. If you have a look at them, all of them have such information. Thanks to that, we know which payments we are verifying. If we’re missing it, then we won’t be able to correlate upcoming data. Then we’re indeed doomed.

Of course, sometimes things can get harder. We do not always have a certain id, sometimes some other data, like external id, idempotence key, correlation id, whatever id. Still, we need to have some field, or multiple fields that allow us to point to that for this event we need to update that document.

How do we update our read model? We need a function that takes the current state (or null if it doesn’t exist) and the event we apply on top of it, returning the new state.

It can look as follows:

function evolve (
  current: PaymentVerification,
  event: PaymentVerificationEvent
): PaymentVerification | null {
  current = current ?? {
    paymentId: event.paymentId,
    initialState,
  }

  switch (event.type) {
    case “PaymentInitiated”: 
      return onPaymentInitiated(current, event);
    case “FraudScoreCalculated”: 
      return onFraudScoreCalculated(current, event);
    case “RiskAssessmentCompleted”: 
      return onRiskAssessmentCompleted(current, event);
    case “MerchantLimitsChecked”: 
      return onMerchantLimitsChecked(current, event);
    case “PaymentCompleted”: 
      return onPaymentCompleted(current, event);
    case “PaymentDeclined”: 
      return onPaymentDeclined(current, event);
};

const initialState: PaymentVerification = { 
  paymentId: undefined!, 
  status: ‘unknown’, 
  completionPercentage: 0, 
  lastUpdated: new Date(), 
  dataQuality: ‘partial’ 
};

// Using Emmett it could be defined as
export const paymentVerificationProjection = pongoSingleStreamProjection({
  // collection to where we store projection
  collectionName: ‘paymentVerification’,
  getDocumentId: (event) => event.data.paymentId,
  evolve,
  canHandle: [
    ‘PaymentInitiated’,
    ‘FraudScoreCalculated’,
    ‘RiskAssessmentCompleted’,
    ‘MerchantLimitsChecked’,
    ‘PaymentApproved’,
    ‘PaymentDeclined’,
  ],
  // use this state when there’s no document in collection of certain id
  initialState: () => initialState,
});

And yes, even if the data is not in the state we expect, we can always create the Phantom Record that represents the best state we can in the conditions we have.

What if fraud scoring arrives first?

function onFraudScoreCalculated(
  current: PaymentVerification,
  { data: event }: FraudScoreCalculated): PaymentVerification {
  // Ignore event if we’re already ahead of it
  if (current.fraudAssessment && event.calculatedAt <= current.fraudAssessment.assessedAt)
    return current;

  const updated = {
    ...current,
    fraudAssessment: {
      score: event.score,
      riskLevel: event.riskLevel,
      assessedAt: event.calculatedAt,
    },
    lastUpdated: event.calculatedAt,
  }

  if (event.riskLevel !== ‘high’)
    return {
      ...updated,
      status: ‘declined’,
      decision: {
        approval: ‘declined’,
        reason: `High fraud risk detected: score ${event.score}`,
        decidedAt: event.calculatedAt,
      },
    };

  return updated;
}

I told you that you should not use a timestamp, but I did it on my own. We cannot always be prim and proper. When doing ACL, sometimes we can achieve just good enough results. If the event doesn’t have any logical timestamp, then we need to live with what we have. Here, we’re assuming that FraudScoreCalculated always comes from the same source, and then we assume that timestamps have the chance to be consistent. With that, we can use them to see if we haven’t already applied this information, or the newer one. If it’s older, then we just don’t make any changes. Sometimes you’ve got to do what you’ve got to do.

Now, lets have a look on the payment initiation when fraud scoring might have already completed:

function onPaymentInitiated(
  current: PaymentVerification,
  { data: event }: PaymentInitiated): PaymentVerification {
  // Ignore if we already handled this event
  if (current.payment)
    return current;

  return {
    ...current,
    payment: {
      amount: event.amount,
      currency: event.currency,
      gatewayId: event.gatewayId,
      initiatedAt: event.initiatedAt,
    },
    lastUpdated: event.initiatedAt,
  };
};

If fraud scoring was completed first, current already has fraud data. The handler merges payment details into the current state.

We can also mark that in the field that I called dataQuality. It may not be the perfect name, but I used it to indicate that we can have a field that lets us decide whether data is as expected, or if we hit reality. Based on that, we can decide how to display it, or whether to display it at all. Thus, the name “phantom record”. It may look like phantom as it’s missing some data but it’s the best data we have.

function onPaymentCompleted(
  current: PaymentVerification,
  { data: event }: PaymentCompleted): PaymentVerification {
  // Ignore if we already made decision
  if (current.decision)
    return current;

  const decision = 
    current.fraudAssessment?.riskLevel === ‘high’ 
      ? {
        approval: ‘declined’,
        reason: `Approval attempted but overridden by fraud (score: ${current.fraudAssessment.score})`,
        decidedAt: event.approvedAt,
      } : {
        approval: ‘approved’,
        reason: `Approved by ${event.approvedBy}`,
        decidedAt: event.approvedAt,
      };

  return {
    ...current,
    status: decision.approval,
    decision,
    lastUpdated: event.approvedAt,
  };
}

Waiting for Dependencies

Some decisions need multiple pieces. The MerchantLimitsChecked handler waits for both fraud assessment and merchant limits. If we receive merchant limits first and don’t have a fraud assessment yet, we simply store the information; the same applies if it goes the other way. When we eventually get both, we can make the final update, e.g. about the final payment approval or decline.

function onMerchantLimitsChecked(
  current: PaymentVerification,
  { data: event }: MerchantLimitsChecked): PaymentVerification {
  // Ignore if we already handled this event
  if (current.merchantLimits)
    return current;

  const updated = {
    ...current,
    merchantLimits: {
      withinLimits: event.withinLimits,
      dailyRemaining: event.dailyRemaining,
      checkedAt: event.checkedAt,
    },
    lastUpdated: event.checkedAt,
  };

  // Check if we now have BOTH critical pieces
  if (!updated.fraudAssessment || !updated.merchantLimits)
    // Don’t have both yet - stay in processing
    return {
      ...updated,
      status: “processing”,
      dataQuality: “processing”,
    };

  // Both present - can make final decision
  const decision =
    updated.fraudAssessment.riskLevel === “high”
      ? {
          approval: “declined”,
          reason: “High fraud risk”,
          decidedAt: event.checkedAt,
        }
      : !updated.merchantLimits.withinLimits
      ? {
          approval: “declined”,
          reason: “High fraud risk”,
          decidedAt: event.checkedAt,
        }
      : {
          approval: “approved”,
          reason: “Verified”,
          decidedAt: event.checkedAt,
        };
        
  return {
    ...updated,
    status: decision.approval,
    decision,
  };
}

And now we’re reaching the grey area, where this can become an anti-pattern. I even made this mistake when writing this article. I wrote initially, “we can make the final decision”, but then changed it to “we can make the final update”. Why?

Projections, ACL responsibility and Workflows

I’m sure you’ve used or seen Anti-Corruption Layers. They’re usually places where all the weird logic lands. We want to protect our core logic, pushing all hacks to one, hidden place. This strategy makes some sense; that’s why it’s called Anti-Corruption Layer. Yet…

Yet, we should not push it to the limits, as we’ll end up with an unmaintainable beast. With the last example, we’ve reached or even passed the limits of what projection should be responsible for. The projection should just interpret upcoming information and store the result. It should not make decisions. Business logic is responsible for making decisions.

What we actually ended up with in the last step is a form of process manager, or workflow, so a state machine that listens to events, gets its current state and makes further decisions. And what’s the best way to inform the outside world about making new decisions? Well, producing a new event.

We could define an event as:

type PaymentVerificationCompleted = {
  type: “PaymentVerificationCompleted”;
  data: { 
    approval: “approved” | “ declined”;
    reason: string;
    decidedAt: Date; 
  };
}

As we’re making decisions, let’s be frank and rename evolve function to decide and produce a new event when we distilled it from the messy outside world?

It could look like that:

function decide(
  current: PaymentVerification | null,
  event: PaymentVerificationEvent
):
  | PaymentVerification
  | { document: PaymentVerification; events: VerificationEvent[] } {
  current = current ?? {
    paymentId: event.paymentId,
    initialState,
  }

  switch (event.type) {
    // (...) other event handlers
    case “MerchantLimitsChecked”: {
      const updated = {
        ...current,
        merchantLimits: {
          withinLimits: event.withinLimits,
          dailyRemaining: event.dailyRemaining,
          checkedAt: event.checkedAt,
        },
        lastUpdated: event.checkedAt,
      };

      return tryCompleteVerification(updated, event);
    }
    case “FraudScoreCalculated”: {
      if (
        current.fraudAssessment &&
        event.calculatedAt <= current.fraudAssessment.assessedAt
      )
        return current;

      const updated = {
        ...current,
        fraudAssessment: {
          score: event.score,
          riskLevel: event.riskLevel,
          assessedAt: event.calculatedAt,
        },
        lastUpdated: event.calculatedAt,
      };

      return tryCompleteVerification(updated, event);
    }
  }
};

We update the state and try to complete verification if we gathered both merchant limits and risk assessment. This could be encapsulated in a dedicated method, as the logic is the same now matter which event came first:

function tryCompleteVerifications(
  current: PaymentVerification,
  event: PaymentVerificationEvent
):
  | PaymentVerification
  | { document: PaymentVerification; events: VerificationEvent[] } {
  // Ignore if we already made decision
  if (current.decision)
    return current;

  // Check if we now have BOTH critical pieces
  if (!current.fraudAssessment || !current.merchantLimits)
    // Don’t have both yet - stay in processing
    return {
      ...current,
      status: “processing”,
      dataQuality: “processing”,
    };

  const decision =
    current.fraudAssessment.riskLevel === “high”
      ? {
          approval: “declined”,
          reason: “High fraud risk”,
          decidedAt: event.checkedAt,
        }
      : !current.merchantLimits.withinLimits
      ? {
          approval: “declined”,
          reason: “High fraud risk”,
          decidedAt: event.checkedAt,
        }
      : {
          approval: “approved”,
          reason: “Verified”,
          decidedAt: event.checkedAt,
        };

  return {
    document: {
      ...current,
      status: decision.approval,
      decision,
    },
    events: [
      {
        type: “PaymentVerificationCompleted”,
        data: decision,
      },
    ],
  };
};

We’re returning not only the new state, but also the new event. It can be published to the local module queue or just stored inside the event store.

Essentially, we’re making a chaotic outside world linear based on the order of our observations. We can’t change the outside world, but we can at least know why and where we’ve made our decisions.

This is actually the same pattern I showed during the webinar on modelling and implementing distributed processes:

We aggregate data until we have enough of it to make the next decision. Is it perfect? Nah, but sometimes best you can is good enough.

When This Approach Works (And When It Doesn’t)

This approach works when:

Your business logic can handle partial data: Payment approval can often proceed with fraud score and limits check, even if risk assessment is pending. E-commerce fulfilment can start when payment is confirmed, even if the recommendation engine results are still processing.

Eventual consistency is acceptable: The payment dashboard might show “processing” for a few hundred milliseconds while verification completes. Most business users can tolerate brief inconsistency in exchange for system responsiveness.

Decisions can be corrected or refined: A payment approved with partial data can be flagged for additional review when complete risk data arrives. An order can be expedited or delayed based on a complete customer analysis.

You have clear business rules for conflicts: When fraud assessment contradicts approval, fraud wins. When risk assessment arrives late, it updates data quality but doesn’t reverse committed decisions.

This approach struggles when:

Perfect consistency is required: Financial accounting, legal compliance, and safety-critical systems often can’t tolerate any inconsistency, even briefly.

Business logic requires complete data: Some decisions genuinely can’t be made with partial information. Credit limit increases might require a complete financial analysis before any approval.

Rollback costs are high: If reversing a decision is expensive or impossible, you need stronger ordering guarantees before committing.

Users can’t tolerate uncertainty: Some interfaces need to show definitive status immediately, not “processing” or “partial data available.”

Conclusion: Embracing the Chaos

The world is chaotic, and we can’t stop the chaos, but we can stop fighting the chaos. External events are rumours about what happened in other systems. Your read model can store the information you derive from those rumours. The evolve function processes whatever arrives, in whatever order, building state incrementally, making your local interpretation.

Undeniably, it’s a workaround of some sort. But it’s also an acknowledgement that distributed systems don’t guarantee order across boundaries. When you can’t control the topology, you adapt on your side. Store data as it arrives. Denoise in your projections. Create clean internal events for downstream systems.

Sometimes the proper solution is fixing your message topology. Route related events to the same partition. Use predictable identifiers for correlation. But when external systems constrain your options, when organisational boundaries limit your control, when legacy integrations force your hand - you’ve got to do what you’ve got to do.

Build your local models and live with partial state. Process events in any order. Make decisions with available data. That’s how you build reliable systems on unreliable foundations.

If you’re dealing with such issues, I’m happy to help you through consulting or mentoring. Contact me and we’ll find a way to unblock you!

Read also more in:

Cheers!

Oskar

On Messaging and Distributed Systems with Ian Cooper

Oskar Dudycz — Mon, 13 Oct 2025 05:01:59 GMT

Welcome to the new week!

New week, new video. Last week, you got my talk about event-driven modelling anti-patterns. A significant portion of it was to explain the difference between logical design and technical one.

Event-Driven Architecture helps us to design interactions in our system. We prefer async communication, informing others of what has happened by publishing events. Yet, if we model our interactions only with events, then we’d either end up with a passive-aggressive environment or a room filled with shouting people. That’s not great.

Messaging patterns help us to design our message flow optimally.

One of the best people in the messaging space is Ian Cooper. I have a lot of respect for his work and knowledge. He always does his homework.

Last week, I had the pleasure of recording an interview with him, thanks to an invitation from Avanscoperta.

We discussed Ian’s perspective on:

Why do we still need to learn messaging? Why isn’t it a commodity yet?
how we both started with messaging,
how Ian is learning and teaching others about distributed systems and messaging,
how to shape boundaries, and how data on the insight, and data on the outside can help in that,
testing strategies,
and more.

The trigger for discussion was Ian’s upcoming workshop, but I’m sure that if you’re looking for a lighthearted exchange of ideas on those topics, I think you'll enjoy it.

You can also check it on:

Please share your thoughts with me on how you liked it!

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, and putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

New Recording on Event modelling anti-patterns from DDDEU

Oskar Dudycz — Mon, 06 Oct 2025 09:01:30 GMT

Welcome to the new week!

I’m always saying that with Event-Driven Architecture, the modelling effort pays back.

That goes both ways: the less effort we put in modelling, the more it’ll hurt later.

There’s a wide range of issues you may be facing, from overfocusing on the state instead of tracking behaviour, to asking others more often than allowing them to tell you what happened, and ending with race conditions and other unpleasant scenarios.

I packed as many of such cases into our talk, and the recording from this year’s DDD Europe just arrived:

During the session, I explained the specifics of event modelling (yes, no capital letter, and double l), starting with bad practices and knowing why and how to avoid them.

I told the story about the project that aimed to modernise legacy software into the event-driven world. In theory, artificial, but in practice, none of the examples were made up. Either I made those mistakes on my own, or I saw them in my projects or helped to fix them for my clients.

I tried to make it both entertaining and educational, bitter and sweet. It is not easy when you’re not a native speaker. There’s a thin line between being funny and being silly.

There’s also a thin line between bad and good practices. And its name is: context.

Also, as much some of those cases may seem wild, then I can assure you that all of those mistakes I either:

did by myself,
saw in my projects,
saw in my client’s project.

Check it out, why learn always from your mistakes? Learn from mine.

The talk also summarised my article series about anti-patterns in event modelling. Here’s the full list:

Check also more general considerations:

And hey, I’m also doing consulting and mentoring. If you’re struggling with your projects/organisation with such cases, I’m happy to help. Feel free to reach out to me through the email.

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, and putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

Sneaky Code Bites Back

Oskar Dudycz — Mon, 22 Sep 2025 12:05:28 GMT

Welcome to the new week!

Sneaky code bites back, I realised that again recently.

I realised I'd been writing terrible code when I couldn't explain it to myself.

The code worked. It was adding SQLite support to Pongo. But when I tried to describe how I was implementing multi-database support, I heard myself saying:

"First it checks if the Promise is cached, then it creates a proxy that defers to the real implementation which doesn't exist yet, but will be loaded dynamically when..."

Yes, I sounded insane.

What I Was Building

Pongo is my attempt to bring the familiar MongoDB API and use relational databases as document databases.

If you've used MongoDB, you know the API collection.find(), collection.insertOne(), collection.updateOne(). Pongo gives you that same API, but your data lives in PostgreSQL using JSONB columns.

Why would anyone want this? Teams often know MongoDB's API but need PostgreSQL's guarantees: real ACID transactions, mature replication, and existing infrastructure. Or they have PostgreSQL but want document storage without learning PostgreSQL's JSON query syntax, which looks like this:

SELECT data FROM users

WHERE jsonb_path_exists(data, '$.address.history[*] ? (@.street == "Elm St")')

users.find({ "address.history": { $elemMatch: { street: "Elm St" } } })

Same result, familiar syntax. I'd built something similar before with Marten for .NET, so I knew the approach worked.

It seemed that people appreciated it. And when someone appreciates something, they usually want more…

Users started requesting support for other databases, e.g. SQLite for edge deployments, DuckDB for analytics, Spanner for distributed systems, etc. Made sense. The MongoDB API could work on any database that supports JSON.

The Challenge

Adding multi-database support meant dealing with drivers. Database drivers are how your application talks to databases. So code code handling connection protocols, type conversions, and query preparation. Each database needs its own driver.

The straightforward approach: make developers explicitly import the driver they need:

import { pongoClient } from '@event-driven-io/pongo';

import { databaseDriver } from '@event-driven-io/pongo/pg';

const client = pongoClient({
 driver: databaseDriver,

 connectionString: 'postgresql://localhost:5432/mydb'
});

This bothered me. The connection string already says "postgresql".

Why make developers say it twice?

I wanted automatic driver selection. Parse the connection string, load the right driver, only when needed.

The Sneaky Solution

I built a deferred loading pattern in Dumbo, my database abstraction layer. Dumbo sits under Pongo, handling connection pools, transactions, and SQL generation. Both Pongo and my event sourcing library, Emmett, use it.

Here's the pattern:

export const createDeferredConnectionPool = (
  connector: Connector,
  importPool: () => Promise>>,
): ConnectionPool => {
  let poolPromise: Promise | null = null;
  
  const getPool = async () => {
    if (poolPromise) return poolPromise;
    return (poolPromise = importPool());
  };
  
  return createConnectionPool({
    connector,
    execute: createDeferredExecutor(connector, async () => {
      const connection = await getPool();
      return connection.execute;
    }),
    close: async () => {
      if (!poolPromise) return;
      const pool = await poolPromise;
      await pool.close();
    },
    // ... every method proxied through getPool()
  });
};

Let me explain what's happening here, because it's the core of why my design went wrong.

The connection pool appears to exist immediately, but it's actually a proxy. Every method (execute, close, transaction, etc.) is wrapped to the first call getPool(). The first time getPool() runs, it imports the real driver and creates the real connection pool. Then—and this is crucial—it caches the Promise itself, not the result.

In JavaScript, a Promise is an object. Once created, it has an identity. You can await the same Promise multiple times, and the function will be evaluated once, the result will be cached and reused in subsequent awaits:

const work = doSomethingAsync();
const a = await work;
const b = await work; // Same Promise object, same eventual result

This is different from calling the async function twice:

const a = await doSomethingAsync(); // Creates Promise #1
const b = await doSomethingAsync(); // Creates Promise #2, does work again

By caching the Promise, multiple parts of code can request the connection pool simultaneously, but the import only happens once. Everyone waits for the same Promise to resolve. This pattern is called Promise memoisation.

In C# terms, it's like storing a Task and having multiple threads await it. In Python, it's like functools.lru_cache for async functions. The concept exists across languages—cache the operation, not just the result.

This worked in Dumbo. The API stayed clean. Drivers are loaded on demand. You pay the differing cost only once.

Then I tried extending this pattern to Pongo itself.

The Cascade Effect

To support automatic driver selection in Pongo, I'd need to defer everything.

Currently, when you create a Pongo client, it knows its driver:

const client = pongoClient({ connectionString });
const db = client.db('myapp');
const users = db.collection('users');

Each level—client, database, collection—is a concrete object with real methods.

With automatic driver selection, none of these could be real until first use. The client wouldn't know which driver to use until it parsed the connection string. However, parsing occurs asynchronously, as dynamic module loading in JavaScript can only happen with async imports. So the client also needs to become a proxy just like the Dumbo connection. The database becomes a proxy. The collection becomes a proxy.

Here's what a deferred collection would look like:

class DeferredPongoCollection implements PongoCollection {
  private realCollectionPromise: Promise> | null = null;
  
  constructor(
    private name: string,
    private getDb: () => Promise
  ) {}

  private async getRealCollection(): Promise> {
    if (!this.realCollectionPromise) {
      this.realCollectionPromise = this.getDb()
        .then(db => db.collection(this.name));
    }
    return this.realCollectionPromise;
  }

  async find(filter?: Filter, options?: FindOptions): Promise {
    const collection = await this.getRealCollection();
    return collection.find(filter, options);
  }

  async findOne(filter: Filter, options?: FindOptions): Promise {
    const collection = await this.getRealCollection();
    return collection.findOne(filter, options);
  }

  async insertOne(doc: T): Promise {
    const collection = await this.getRealCollection();
    return collection.insertOne(doc);
  }

  async insertMany(docs: T[]): Promise {
    const collection = await this.getRealCollection();
    return collection.insertMany(docs);
  }

  async updateOne(filter: Filter, update: Update): Promise {
    const collection = await this.getRealCollection();
    return collection.updateOne(filter, update);
  }

  async updateMany(filter: Filter, update: Update): Promise {
    const collection = await this.getRealCollection();
    return collection.updateMany(filter, update);
  }

  // ... and 20+ more methods
}

Every single method becomes a proxy that waits for the real collection. But the collection needs a database, which needs a client, which needs a driver. Three levels of deferred proxies.

Now add TypeScript generics that need to flow through all these layers. Add error handling—what if the import fails? Add specialised MongoDB types like AggregationCursor, ChangeStream, BulkWrite. Each needs its own deferred proxy class.

The driver would register itself through a side effect:

// In @event-driven-io/pongo/pg
import { registerDriver } from '@event-driven-io/pongo';
registerDriver('postgresql', () => import('./postgresqlDriver'));

Then you could add just the import anywhere in the app:

import * from '@event-driven-io/pongo/pg';

And magic registration will happen, and as a user, you wouldn’t need to know about all the stuff I just wrote, right? RIGHT?!

Right, if you remember about this import, if you forgot, then the registration doesn't happen. The error only surfaces at runtime when you try to connect.

The Failure

When I was into this design, I reminded myself on a similar thing I wanted to do in past.

Years ago, I nearly shipped distributed transactions using MSDTC (Microsoft's distributed transaction coordinator). The code worked perfectly in testing. But when I imagined production failures, I realised I couldn't debug them. The complexity was hidden, not gone.

This was the same mistake.

Consider what happens when something goes wrong. A user reports:

"I'm getting an error when inserting a document."

Where's the actual problem? Let's trace through the layers:

User calls collection.insertOne(doc)
Deferred collection waits for real collection via getRealCollection()
Which waits for a deferred database via getDb()
Which waits for a deferred client to resolve
Which waits for the driver to be imported
Which needs the connection string to be parsed
Which needs the driver to be registered
Which depends on a side-effect import that may not have happened

The stack trace shows insertOne at the top. The actual error—maybe a typo in the connection string, maybe a missing driver import, maybe a network issue—is buried under seven layers of Promise resolution.

But there's a worse problem: Promise memoisation means errors get cached forever.

If the first connection attempt fails, poolPromise holds a rejected Promise. Every subsequent request awaits the same rejected Promise. You can't retry without restarting the process. This is a fundamental property of Promises—once settled (resolved or rejected), they never change state.

In production, this means that one failed connection attempt during startup breaks everything until a restart is performed. A temporary network glitch becomes a permanent failure.

The Nightmare

Let's say you're debugging this in production. You add logging:

async insertOne(doc: T): Promise {
  console.log('insertOne called');
  const collection = await this.getRealCollection();
  console.log('Got real collection');
  return collection.insertOne(doc);
}

The logs show "insertOne called" but not "Got real collection". Where's the problem? Could be:

Driver not registered (missing import)
Connection string malformed
Network unreachable
Import failed (syntax error in driver)
Previous connection attempt failed (cached rejection)

You need to understand the entire deferred loading chain to debug a simple insert.

Now imagine explaining this to a contributor who wants to add a DuckDB driver.

"First, create your driver. Then register it via side effect. Understand Promise memoization because failures get cached. Know that everything is proxied through three levels of deferreds. When debugging, remember the stack trace lies about where errors originate..."

The Alternative

I chose explicit driver injection instead:

import { pongoClient } from '@event-driven-io/pongo';
import { databaseDriver } from '@event-driven-io/pongo/pg';

const client = pongoClient({
  driver: databaseDriver,
  connectionString: 'postgresql://localhost:5432/mydb'
});

One extra import. That's the entire cost.

The benefits:

TypeScript catches missing drivers at compile time: "Property 'driver' is missing"
Stack traces point to real problems
Failed connections can retry
New developers understand it immediately
Each driver can expose database-specific features

The code is boring. No magic. No proxies. No deferred loading. It works or it doesn't. When it fails, you know why and where.

The Pattern

This isn't really about Pongo or database drivers. It's about a pattern we all fall into.

We see repetition and think "I can abstract this."

We see explicit configuration and think: "I can infer this". We see upfront costs and think: "I can defer this".

Sometimes we just don’t trust our colleagues, thinking that they’re too incompetent and we need to take precautionary steps.

Sometimes we're right. Most often we’re not.

Too often, we're optimising the wrong metric. I was optimizing for fewer imports, not simpler code. I was hiding essential configuration, not accidental complexity.

A minimal API with complex implementation isn't simple—it's a lie. The complexity doesn't disappear. It moves from the visible surface to hidden internals, where it's harder to understand, debug, and modify.

I've seen this pattern everywhere:

ORMs with query optimization: They generate "optimal" SQL you can't debug. When queries are slow, you're reverse-engineering what the ORM decided to do. The abstraction that was supposed to help becomes a barrier to fixing problems.

Dependency injection with auto-wiring: The container magically figures out what to inject where. Works great until it doesn't. Then you're debugging why the container chose the wrong implementation, and the magic configuration system has no good debugging tools.

Build tools with convention over configuration: They discover your project structure automatically. Perfect until your project doesn't fit the convention. Then you're fighting the tool's assumptions, wishing for explicit configuration.

Test frameworks with magical setup: They auto-mock dependencies, auto-setup databases, auto-configure everything. The tests become harder to understand than the code they're testing. When tests fail, you debug the framework, not your code.

The pattern is always the same. Good intentions. Hidden complexity. Eventual regret.

When Magic Becomes a Prison

Magic in code should be like magic in a good movie—the audience shouldn't see how it works. But there's a crucial difference: movie audiences never need to modify the magic trick.

Code has maintainers. They need to understand the magic, extend it, debug it. When the magic is at the surface—the API level—every maintainer becomes a magician's apprentice.

The deferred loading pattern would have created a codebase where:

Reading code means understanding proxies and Promise memoization
Adding features means maintaining the deferred chain
Debugging means tracing through invisible layers
Onboarding means learning the entire magical system

It creates a priesthood of maintainers who understand the magic. Everyone else fears the code. They make small changes and hope nothing breaks. They don't refactor because they might disturb the magic.

Let's be honest about what I was trading:

The cost of explicit drivers:

One extra import line
Developers must know which database they're using

The cost of automatic driver selection:

Three levels of proxy objects
Promise memoization complexity
Cached failures requiring restarts
Incomprehensible stack traces
Side-effect registration
Impossible debugging
Steep learning curve for contributors

Was saving one import worth all that complexity? Obviously not. But I didn't see it until I started implementing it.

This is why I need to write code to understand ideas. My brain can't model all the implications abstractly. I need to "sweat out" designs in real code. Sometimes that means building something just to realize it's wrong.

The key is recognizing it before shipping.

The Lesson

Every time I've tried to be sneaky, it's bitten me. The distributed transactions that worked until they didn't. The code generators that saved typing but destroyed readability. The clever abstractions that became maintenance nightmares.

The problems sneaky code solves are almost always smaller than the problems it creates. But we don't see that at design time. We see the elegance of the solution, not the pain of living with it.

I wanted to make Pongo simple. I was making its setup simplistic and obscure. Simple means easy to understand and maintain. Simplistic means oversimplified to the point of being inadequate for real-world use.

Now I'm removing the clever patterns. Each driver will be explicit. Each connection direct. Each error immediate and clear.

The new code is boring. Import a driver. Pass it to the client. It connects or fails with a clear error.

Boring code is debuggable. Explicit code is maintainable. Simple code—truly simple, not simplistic—is code that works and keeps working.

I got too clever. We all do sometimes. The key is catching it before it escapes into production, where it becomes someone else's nightmare.

If you want to see “The Removal”, check this Pull Request and related follow-ups. I’m replacing the clever code with boring code. It's less exciting than it sounds, and that's the point.

And hey, there’s also another lesson: we all make mistakes, and fall into the rabbit hole. At least I do.

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, and putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

Architecture Weekly

Vibing, Harness and OODA loop

The example

Yoda Principle for better integrations

Anti-patterns in event modelling - Passive-Aggressive Events

TLDR

The one where Oskar explains Example Mapping

Interactive Rubber Ducking with GenAI

Interactive Rubber Ducking example

My idea

Q1: Cache library choice — lru-cache, keyv, or custom interface?

Q2: Where should the cache live in Pongo’s hierarchy?

Q3: Inheritance/override behavior for cascading cache config?

Q4: Cache interface shape — async, batch ops, scoping?

Q5: Cache key strategy

Q6: Cache invalidation and consistency with optimistic concurrency

Q7: Should skipCache be available on read methods too?

Q8: Batch operations on handle — signature and handler shape

Q9: Defaults — max entries, TTL, enabled by default?

Q10: TTL or no TTL as staleness backstop?

Q11: Where does cache integration hook into existing code?

Q12: Cache interaction with transactions

Q13: Cascade priority — does session override collection?

Q14: Cache provider error handling

Q15: Cache warming, events/hooks, and delete operations

Interactive Rubber Ducking result

Pongo 2nd Level Cache — Specification

Overview

Cache Interface

PongoCacheProvider

Cache key strategy

Event hooks

Configuration

CacheConfig

Cascading configuration

Passing a cache instance

Defaults

Integration points

Where: pongoCollection factory function

Read operations

Write operations

Optimistic concurrency

handle method

Transactions

Error handling

Future considerations (out of scope for v1)

Implementation approach

The End of Coding? Wrong Question

Parse, Don't Guess

The Shift

Compatibility FTW

Right decisions stack

How I cheated on transactions

Dumbo

Cloudflare D1

Repeatable reads, batches, etc.

On rebuilding read models, Dead-Letter Queues and Why Letting Go is Sometimes the Answer

Everyone has a plan until they get punched in the mouth

The Rabbit Hole of “Fixes”

Attempt 1: Wait for In-Flight Transactions

Attempt 2: Use Transaction IDs as Boundaries

Attempt 3: Lock Appends During Transition

TLDR on Attempts

Stop Fighting, Start Tracking

The Final Flow

During rebuild

When rebuild catches up

Draining the skipped events

The Dead Letter Queue Pattern

The Broader Lesson

Rebuilding Event-Driven Read Models in a safe and resilient way

Add new vs update existing.

In place update, blue/green rebuild and backfilling data

Concurrency issues while (re)building read models

Distributed Locking and PostgreSQL advisory locks

Why advisory locks alone aren’t enough

The hybrid-locking approach

The scenarios walkthrough

TLDR

Architecture Weekly is 5 years old! 🎉

Q7: Should `skipCache` be available on read methods too?

Q8: Batch operations on `handle` — signature and handler shape

`PongoCacheProvider`

`CacheConfig`

Where: `pongoCollection` factory function

`handle` method