This is about what we can do on Tuesday morning, when someone from support writes: “the price on this reservation looks wrong”.

The bug

Imagine we’re working on a hotel reservations system. Pricing there is fiddly: rate plans get revised, extras are added after booking, and the city sets the tourist tax on its own schedule.

Let’s say that the tourist tax used to be a single value in configuration, 4.50 per person per night, multiplied by nights and guests. That was enough for years, because the rate never moved.

Then the city announced an increase to 7.00 from 1 April. It appeared that one value can’t express that: a stay from 30 March to 2 April has nights on both sides of the boundary. So we added rates with validity periods and a lookup to resolve them, and shipped it on 12 March.

The lookup resolves the rate once per reservation, from the check-out date, and applies it to every night. For a stay inside a single period that gives the right answer, which is every case the tests covered. For a stay crossing 1 April, it charges 7.00 for the March nights too. Crazy, but well, that’s how life is sometimes.

A rate change has people checking numbers they’d normally trust, and on 19 March a front office manager reported that the projected tax for early-April arrivals doesn’t match what she gets counting by hand. We confirmed it and deployed a fix at 16:02 the same day.

Then the actual work starts. Which reservations are wrong, by how much, and what do we do about them?

What we have, with state

Take the reservation support asked about. Seven nights, two guests, 27 March to 3 April, booked on 14 March, breakfast added two days later. Here’s the row:

| id	| check_in   | check_out  | rate_plan_id | total_amount	| updated_at       |
|-------|------------|------------|--------------|--------------|------------------|
| 8f21	| 2026-03-27 | 2026-04-03 | RP-STD-2026	 | 1358.00	| 2026-03-16 14:02 |

The total is wrong. What should it be? The row doesn’t say how the number was produced: no breakdown of rooms, extras and tax, no record of which nights got which tax rate, and no version on rate_plan_id, so we can’t tell which prices were in force on 14 March. updated_at says 16 March, which is the breakfast, not the booking.

So we recompute and write the migration:

It matches 1,240 rows. We run it at 17:30 on 19 March.

What that migration does

Around 900 of those rows genuinely came from the broken build. The other 340 were booked before 12 March, had correct totals, and were caught because someone changed a room or added a note during the window. updated_at moves for all of that, and there’s nothing in the row that separates “this changed” from “this changed because of the bug”. We recalculate them along with the rest.

recalculate_total reads the rate plan as it stands today, and the plans were revised on 15 March for the season. RP-STD-2026 went from 180.00 a night to 205.00. For reservation 8f21 that means:

|	  | before	| after migration   | correct  |
|---------|-------------|-------------------|----------|
| rooms	  | 1260.00	| 1435.00	    | 1260.00  |
| extras  | 168.00	| 168.00	    | 168.00   |
| cityTax | 98.00	| 73.00	            | 73.00    |
| total	  | 1526.00	| 1676.00	    | 1501.00  |

The migration fixed the tax and repriced the rooms. It did that to every reservation booked before 15 March, roughly 400 of them, and each guest gets a confirmation with a total higher than the one they agreed to.

And then migration number two

Complaints reach us around 2 April, and we work out what happened. Now we need to put the room prices back. To do that, we need two things per reservation: the date it was booked, so we know which rate plan version applied, and what the total was before we touched it.

We have neither. updated_at reads 2026-03-19 17:30 for all 1,240 rows we updated, so it no longer distinguishes the ones we broke from the ones we fixed from the ones we shouldn’t have touched. total_amount holds the value our migration wrote. The booking date was never stored separately, and the only thing that used to approximate it was updated_at.

So migration two is written against data that migration one produced. We reconstruct booking dates from confirmation emails where we still have them, guess at the rest, and pick a WHERE clause that we hope covers the right subset. If we get that wrong, the input to migration three is the output of migration two. I’ve been on both ends of this, and it doesn’t get more pleasant.

We can prepare for this in a state-based system, of course. Keep the breakdown in a JSON column, store the rate plan version on the reservation, add a price_history table, add a corrected_by marker. I’ve done all of those. The catch is that we add each one after the incident that taught us to, on the table where that incident happened, and today’s incident is on a different table. What we’re building at that point is a partial event log, decided per column, under time pressure.

The same bug, event-sourced

Here’s the stream for the same reservation:

| when         | event                        | data                                                       |
|--------------|------------------------------|------------------------------------------------------------|
| 14 Mar 09:40 | `ReservationInitiated`       | 27 Mar – 3 Apr, DBL, 2 guests                              |
| 14 Mar 09:41 | `RatePlanApplied`            | RP-STD-2026 v4, 180.00 / night                             |
| 14 Mar 09:41 | `ReservationPriceCalculated` | rooms 1260.00, cityTax 98.00, total 1358.00                |
| 14 Mar 09:41 | `ReservationConfirmed`       |                                                            |
| 16 Mar 14:02 | `ExtraAdded`                 | breakfast, 2 guests, 168.00                                |
| 16 Mar 14:02 | `ReservationPriceCalculated` | rooms 1260.00, extras 168.00, cityTax 98.00, total 1526.00 |

The corrupted value is cityTax, and it’s sitting there in plain sight. With two guests, a March night costs 9.00 in tax and an April night 14.00, so five March nights and two April nights come to 45.00 + 28.00 = 73.00. The event says 98.00, which is all seven nights at 14.00. Everything else in that event is right, and RatePlanApplied still says the booking was made at 180.00 a night under version 4, so we can calculate the correct total without reconstructing anything:

| What    | recorded | correct |
|---------|----------|---------|
| cityTax | 98.00    | 73.00   |
| total   | 1526.00  | 1501.00 |

That’s the difference from the table. The migration above went wrong because the inputs to the original calculation were gone, and here they aren’t. The 14 March event still says what it said on 14 March, whatever was appended later.

Knowing which events to look at

The other half is metadata. Here’s the full event:

{
  "type": "ReservationPriceCalculated",
  "data": {
    "reservationId": "8f21",
    "nights": [
      { "date": "2026-03-27", "room": 180.00, "cityTax": 14.00 },
      { "date": "2026-03-28", "room": 180.00, "cityTax": 14.00 },
      "...",
      { "date": "2026-04-02", "room": 180.00, "cityTax": 14.00 }
    ],
    "cityTax": 98.00,
    "total": 1358.00
  },
  "metadata": {
    "timestamp": "2026-03-14T09:41:12Z",
    "correlationId": "b31e...",
    "causationId": "a07c...",
    "userId": "booking-engine",
    "buildSha": "a4f9c2e",
    "schemaVersion": 3
  }
}

There’s one interesting metadata: buildSha. What is it? It’s the commit the service was running when it appended the event. It’s an environment variable and a few lines in whatever builds our metadata. On 19 March, it lets us query by the thing we actually care about:

SELECT stream_id, data, metadata
FROM events
WHERE type = 'ReservationPriceCalculated'
  AND metadata->>'buildSha' = 'a4f9c2e';

That returns 900 events, and they’re the 900 the broken binary produced. The 340 reservations that were merely touched during the same week don’t appear, because their price events came from a different commit.

The same metadata helps with the diagnosis. correlationId groups everything that came from one user action, and causationId chains back to the command that triggered this event. When the bad value comes out of a handler three hops from the request, that chain is how we find which request it was.

Without the SHA, we’re still ahead, because the position and timestamp of an appended event don’t move once it’s written. Still, I recommend to add the sha.

Fix forward, don’t rewrite

The instinct is to go into the store and edit those events. I’d steer you away from that. Events are immutable, and, surprising as it sounds, having precise history, bugs included, is a valid scenario. It’s often the only way to see later what really went wrong. I put it this way in the migration section of Simple patterns for events schema versioning: “you should not change the past... even including bugs, is a valid scenario.”

Instead, we could append a corrective event:

type ReservationPriceCorrected = {
  type: 'ReservationPriceCorrected';
  data: {
    reservationId: string;
    previousTotal: number;
    correctedTotal: number;
    reason: string;
  };
};

It’s the accountant’s move: we don’t rub out a ledger line; we post a correcting entry next to it. Then we rebuild the read models so the downstream views show the corrected total, and fix the bug so it stops producing new ones.

There’s a side effect that’s easy to miss. The correction is an event, so the next person to open this stream sees that a bug happened, when it was corrected and why. Nobody has to remember, and if we surface it in the UI, the front desk sees it too.

If we get the correction wrong too

We might. Say our correction script has its own bug and writes 1501.00 as the room total rather than the reservation total.

The recovery is the same operation as before, because the 14 March events are still there. We read the original calculation, the rate plan version that applied, our correction and its buildSha, and we append a second correction computed from the same inputs we had the first time. Nothing we did on 19 March destroyed the material we need on 2 April.

That’s the loop from the state-based version broken. Each attempt adds a fact rather than replacing the evidence, so attempt three is calculated from the booking, not from attempt two.

The customer already fixed it

Another reservation, four nights, two guests, 29 March to 2 April, at 210.00 a night. Rooms 840.00, tax charged 56.00, correct tax 41.00.

| when         | what                            |                                |
|--------------|---------------------------------|--------------------------------|
| 14 Mar 09:41 | `ReservationPriceCalculated`    | cityTax 56.00, total 896.00, build a4f9c2e |
| 15 Mar 11:20 | guest calls the hotel           | "that's not what I was quoted" |
| 15 Mar 11:26 | `ReservationPriceCorrected`     | total 839.00, by anna.k        |
| 17 Mar 08:03 | `ReservationConfirmationResent` |                                |
| 19 Mar 16:02 | we deploy the fix               |                                |
| 19 Mar 17:30 | we run the bulk correction      |                                |

Anna’s reason for that correction reads: city tax recalculated to 41.00, 5% goodwill on rooms agreed with FO manager. Our recalculation gives 881.00, and it’s correct by the pricing rules. Her 839.00 is also correct, because she took 42.00 off the rooms to settle an argument she was having on the phone.

If our bulk correction is a blind recalculation, at 17:30 it sets the reservation back to 881.00 and undoes her. The guest gets a third number in a third email and calls again. Support opens the reservation, sees the current total, and has no idea why it changed or that it was us. Anna’s note now describes a value that no longer exists.

With the stream, logically the check could look like in pseudo code (hey TS, no pun intended):

const needsCorrection = (events: ReservationEvent[]) => {
  const bad = events.findIndex(
    (e) =>
      e.type === 'ReservationPriceCalculated' &&
      e.metadata.buildSha === BROKEN_BUILD,
  );

  if (bad < 0) return false;

  // someone already dealt with it, leave it alone
  return !events
    .slice(bad + 1)
    .some((e) => e.type === 'ReservationPriceCorrected');
};

Everything it skips goes on a list for a person to look at. If we’d rather model the goodwill as its own DiscountGranted event, the rule becomes “skip if any price-affecting event follows the bad one”, which is the safer version anyway.

We can write that check because the events are there to check. In the mutable model, a bug and a deliberate correction look identical: a number in a column. There’s nothing to branch on, unless we can reconstruct intent from an audit table, assuming the support tool writes to one.

We could do such a check and fix as part of the one-time async job; we could even generalise it and run it as the SQL migration, running it only if it wasn’t run. We could optimise it by loading multiple reservations at once, then doing bulk append, if our store allows that, but the logical work remains the same. The rest is mostly performance optimisation.

Make the correction a feature

Better still, treat the corrective operation as a requirement and build it before we need it.

Ask the business “how do you fix this today?”. There’s almost always an existing answer: a form, a manager’s approval, a note in the folder. Model that. CorrectReservationPrice as a command, with permissions, validation and a mandatory reason, producing ReservationPriceCorrected.

Our bulk fix then goes through that command, with the same rules and the same events as when Anna runs it from the front desk. Nobody connects to the production database at 23:00.

And when the business says something can never happen, the follow-up is still: fine, how often?

There’s a harder version of this. I once watched a hotel checkout get stuck on a blocked financial account, which then jammed the entire night audit, with no way out except a migration or a hotfix. That’s the worst place to end up, telling a customer they can’t operate until we ship code. Bugs, bad data and stuck processes aren’t edge cases we can design away, which is the argument in No, it can never happen!: “We cannot assume that we won’t have a bug, or our hardware or network won’t have any random failure.” And if there’s no way to correct something inside the application, someone will do it with an UPDATE at 23:00. Read the horror story described there.

And check What texting your Ex has to do with Event-Driven Design.

“But now the history contains a bug”

Yes. The wrong price was real. It was shown to the guest, it went into a confirmation, it may have reached the invoice and the tax report. A log that shows only the corrected price can’t explain why the guest called.

If we need a clean stream later, for a regulator or a migration, still don’t edit in place. Copy and transform into a new stream, and keep the original.

None of this stops us shipping the bug, makes repricing 900 reservations free, or spares us the conversations with guests whose total changed twice.

What we get is better material to work with. We can name the events the broken build produced. We can read the value it wrote, with the inputs beside it. And we can see which reservations someone has already handled, and what they agreed to.

In a state-based system, fixing data overwrites the evidence we’d need to check whether the fix worked, which is how one migration ends up repairing the last one. Keeping the record of the mistake is what lets us check our own work afterwards.

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, and putting pressure on your local government or companies. You can also support Ukraine by donating, e.g. to the Ukraine humanitarian organisation, Ambulances for Ukraine or Red Cross.

There is one question I keep getting about my event-sourced code: why don’t I use Result?

It usually comes after someone sees a decision throwing an exception. They would put the failure in a Result, leave try/catch at the application boundary, and make the distinction explicit in the return type. That is a fair question, but the return type is only part of the answer.

I don’t mind using exceptions. I also return business failures as events. Which one I choose depends on what the failure means and what the application needs to do with it. What’s more, I don’t mind Result either, as long as I’m using a language where it’s a native and idiomatic way to use it. Stil…

Some failures are worth retaining as business data. An out-of-stock request can contribute to an unmet-demand report. A declined payment can start another process or be grouped by its reason. An item limit can explain why customers abandon an operation.

Gojko Adzic calls the systematic use of unexpected or marginal usage patterns to improve a product lizard optimization. Repeated out-of-stock requests can reveal demand we do not serve or a user experience that encourages impossible quantities. Turning every attempt into only a response or an exception log makes that pattern harder to find. An event gives the application data it can query, project, and use in another process.

Event Sourcing can help take advantage of unwanted and unexpected failures. We can model negative outcomes as events, just like regular ones. Our business logic informs us of what happened, not whether it’s a success or an error. It’s just an outcome of our decision.

So essentially we’re getting three options to deal with an unexpected scenario.

• throw an error (e.g. OutOfStockError) and catch it at the boundary;

• return a Result with the failure on its error track;

• return an event (e.g. ProductItemOutOfStock) and decide separately what we do with it.

Having that, my answer is rarely “just throw” or “just return Result.” Before choosing either, I need to know what should be persisted, what the caller should receive, and whether the failure belongs in the application’s exception path.

An error as a business fact

Let’s say that the customer asked for a quantity we cannot supply. Recording that outcome can be useful even though the cart did not change. The cart may also reach its item limit or fail payment authorisation. These are expected business outcomes, and the decision can describe each one with a distinct event:

const addProductItem = (
  command: AddProductItem,
  state: ShoppingCart,
): ProductItemAdded | ProductItemOutOfStock | ShoppingCartItemLimitReached => {
  if (state.status === “Confirmed”)
    throw new IllegalStateError(
      “Cannot add a product to a confirmed shopping cart”,
    );

  const { shoppingCartId, productItem, availableQuantity, maximumItems, now } =
    command.data;

  if (productItem.quantity > availableQuantity)
    return {
      type: “ProductItemOutOfStock”,
      data: {
        shoppingCartId,
        productId: productItem.productId,
        requestedQuantity: productItem.quantity,
        availableQuantity,
        attemptedAt: now,
      },
    };

  if (state.productItems.length >= maximumItems)
    return {
      type: “ShoppingCartItemLimitReached”,
      data: {
        maximumItems,
        requestedProductId: productItem.productId,
      },
    };

  return {
    type: “ProductItemAdded”,
    data: { shoppingCartId, productItem },
  };
};

ProductItemOutOfStock carries the requested and available quantities. ShoppingCartItemLimitReached carries the configured limit and the product that crossed it.

We could also model a single ProductItemAddingFailed event instead. It could have a reason string, but then we’d use the granularity and precision of the information. With distinct event types, a projection, workflow, endpoint, or test can check the relevant outcome directly.

What’s more, we could also add a FailedToAddProductToClosedShoppingCart and get rid of throwing entirely. We’ll get to that, but for now…

Broken rules can throw

Not every failure needs a business event. A confirmed cart can no longer be modified, so AddProductItem is not a valid operation for that state. A command without required data is invalid as well. An unavailable event store, a stock-service timeout, or a serialisation error means the operation could not be completed reliably. I use exceptions for these cases.

The throw happens before an event is returned, so the handler has nothing to append. The exception retains its stack and reaches the application’s error boundary.

Some people prefer to use Result and hope for deterministic decision-making. Still, in most popular dev environments (TypeScript, Java, C#, Python, etc.), we can get exceptions. Even in strongly-typed Rust, we can get a panic or other cases. In the presented code loadShoppingCart can reject because of data unavailability, appendToStream can report a concurrency or connection error, and base code library code can throw on a random, stupid null pointer. The application still needs an error boundary somewhere:

try {
  const result = await handleAddProductItem(command);
  return mapResultToResponse(result);
} catch (error) {
  return mapToErrorResponse(error);
}

That boundary might be inside WebAPI route, a message handler callback, or wrapped with a conventional error middleware. The remaining choice is whether expected business outcomes should use the same path.

What Result would add

Result puts the success value and the expected failures on separate branches, visible in the type like:

type Result<Success, Failure> =
  | { success: true; value: Success }
  | { success: false; error: Failure };

The cart decision becomes:

type AddProductItemFailure =
  | ProductItemOutOfStock
  | ShoppingCartItemLimitReached;

const addProductItem = (
  command: AddProductItem,
  state: ShoppingCart,
): Result<ProductItemAdded, AddProductItemFailure> => {
  const { shoppingCartId, productItem, availableQuantity, maximumItems, now } =
    command.data;

  if (productItem.quantity > availableQuantity)
    return {
      success: false,
      error: {
        type: “ProductItemOutOfStock”,
        data: {
          shoppingCartId,
          productId: productItem.productId,
          requestedQuantity: productItem.quantity,
          availableQuantity,
          attemptedAt: now,
        },
      },
    };

  if (state.productItems.length >= maximumItems)
    return {
      success: false,
      error: {
        type: “ShoppingCartItemLimitReached”,
        data: {
          maximumItems,
          requestedProductId: productItem.productId,
        },
      },
    };

  return {
    success: true,
    value: {
      type: “ProductItemAdded”,
      data: { shoppingCartId, productItem },
    },
  };
};

Technically it looks more or less the same: ProductItemAdded is returned on the success branch, while the other two errors are returned on the failure branch. The caller checks the branch before returning the event. In this application, adding the product changes the cart and produces a 204 response. Running out of stock or exceeding the limit leaves the cart unchanged and produces a 409 response. The handler appends only the success value:

const handleAddProductItem = async (
  command: AddProductItem,
): Promise<Result<ProductItemAdded, AddProductItemFailure>> => {
  const state = await loadShoppingCart(command.data.shoppingCartId);
  const result = addProductItem(command, state);

  if (result.success) 
    return PreconditionFailed();
  
  await eventStore.appendToStream(
    shoppingCartStreamName(command.data.shoppingCartId),
    [result.value],
  );

  return NoContent();
};

If we used event, it could look like that:

const handleAddProductItem = async (
  command: AddProductItem,
): Promise<Result<ProductItemAdded, AddProductItemFailure>> => {
  const state = await loadShoppingCart(command.data.shoppingCartId);
  const event = addProductItem(command, state);

  if (event.type === “ProductItemOutOfStock” || event.type === “ShoppingCartItemLimitReached”) 
    return PreconditionFailed();
  
  await eventStore.appendToStream(
    shoppingCartStreamName(command.data.shoppingCartId),
    [result.value],
  );

  return NoContent();
};

Both versions preserve the same three event types. Result adds a success-or-failure classification around them. Which can be fine, but if you look again at the sample, it’s still pretty easy to ignore the different consequences of the events. We’re just one check on success further from it.

I saw many codebases where people were just blindly muting all errors by default, getting no benefit from it, but just polluting the codebase. Also keeping in mind that they still need to wrap the codebase with try/catch and have a conventional mapping layer.

We can also see that the Result wrapper in our case doesn’t add much besides success/error classification. If we’d like to return different response status based on the failure scenario, we’d need to add the same switch. So we’re not removing the decision; we’re just making it easier to ignore.

Of course, many languages made it streamlined by adding pipe operator etc. Yet, again, many popular environments don’t have it. TypeScript has no pipe operator; the TC39 proposal is not (yet?) part of the language. Repeating the branch adds checks throughout the call chain; introducing a helper library adds vocabulary that is not native to the language. Those libraries are extremely noisy and require tribal knowledge and onboarding.

Scott Wlaschin, who popularised railway-oriented programming, makes the same qualification: Result models expected alternatives, but it is not intended to wrap every function or replace exceptions.

Same for events; we should always discuss with the business if this information is important for them and if they want to keep it. We should also ask what should happen and if we can and should recover in our business application when this scenario happens.

ProductItemAdded, ProductItemOutOfStock, and ShoppingCartItemLimitReached just tell what has happened. Domain doesn’t need to care how the application handles it. It can mean failure for the current request and still be worth recording because a projection or workflow consumes it. Adding those rules to Result would move persistence concerns into the domain decision and add premature classification. When we change the behaviour in the application and record one of those scenarios, we would need to change domain code, even if business rules are the same. Just to formally move one case from failure to success.

Adding Result changes every calling layer without replacing the persistence rules, batch handling, or exception boundary. I do not see much of a gain here. Of course, a codebase that already uses Result as its error channel may judge it differently, especially when the alternatives are not events. Thus…

Returning an event doesn’t mean persisting it

ProductItemOutOfStock might be needed after the request. Measuring unmet demand, suggesting alternatives, or showing failed attempts to support staff requires a durable event. Still, if our business doesn’t care about the fact and we just want to log it, or forward to the HTTP Response, then we may not need to record it. Appending it would retain data the application does not need. Omitting it from the decision would instead force the endpoint to reread the outcome.

For this example, the decision returns ProductItemOutOfStock to the caller and the stream remains unchanged. I call this selective persistence: the decision describes the outcome, while the application decides whether it should be durable.

Command Handling can be described by the following steps:

1. Read events from the stream and build the state from them (in other words aggregate stream).

2. Run the business logic using the command and the state. Use the default (initial) state if the stream does not exist.

3. Append the result of the business logic (so events) at the end of the stream from which you’ve read events. Use the read version (or the one provided by the user) for an optimistic concurrency check.

In a nutshell, we could generalise it as:

function handleCommand<State, Command, Event>(
  eventStore: EventStore, 
  streamId: string,
  decide: (state: State) => Event[],
  evolve: (event: Event, state: State) => State,
  initialState: () => State,
  // 👇 See what I did here
  skipEvent?: (event: Event) => boolean
  ) {
  // 1. Get the state
  const { state, currentStreamVersion } = await eventStore.aggregateStream(streamName, {
    evolve,
    initialState,
  });

  // 2. Run business logic
  const events = decide(state);

  // 3. Filter out “failure” events 
  const eventsToAppend = skipEvent ? events.filter(skipEvent) : events;
  
  // 4. Append events
  const appendResult = await eventStore.appendToStream(streamName, [event], {
    expectedStreamVersion: currentStreamVersion,
  });

  // 5. Return result
  return {
    ...appendResult,
    events, // 👈 all events
    appendedEvents, // 👈 only those that were appended
    newState: evolve(aggregation.state, event),
  }
}

A command handler reads a stream, rebuilds its state, runs a decision, applies the produced events, and appends them with an optimistic concurrency check. Emmett takes care of that flow. The application provides evolve to apply an event to state and initialState for a new stream.

We can pass skipEvent function that inspects the decision’s outcome before anything is appended. The decision still returns the same events; the function only changes how the handler treats them.

In Emmett, I allow it to encapsulate the Command Handler setup by:

// 1. Command handler setup
const handle = CommandHandler<ShoppingCart, ShoppingCartEvent>({
  evolve,
  initialState,
  middleware: [
    skipOn(
      (event) => event.type === “ProductItemOutOfStock” 
        || event.type === “ShoppingCartItemLimitReached”
    ),
  ],
});

// 2. Usage
const result = handle(eventStore, shoppingCartId, addProductItem);

I think that this setup clearly shows that we’re separating the business logic from how the business logic handles that, by explicitly using skipOn middleware, we’re telling that when such an event happens, we don’t want to store it, but we’ll still return it in the result.

This gives us the option to still do e.g. error mapping to HTTP status as:

import {
  Conflict,
  NoContent,
  on,
  ResponseFromEvents,
  toWeakETag,
} from “@event-driven-io/emmett-expressjs”;

const addProductItemApi = (router: Router) =>
  router.post(
    “/shopping-carts/:shoppingCartId/product-items”,
    on(async (request: AddProductItemRequest) => {
      const shoppingCartId = request.params.shoppingCartId;
      const productId = String(request.body.productId);
      const availableQuantity = await inventory.getAvailableQuantity(productId);

      const command = {
        type: “AddProductItem”,
        data: {
          shoppingCartId,
          productItem: {
            productId,
            quantity: Number(request.body.quantity),
          },
          availableQuantity,
          maximumItems: shoppingCartPolicy.maximumItems,
          now: new Date(),
      },

      const { events, nextExpectedStreamVersion } = await handle(
        eventStore, 
        shoppingCartId, 
        (state) => addProductItem(state, command),
      );

      if(events.some(event => event.type !== “ProductItemAdded”))
        return PreconditionFailed();

      return NoContent({
        eTag: toWeakETag(result.nextExpectedStreamVersion),
      }),
  });

We could, of course, do more advanced error or status mapping by returning exact data, etc. But this example proves that we’re not losing anything compared to the Result scenario. The code is still pretty simple, as with throwing, but more explicit and straightforward.

Several decisions on one stream

You may be wondering why I added middleware, which sounds posh for simple filtering. But let’s discuss one more scenario: a batch import from the external e-commerce system.

A batch import often contains the contents of multiple shopping carts. Passing all of them to one command handler would be the wrong boundary here: each cart has its own stream and its own concurrency check.

Even a single imported shopping cart can still require several operations. Especially if we’re trying to deduce what has happened to a shopping cart in the external system, or we should add some specific operations on top (e.g., discounts, etc.).

The common scenario for imports is translating state changes (e.g., products in the shopping cart) into commands. This gives at least a chance to make our domain context meaningful. Not perfect, but we’ve got to save ourselves sometimes.

After translation, we may end up with a sequence of granular business operations. The mapping could look like that:

const importCartCommands: AddProductItem[] = importedCart.productItems.map(
  (productItem) => {
    type: “AddProductItem”,
    data: {
      shoppingCartId: importedCart.shoppingCartId,
      productItem,
      availableQuantity: stockByProductId[productItem.productId],
      maximumItems,
      now,
    },
  }),
);

And feed it into our command handler:

const { events, nextExpectedStreamVersion } = await handle(
  eventStore, 
  shoppingCartId, 
  importCartCommands.map(command => (state) => addProductItem(state, command)),
);

Every command uses the existing addProductItem decision and sees the provisional state produced by the previous accepted products. That allows us to apply our own rules and also handle idempotence correctly. Persistence waits until every command has been accepted.

After batch processing, we can gather the result events. What’s more, each operation should use the state with the applied event from the previous command. We store the result in the shopping carts as a single atomic batch once the last command has been handled.

Technically it’s the same command logic, just wrapped with foreach, as we now have a sequence of decisions instead of the single one.

What should happen if we encounter unwanted scenarios? So e.g. ProductItemOutOfStock or ShoppingCartItemLimitReached? Should we stop processing or continue? Let me be a consultant for a moment: It depends! Fine, but on what? On your business logic.

In our case, we probably wouldn’t want to stop adding product items if one of them was out of stock. Tho, we may decide that there’s no point in processing it further if we already reached the items limit.

How to do it? We’d need to define a mapping function with different possible outcomes.

type MapDecision = <Event>(event: Event) => void | ‘APPEND’ | ‘SKIP’ | ‘STOP’ | ‘REJECT’;

Where:

• APPEND or no result - success, append this event,

• SKIP - ignore this outcome, don’t append it, but continue processing,

• STOP - ignore this outcome, stop processing and append only those that were already accepted,

• REJECT - reject the whole batch, even if there were some successful events.

The example could be defined for all shopping cart events not to need repeat it in all functions:

const mapShoppingCartDecision = (event: ShoppingCartEvent) => {
  switch(event.type) {
    case “ProductItemOutOfStock”:
      return “SKIP”;
    case “ShoppingCartItemLimitReached”:
      return “STOP”;
    case “ShoppingCartWasAlreadyConfirmed”:
      return “REJECT”;    
  }
}

We could pass it to our command handler and decide what to do with our processing.

In Emmett, I wrapped it as:

const handle = CommandHandler<ShoppingCart, ShoppingCartEvent>({
  evolve,
  initialState,
  middleware: [
    skipOn(
      (event) => event.type === “ProductItemOutOfStock”,
    ),
    stopOn(
      (event) => event.type === “ShoppingCartItemLimitReached”
    ),
    rejectOn(
      (event) => event.type === “ShoppingCartWasAlreadyConfirmed”
    ),
  ],
});

Still, as you see, we’re building on top of our business logic. It remains the same; we’re just handling persistence, statuses, etc.

Returned events show which command handling passed before the rejection and which event stopped the import. The importer can report that outcome against the source record and continue with the next cart, which has a different stream.

For one product, rejecting meant leaving the stream unchanged. For an imported cart, it may also mean discarding the accepted events produced earlier for the same record.

If every command is accepted, the handler appends their events together to that cart stream. If one decision returns ProductItemOutOfStock or ShoppingCartItemLimitReached, result.events contains the outcomes produced up to that point, the imported record makes no change to the cart, and later commands are not considered.

The import job can map those events to a record-level outcome. It can report the exact business issue to the source system and continue with the next cart:

const validatedProductIds = result.events.flatMap((event) =>
  event.type === “ProductItemAdded” ? [event.data.productItem.productId] : [],
);

const failure = result.events.find(
  (event) =>
    event.type === “ProductItemOutOfStock” ||
    event.type === “ShoppingCartItemLimitReached”,
);

const importOutcome = (() => {
  switch (failure?.type) {
    case “ProductItemOutOfStock”:
      return {
        status: “WaitingForStock” as const,
        validatedProductIds,
        productId: failure.data.productId,
        requestedQuantity: failure.data.requestedQuantity,
        availableQuantity: failure.data.availableQuantity,
      };

    case “ShoppingCartItemLimitReached”:
      return {
        status: “Rejected” as const,
        validatedProductIds,
        productId: failure.data.requestedProductId,
        maximumItems: failure.data.maximumItems,
      };

    default:
      return {
        status: “Imported” as const,
        importedProductIds: validatedProductIds,
        streamVersion: result.nextExpectedStreamVersion,
      };
  }
})();

await importStatusStore.record(importedCart.externalId, importOutcome);

switch (importOutcome.status) {
  case “Imported”:
    await salesChannel.confirmImport(
      importedCart.externalId,
      importOutcome.streamVersion,
    );
    break;

  case “WaitingForStock”:
    await pendingImports.waitForStock(
      importedCart.externalId,
      importOutcome.productId,
    );
    break;

  case “Rejected”:
    await salesChannel.rejectImport(importedCart.externalId, {
      productId: importOutcome.productId,
      maximumItems: importOutcome.maximumItems,
    });
    break;
}

A saved-list restore uses different rules. Available products return to the cart, unavailable ones are reported, and reaching the cart limit ends the restore. Products added before either outcome remain valid, so this operation is not atomic.

The additions run in one call because every decision needs the cart state produced by the accepted products before it. Otherwise, each decision would evaluate the item limit against an outdated cart.

The loop stages and applies available products. It returns an unavailable product without staging it and continues. It also returns the item-limit event without staging it, then stops because the cart cannot accept another distinct product

And here we’re getting back to the essential question.

So should we throw or not?

If we’re getting the request from our UI, we’re the authority, so throwing could be fine; we return the status information telling the computer says no. (Read more in How to validate business logic).

Still, should we throw on an import scenario? Who’s the authority here? What we’re doing is internalising decisions that someone else already made and is just informing us about, or telling us to do our job based on this information.

So throwing is not a valid option here. The external system told us that its cart contains these products. That is true about the source system, but it does not mean our cart can accept them. The importer translates the external record into commands, and the returned events describe how our local model understood each products.

For an accepted record, the ProductItemAdded events identify the imported products and the resulting stream version is recorded. If stock is missing, the importer records the checked products and the missing quantity. It can continue with other records, then rebuild this cart’s commands with current availability after an inventory update. Retrying the unchanged commands immediately would only reproduce the same outcome. If the cart exceeds its item limit, waiting for stock cannot help; the source record needs correcting, so the importer marks it as rejected.

Of course we could still throw if this operation is synchronous and we want to give quick feedback to the caller. Still, typically, we run batch operations as a background process; also, to have proper resilience, you already saw above how much integration could happen during such an import. Services we integrate with can be unavailable, have transient issues, time out, etc.

Those failures enter the import job’s retry or dead-letter policy because there is no reliable business outcome to report yet. The returned events let the application choose between continuing, waiting for a relevant business change, and rejecting one record. Exceptions would just stop processing and not allow recovery and continue with next steps (Read more in No, it can never happen!)

There can be various error cases also on our side.

If another request changes the cart before our append, Emmett handles the concurrency conflict by loading the stream again and rerunning the decisions against its new state. Decision middleware runs again as well.

An out-of-stock import is not a concurrency conflict. The cart state did not become stale; the input said that only two items were available. Running the same command again still returns ProductItemOutOfStock. The importer waits for an inventory change and then builds a new command with the current availability.

A custom retry policy can inspect a completed attempt, but it runs after that attempt’s append phase. Retrying a result is safe only when nothing was appended, and useful only when state or input can change.

A decision and its middleware can therefore run several times during one handler call. Code that only reads the command and state can be rerun. Sending an email or charging a card inside that code would repeat the external effect. In these examples, required external data is fetched before invoking the handler and included in the command. Effects caused by appended events run later in processors, after persistence.

Middleware could also be useful for authorisation, logging, and measurements. Some of that work belongs around each decision; some should run only once for the complete handler call. Its position relative to the retry boundary determines whether it sees aggregate state and whether a retry repeats it:

await beforeAll(input);

const result = await retry(async () => {
  const state = await aggregate(streamName);

  for (const decision of decisions) {
    await runWithDecisionMiddleware(decision, state);
  }

  return appendSelectedEvents();
});

await afterAll(result);

return result;

The callback before the retry boundary receives the complete input once. It can authorise the command batch without repeating that work after a concurrency conflict. State is not available yet because the stream has not been loaded.

Middleware inside the boundary receives each decision, and the state is rebuilt for the current attempt. A concurrency retry runs it again.

The callback after the boundary receives the final result once, after persistence. It can record measurements based on appended events or the new stream version. If it throws, already appended events remain committed. Validation belongs earlier; notifications that must be delivered need a durable handler or outbox.

Never Throw in Asynchronous Handlers

As you see, message processing can end up being pretty complex. That’s also why we use Event-Driven Architecture, as it helps to make that explicit and reason about it.

The import we discussed could be (and probably should be) triggered by an event (e.g. through a webhook), as we’re informed that those items were already processed and we should just accept or ignore it. What has been seen cannot be unseen.

Whether throwing is safe depends on what sits above the code to catch it. As discussed, a command handler behind an HTTP endpoint, its usual home, has the request: a thrown error unwinds into a response.

Asynchronous handlers, the projections, reactors, and workflows that react to events, have nothing above them. A throw propagates up, stops the processor, and leaves later events unhandled. The endpoint is only the usual home, not a rule: a command handler run from a reactor loses the same safety net. Instead of throwing, we need to turn the problem into data, so return an event.

A message handler can turn a declined shopping cart operation into a compensating workflow. For instance, for ProductItemOutOfStock order more products from the producer. It can, of course, just skip a message that should not block processing, or stop without advancing further.

The same for read models. A projection cannot prevent the source event from being recorded because that event is already in the stream. Throwing only prevents the read model from advancing. If an event carries a value the read model did not expect, throwing does not undo it. The check that should have stopped it belongs in the business logic, before the event was ever recorded; by the time the projection runs, it is too late.

It builds a read model from events that are already recorded, and a recorded event is a fact: the projection only interprets it, so it has nothing to reject. If an event carries a value the read model did not expect, throwing does not undo it. The check that should have stopped it belongs in the business logic, before the event was ever recorded; by the time the projection runs, it is too late.

So accept the event and build the best read model you can from it. Skip a duplicate, fall back to a default, clamp a value back into range, whatever keeps the read model sensible and moving. Its error handling must account for delivery and recovery rather than reuse the rules of a command decision.

Conclusion

The decision returns ProductItemAdded, ProductItemOutOfStock, or ShoppingCartItemLimitReached because the application needs to know which one happened. Out of stock is an unfavourable outcome, but that alone says nothing about storage. During an import it cancels the current source record. During a saved-list restore it is skipped while earlier additions remain. At the HTTP endpoint it becomes a conflict response. The event stays the same; its handling depends on the use case.

I do not try to remove exceptions from this design. Adding a product to a confirmed cart is an invalid operation. An unavailable event store prevents the handler from producing a reliable result. Both go to the application’s error boundary.

Result could label some of the returned events as failures, but persistence, continuation, and retry still need their own decisions. The exception path remains as well. The event types already distinguish the business outcomes, so I prefer to leave them as they are.

So when someone asks me whether they should throw an error or use a Result type, I sometimes tell: actually, you can do neither.

Cheers!

Oskar

p.s. if you liked this one, check also:

• Idempotent Command Handling

• Against Railway-Oriented Programming by Scott Wlaschin

• Emmett: Error Handling

• Emmett: Command Handling

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, and putting pressure on your local government or companies. You can also support Ukraine by donating, e.g. to the Ukraine humanitarian organisation, Ambulances for Ukraine or Red Cross.

If this story were written in Fakt, Bild, or Sun, it could have been titled:

Trained for 6 years. Finished as the last one.

That’d be almost true about what has happened to me. Almost would already be a lot for Fakt, Bild or Sun. Now, let’s go with the full story.

I’m a weird type of guy, I don’t have a driving license. And neither did I have it. I learned to ride a bike when I was 9 years old and got it for my First Communion. And I didn’t learn to swim till I was 35. Yes, that happened 6 years ago.

Did I have some trauma, like drowning? Nah, just didn’t happen. Maybe because my father couldn’t swim, maybe because I didn’t have motivation. Dunno, just didn’t happen.

Of course, not having certain skills can be fine; you can learn to live without them. You can even be considered a fashionable hipster. Some folks consider my lack of a driver’s license as a personal worldview stand. The fact that I like to cycle or ride a train suggests to some that I’m an ecologist. And don’t get me wrong, I like nature, I try to be a good citizen, but that’s not my main motivation.

This is just what I do, not who I am.

So am I a swimmer now? Well, it seems that I can swim now. I took classes from 2018 to 2019, then stopped for about 3 years, and got back into it when I signed my daughter up for swimming classes.

Now, when I say I’m practising swimming, everyone assumes I’m a triathlete preparing for a competition. And well, I’m not.

I don’t even like to swim much. I mean, it’s fine, I don’t mind, but it’s not that it’s my passion. I don’t have thoughts like:

Hell, I’d like to swim now.

And I still have thoughts like:

Hell, I’d play football now.

Which I don’t do, as I don’t play football anymore. Even though it was my passion for most of my life, I wasn’t talented at it, but I liked doing it. I even played as a junior for my hometown club (Miedź Legnica) and later in amateur leagues. And in those amateur leagues, I tore something called the lateral patellar retinacula. Actually, two of them, in both knees. Which means that my knees are unstable, and if I continued to play, it’d end up with some challenging surgery.

That also means that I had to decide to be unsuccessful somewhere else.

So, I’m an unsuccessful footballer and now an unsuccessful swimmer in the making.

I tried running and going to the gym, but it was always boring for me. I could do it for a few months, but then I quit. And off we go, for not training.

And I tried swimming.

Why? We don’t need to be capable of everything; it’s often a fine choice not to learn everything. At some point, a lack of certain skills can be limiting. You already know that it wasn’t my worldview to always have my feet on solid ground, but the main motivations were:

• It’s nice to do some swimming on holidays,

• It’s NOT nice to be a lame dad who can’t swim with his daughter,

• I kindaish always wanted to learn that, just didn’t happen,

• It’d be good to do some exercises, keeping in mind my sitting style of work,

• I had a swimming pool 10m by walk,

• I had and still have a great teacher. Thanks, Justyna!

So I started learning to swim again in 2024. It’s an interesting exercise when you learn something new as an adult. It’s more challenging because you’re overthinking rather than acting on your instincts. Or maybe your old habits are making your instincts unintuitional. It also shows how everyone has a different way of learning.

Take Breaststroke style (in Polish called “Żabka” - frog). Most people claim that it’s the easiest style to learn. For me? Madness, how much stuff you need to remember and coordinate: hands, back, legs.

It also shows how everything is connected. It appeared that I couldn’t do it properly because I wasn’t stretched enough. Only after I started stretching was I able to move my back and twist my legs correctly. Well almost. Almost in the Fakt, Bild and Sun way.

Ok, so I learned to swim kindaish. But the best progress I made was last year.

When we started the season in September, my coach told our group that we’d start competing. I said hell yeah, and Justyna took the irony for a real agreement. So it started.

It appeared that the shortest possible distance is 475 meters, which is around 19 swimming pool lengths. Ah, and it’s on the lake, deep for 22 meters. I didn’t know about that last part when I signed for it. I also didn’t know if I could swim for so long with the crawl style. I had probably a record of around 6-8 swimming pools.

And well, I made it.

Was I stupid or just dumb? I hope that neither of those. I used the best way to learn: practical and deadline-driven.

Instead of one training per week, I started doing two. I took some private lessons with Justyna and also started doing longer swimming sessions. Slowly but surely, I beat my record to 10, 15, 20, then 40.

Yes, it was stressful; yes, it required a patient, slow process, but it gave me at least the trust in myself that if I can swim 40, then I’ll probably make 19.

It also helped to do a real dry run in the lake and the sea. If you only swim in the swimming pool, you’ll be surprised how different swimming in a lake is.

The water is dirty; it’s super easy to lose your bearings and go in a longer circle than you should.

And guess what? The real competition brings even more surprises. The crowd of other swimmers is a challenge. Constant pushing on the start, distracting from catching the flow. Then stress, and moments of doubt about yourself. You can prepare for that, but only to some degree; the real learning comes through the next stage, which starts in competition.

Were I actually the last one? No, I was 53 out of 59. Seems I wasn’t all bad. Just half bad. I had 14m 20s time. My best on practice was 12m.

Still, not too bad for a guy who learned to swim at 35, if you ask me. I’m super proud of myself.

My friend Damian (jokingly) told me that I should write this as

“10 things I learned about EDA from practicing swimming and preparing for a competition”

And that’s not going to happen, but let me give an inspirational quote from the man, the legend, Michael Jordan:

I don’t compete with other people. I compete with what I’m capable of.

And that’s what I did. Of course, Michael wasn’t competing with others, as he was the best, GOAT. I wasn’t competing, as I was one of the worst.

Yet, I was still much better than my past self. And I also achieved more than those who haven’t even tried.

Interestingly, I was sometimes perceived as humble, and sometimes as too self-assured. Maybe it comes from a common answer when someone asks me if I can do something:

I don’t know; I haven’t tried it yet.

But I think that stems from my belief that we are all human. We’re not worse than others, but we’re also not better than others, which can be seen as bold or humble, depending on the focus of this belief.

We won’t know what we’re capable of until we try it. And when we do, we’ll already be much further than your past selves that haven’t tried.

I’m not trying to say that we can achieve everything; I probably won’t ever be a good swimmer. If I could achieve that, then anyone can. I’m not saying that’s going to be easy. It’s not. It requires time. And time is a precious resource. Not infinite. The best way to find it is to find it. But to do it, you need to set a priority, which means something else will become less important.

Pick wisely.

Should you also swim? Well, maybe. You definitely should do some exercises if you’re a developer like me. But precisely swimming? Up to you. Should you get a hobby other than work? Definitely.

Why did I write this article? A bit to brag, as I’m happy I made it.

Brag about being 53 out of 59 after 6 years of training? Seems so! You need to cherish small successes that, from the right perspective, appear to be big ones.

So, are there 10 things you can learn about software engineering from practising swimming and preparing for a competition?

I don’t know; you tell me.

Cheers!

Oskar

p.s. if you liked this one, check also:

• Borys had the best dribbling,

• How playing on guitar can help you to be a better developer?

• Don’t be like Ebenezer Scrooge. A few words about workaholism

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, and putting pressure on your local government or companies. You can also support Ukraine by donating, e.g. to the Ukraine humanitarian organisation, Ambulances for Ukraine or Red Cross.

And, well, that’s probably not what our user wanted. They might just want a sharper steak knife, so something more like:

I’m also to blame; I’m falling into precisely the same trap too often. Like recently, while working on Strictland, the contract testing library I announced in the previous article.

The idea behind it is to detect message schema drift easily. You write such a test:

@Test
void orderPlacedSchema_DoesNotChange() {
    // Strictland specification
    MessageContract.specification(Json.Jackson.defaults())
        .given(new OrderPlaced(orderId, "Alice", placedAt))
        .whenSerialized()
        .thenContractIsUnchanged();
}

It serialises the message and stores the payload inside the same git repository as the test. Thanks to that, you can keep it close and detect contract changes during the review by looking at the changed files.

Initially, files were kept in the same folder as the test file. That made it accessible, as you don’t need to jump from one folder to another and see related test files. The file was named by message type class name (so in this case it’d be OrderPlaced.approved.text). Simple, but what if someone adds:

• Multiple tests, trying to store different snapshots for the same message type?

• Multiple test files dispersed across different folders that test the same message type (e.g., in the features that consume those messages)?

• Multiple message types across different modules/packages with the same name?

• multiple tests for different message-type versions and variants (e.g., with all fields, with only required fields, etc.)

All of that is solvable by some conventions, but what if you’re an open-source library creator like me? Then you can’t assume the same type of thinking, level of tidiness or knowledge of your users. You should be the one to take on the pain and make your users' lives easier. And guess what? That applies not only to open source, but also to other products.

“Taking the pain” and “helping users” can be interpreted as an attempt to please everyone, which translates to the mentioned earlier: “I want it all, and I want it now.”

That’s what I tried to do. I decided to add “selectable storage layouts strategies”:

• Near the test file - so similar to the one we had,

• In the nested folder near the test file - similar, but separating the snapshot files from tests, thanks to that, you could look in the nested folder, keeping the tests not polluted by the numerous tests,

• In the root folder - putting it close to the root folder, keeping snapshots together- still the same repo but giving potentially better accessibility.

And ”selectable storage layouts strategies” were not enough for me. I also wanted to add “various grouping strategies” where one could select whether they want to group message snapshots by:

• Message type name,

• Test files and their names.

• Or just apply no grouping.

It sounded smart to me, and hell, with GenAI, I wouldn’t even need to fight with it, as my friend Claude would do it for me, right? RIGHT?

Now, ”selectable storage layouts strategies” should sound enough as a warning to me and “various grouping strategies” should ring a loud alarm bell. But it didn’t. I planned how to do it and started the implementation.

A few iterations later, the permutations of those various strategies started to pile up. Some of them were just hard to do. Some were conflicting. And most of them weren’t even useful.

Examples? I decided to make all files follow the unified format:

{messageTypeName}.{version}.{variant}.snap.approved.{extension}

And that’s fine, but then what to use as the message type name? Class name? Fine, but what about a flat layout, or grouping by message type name, with multiple messages of the same name?

Let’s use the full class path as the name? Fine, but then what if you have a root folder and a nested folder strategy? Would you nest folders by the class path, and then repeat it in the name?

Or maybe you’d differentiate the file name format depending on the strategy one chose?

Ok, but what if the user decides to change the strategy at some point?

I decided to run a sanity check I should have done earlier. Better later than too late. The discussion inside my head looked more or less like this:

• Me: What’s the actual goal of this library?

• I: To detect message schema drift.

• Me: Ok, then do I need to build the snapshot library?

• I: To some degree, I’m using snapshot comparison to detect that drift.

• Me: Sure, but I probably don’t need to cover all the cases that the generic snapshot library has to, right?

• I: Right.

• Me: Ok, so what’s our bet for users? Are they advanced or just beginning the journey?

• I: At the beginning.

• Me: Then I should help them, set a sane default, and guide them. Especially since this is the early phase of this library, I haven’t yet received much feedback.

• I: Agreed, maybe we could start then with the recommended layout and grouping, and then gather feedback. I can add more strategies as needed.

That sounds like an obvious thing to do, but it requires pausing to find the time to evaluate our assumptions. We shouldn’t always be mad at ourselves for not predicting everything earlier. Some stuff is just hard to predict. Personally, I usually see some steps after I try to implement them. Those are also the steps I explained in the article about the OODA loop. We should:

• Observe - This is the intake of raw, unfiltered information. In our world, this means looking at the state of the system.

• Orient - This is the most critical and difficult stage. It’s where we filter our observations through your experience, culture, and technical knowledge.

• Decide - Based on our orientation, formulate a hypothesis.

• Act - We execute.

Rinse repeat.

If we think about it this way, the time spent considering those various cases is not wasted. It’s similar to the anti-requirements exercise described by Andreas Öhlund and David Boike or Barry O'Reilly’s Residuality Theory. We gather numerous needs and ideas about our product evolution and see what sticks and what remains. Not all ideas will be great; some will be bad, but that doesn’t matter; it gives us food for thought.

Still, we can’t forget about the most important part: subtraction.

We need to subtract all the stuff that’s not needed, leaving only the useful residues, the one that solves the main goal.

For my case, I realised that if I chose the single strategy, it would organise my contract into the root test folder like this:

📁 src/test/resources/
  📁 contract-registry/
    📁 com/acme/orders/OrderPlaced/
      📄 OrderPlaced.1.default.snap.approved.json
      📄 OrderPlaced.2.default.snap.approved.json
    📁 com/acme/orders/OrderInitiated/
      📄 OrderInitiated.1.AllData.snap.approved.json
      📄 OrderInitiated.1.OnlyRequired.snap.approved.json
      📄 OrderInitiated.2.AllData.snap.approved.json
      📄 OrderInitiated.2.OnlyRequired.snap.approved.json
    📁 InvoiceIssuedEvent/
      📄 InvoiceIssuedEvent.1.default.snap.approved.json

By nesting by message type and full path and grouping by contract name, I’m not only keeping snapshots but also creating a Contract Registry.

So this subtraction of some features not only gave me focus on the specific scenario but also enabled me to create scenarios that I would actually like to have eventually, for instance:

• keep the message schema in the same folder as snapshots,

• markdown docs of the message,

• Having defined the folder structure, generate documentation,

• check dependency, and who uses that,

• etc.

That led me to conclude the following two rules:

1. The addition of imaginary features subtracts the actual value.

2. The subtraction of imaginary features adds the actual value.

It’s also aligned with my motto: Removability over Maintainability.

What’s the lesson then?

We’re not wasting time on building too slowly; we’re wasting time on building stuff that doesn’t matter. All those useless features we imagine without evaluating them with people are a wasted opportunity to deliver better for them.

Nowadays, with LLMs, it’s tempting to rush into implementation and add every feature we can imagine. That’s never free. We’re increasing the cognitive load for our users and also for us; we waste time on stuff that doesn’t matter and are constantly distracted.

Yet, if we don’t do sanity checks, trying things in a proof-of-concept way can be a big multiplier. If we stop after that and rethink.

I’d still advocate doing a sanity check earlier than I did in the described scenario and thinking first, but not all can be forseen like that. So:

1. Generate as many ideas as you can.

2. Try to think about how they match together, think about the tradeoffs, the cost of adding them, the cost of removing them and the cost of changing your mind.

3. Define potential solutions.

4. Stop the addition phase and rethink what you came up with. Ask yourself what you’re building and what the actual goal is.

5. Start the subtraction phase and cut as much as you can on the stuff that’s actually not needed. Of course, don’t drop ideas, remember that quantification is subjective. Deliver a simple solution that will give you enough optionality to change your mind.

6. Send it wild and gather feedback.

And that’s precisely what I did: I just released Strictrland 0.4.0 and will plan the next steps based on user feedback!

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, and putting pressure on your local government or companies. You can also support Ukraine by donating, e.g. to the Ukraine humanitarian organisation, Ambulances for Ukraine or Red Cross.

Before I go further, if you can’t wait, you can check it on:

• GitHub - https://github.com/event-driven-io/strictland

• Maven Central - https://github.com/event-driven-io/strictland

So now you already know that it’s a JVM (Java, Scala, Kotlin, etc.) Open Source library.

Let me first show you a sneak peek. Stop for a moment and have a look on this:

@Test
void ensureOrderPlacedCompatibilityWithNewerVersion() {
    // Strictland specification
    MessageContract.specification(Json.Jackson.of(yourObjectMapper))
        .given(new OrderPlaced(orderId, "Alice"))
        .whenDeserializedAs(OrderPlacedWithCoupon.class)
        .thenBackwardCompatible();
}

What would you say if you saw such a test? Think about it, we’ll get back to it.

What’s Strictland?

But again, why did I do it if there are mature solutions like Pact, Spring Cloud Contracts, or Confluent Schema Registry?

If you've used consumer-driven contract testing, the usual approach is to run both the provider and the consumer, record the consumer's expectations against a mock, verify the provider against those expectations, and share those contracts through a broker.

In Strictland, I took a smaller, simpler approach. It serialises the message in a standard unit test and saves the output as a snapshot file that you commit alongside your code.

The test fails when the serialised shape changes, or when it contains a breaking change - up to you to specify expectations. A check confirms that an older and a newer version of the message can still read each other's data (or the other way round).

Because it's only serialisation and a file, the setup stays small:

• The checks are ordinary unit tests in your existing suite, so there's no broker, schema registry, or mock service to run, and nothing to start in Docker.

• The contract is the serialised JSON committed alongside the test, so any format change appears in a normal diff and is reviewed like any other code.

• You write the check beside the message it covers and get the answer in the same fast feedback loop as the rest of your tests.
  The check uses your application's own serializer, so the snapshot is the exact bytes you ship.

• Strictland checks the serialised shape of a message and whether its versions stay compatible. It doesn't exercise a live exchange between running services, so it complements that kind of tooling rather than replacing it.

Strictland checks the serialised shape of a message and whether its versions stay compatible. It doesn't exercise a live exchange between running services, so it complements that kind of tooling rather than replacing it.

It's not as powerful as popular tooling, but it's also much simpler to start catching our mistakes. Traditional solutions allow you to mock protocols, put a man in the middle, and even generate client code. All of that is great if you need it and have experience with it.

Most of the customers I’m helping through consultancy and training aren't there yet. Setting up those tools is a lot of heavy lifting for them and adds additional complexity. And well, maybe they don’t need to be, as this approach served me well in my past projects. I always handcrafted such a tool in my projects, but finally decided to make it properly.

Why such a name?

I’m putting (too?) much effort into my projects’ names.

It's a word game. Contract testing rewards a strict approach to your message shapes, and Mr. Strickland was strict enforcer in Back to the Future. That puts it in good company next to its sibling Emmett, named after Doc Emmett Brown. I also didn’t want to collide with the older JS validation package.

How to use it?

Getting back to the essence… How to use it?

You write a small unit test that locks down a message’s format. Later, you rename a field, change a type, or adjust how a value serialises; the code still compiles and your other tests pass, but that one fails and points at what changed. You fix it in your build before a consumer or a stored event has hit the old format in production.

When a message changes by accident, a snapshot check shows you exactly what moved. When you evolve a message on purpose, a compatibility check confirms an old and a new version can still read each other’s data.

You can start by installing it from Maven Central, by adding dependency.

<dependency>
  <groupId>io.event-driven</groupId>
  <artifactId>strictland</artifactId>
  <version>0.3.0</version>
  <scope>test</scope>
</dependency>

Then adding such test:

MessageContract.specification(Json.Jackson.defaults())
    .given(new OrderPlaced(orderId, "Alice", placedAt))
    .whenSerialized()
    .thenContractIsUnchanged();

The first run serializes the message and writes the result to an approved file named after the class, OrderPlaced.approved.txt, saved next to the test:

{"orderId":"00000000-0000-0000-0000-000000000001","customer":"Alice","placedAt":"2024-01-01T12:00:00Z"}

You review that file and commit it. From then on the check compares against it and fails if the format drifts, so a later change to the format shows up in the same pull request as the code that caused it.

A message under contract goes through one of two checks.

A snapshot check confirms the message still serializes exactly as it did when you last approved it, so nothing reading it downstream breaks. A failure means the format changed: a field renamed, a date format switched, a value newly dropped or added.

A compatibility check is for the version you evolve on purpose, so changing a message doesn’t leave the ones already in your store or on the wire stranded. Use thenBackwardCompatible() to confirm the newer version still reads a message the older one wrote, the events you stored last year, or a request already sent. Use thenForwardCompatible() to confirm a reader that hasn’t upgraded yet still reads a message the newer version writes, so you can ship the new shape before everyone reading it has caught up. Both compare the fields the two versions share and fail if a required one is missing or a shared value has changed.

@Test
void ensureOrderPlacedCompatibilityWithNewerVersion() {
    // Strictland specification
    MessageContract.specification(Json.Jackson.of(yourObjectMapper))
        .given(new OrderPlaced(orderId, “Alice”))
        .whenDeserializedAs(OrderPlacedWithCoupon.class)
        .thenBackwardCompatible();
}

You review that file and commit it. From then on the check compares against it and fails if the format drifts, so a later change to the format shows up in the same pull request as the code that caused it.

As you see, in the specification, you pass your serializer.

var spec = MessageContract.specification(Json.Jackson.of(yourObjectMapper))

I encourage you to use your application's object mapper and pass the same ObjectMapper it uses, so the test checks the exact bytes you ship. That’ll help you avoid nasty surprises, where tests are using a different format than your framework or tooling.

Strictland provides an implementation of a sensible Jackson setup: ISO-8601 dates, nulls kept, unknown properties ignored on read. You can use it with Json.Jackson.defaults(). But it’s more of an accessible thing to get you started quickly.

You can also define your own serializer if you’re using an unsupported (yet?) format or serializer type. See basic examples in:

• CsvMessageSerializer and its tests or,

• SimpleBinaryMessageSerializer and its tests.

Should you use it?

Strictland is young and pre-1.0, so the API can still move between versions. The checks themselves are small and well covered, and the snapshots they produce are just files in your repository, so trying it out costs little and commits you to nothing.

I’d genuinely like your feedback. If something is missing or awkward, tell me through comments, on Discord or open an issue.

Currently, I’m working on making snapshot location more organised and configurable, not to end up with a bloated snapshots tests folder.

I also need to rethink the given name; it’s from Behaviour-Driven Design, but it appears to be a reserved word in Scala 3. Cross-language and cross-stack libs are much fun!

For now, for JVM, as the Java-based customers I’m helping also needed that, but soon want to add .NET and TypeScript/Javascript (and maybe more).

Tell me your honest thoughts, give it a try and a star if you liked it. Play with it, and pass me your feedback. It’ll motivate me to continue working on it!

Would you like to learn more about consumer-driven tests and how Strictland can be used for them?

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

Fork your dependencies, trim them to only your use case, never update unless it breaks for your users. I’ve been vocal about this for 10+ years. I’ve always said that updating is way riskier than latent bugs (which can be tracked and CVEs monitored).

If you are updating a dependency, it’s on you to analyze every single commit in the full transitive set of dependencies. If you dont see anything compelling, dont update!

I remember at HashiCorp once in awhile an engineer would try to update a dep or replace a DIY lib with an external one and id always ask “show me the commit we need.” Dont update for the sake of it.

Feeling pretty swell about this mentality with all the supply chain attacks happening.

That's from Mitchell Hashimoto. A friend sent it to me, and the first word he reached for was bold. Mine too. I mostly agree with him, honestly. But the second I read it, my head jumped to a caveat, and the caveat turned out to be the thing I actually wanted to write about.

You can fork a small library and trim it to what you use. I’ve done it, it’s fine. But could you fork React and maintain it? I couldn’t. I don’t think most teams could either. So “fork your dependencies” is wonderful advice right up to the point where the dependency is too big to hold in your hands, and then it quietly stops being advice at all.

And that caveat says something about the advice itself. I think that it was never really about forking. It’s about knowing exactly what you’ve taken on and being willing to own it, and that’s the bit most of us skip. We don’t decide to take a dependency. We install one. And almost everything that goes wrong with dependencies:

• the supply chain attacks,

• the licence dramas,

• the half-finished SBOMs,

• the things you find out about only after they break

comes back to that one move: we took dependence on someone else's product without ever deciding to. Everything else here is a variation on it.

To fork or not?

I was reminded of this not long ago. I was doing a live Q&A about my TypeScript port of some code, and a whole chunk of it was one question asked five different ways:

Why did you write your own deepEquals instead of just pulling a package?

I tried to explain that it wasn't laziness, and it wasn't not-invented-here pride either. I explained that it’s the code I write once and forget. Of course, it’ll take some time, but not that long as you might think. The reception was, let's say, mixed. I get it, it’s much easier to just install a package and outsource maintenance. Especially when you're in a hurry, you don't decide anything. You reach. And this reach can be a decent interim solution, but then you should reflect on yourself on the next step.

Why do I reach for my own code on the small stuff? Look at which packages actually get hit in all these supply chain attacks we keep reading about. It’s almost always the little ones. The one-liners. Remember the Left-pad Incident? It’s usually some tiny or low-level helper nobody thinks about. They’re everywhere, they’re trivial, and precisely because they’re trivial, nobody is watching them. Which is also exactly why they’re so easy to write yourself. So for me, handwriting a small helper isn’t paranoia. It’s the calmer option, and it’s one of the few cases where I know exactly what’s in my own system.

That's also why, in Emmett, my Event Sourcing library, I'm almost stubborn about keeping the core package free of dependencies and limiting whatever I inject elsewhere. Probably too stubborn, if I'm honest. But I'd rather err that way, because when it goes wrong on the user’s side, it goes wrong far worse than a bit of code I maintain myself. It’s about responsibility for the outcome of your actions.

If I had to compress what I believe into one line, it wouldn’t be “fork everything”, and it wouldn’t be “write everything yourself”. It would be something duller than both:

The number itself isn’t the point. It just tells you how much you’re agreeing to own without ever seeing it.

Because your direct dependencies were never the worst part. You chose those. Most of the time, you can read them, and you can follow what they’re doing. The scary part is the dependencies of your dependencies, and theirs, all the way down, the part you didn’t choose, can’t see, and have no say over.

And I want to be careful here, because it’s easy to let this slide into another round of JavaScript-bashing, and that’s not what I mean. Every ecosystem has this. JS and TypeScript just sit at one far end of it, where there’s a package for absolutely everything. Which is good, in general, you’re not rebuilding the wheel every other week, the way you are in some other places. But it’s also how you end up with a node_modules you couldn’t fully explain if someone put a gun to your head.

At the other extreme, there’s Microsoft and .NET, where the instinct runs so hard the opposite way that it tips into Not Invented Here Syndrome. Neither end is the “right” one. They’re both defaults people drift into without ever making a call.

So for me, it’s not about reaching zero dependencies. But having dependencies that we cautiously agreed upon.

Which takes me to the part that, in my experience, almost nobody does. You can’t make a call on what you can’t see, and if you don’t even have the basic knowledge (e.g. some list) of what you depend on, then every conversation about supply chain risk is a bit of theatre.

Dependency Inventory

In most of the environments, there are tools to generate the Software Bill of Materials - the inventory of your dependency tree. In some, they’re even built in. It’s easy to dunk on NPM, but it’d be better to do due diligence before doing so. Not many people seem to know this, but recent versions of npm ship an npm sbom. So the tooling exists, even in NPM. That isn’t the problem.

The problem is that most organisations have never generated one in their life. No SBOM, no inventory, nothing written down anywhere. So the day the next Log4Shell lands, and there will be a next one, they can’t answer the very first question anyone will ask them: do we run this, and if so, where?

On the other hand, tools often don't help here, even those built to do so. NPM audit mostly does the opposite. I honestly can’t remember the last time I installed something, and the audit didn’t immediately tell me to bump a stack of packages. Most of it is false positives, with no real attempt to say how dangerous any of it is. And that lands you in the oldest trap going: if it’s always red, you stop looking at red. So the one signal that was supposed to make you stop and decide ends up training everyone to decide nothing.

Bus factor and rug pulls

There’s a related thing I can’t quite leave alone, so let me wander into it. I think a lot of teams spend their energy on the symptom and never once look at the source.

Watch what happens in the .NET world whenever a popular package changes the deal. Fluent Assertions went commercial. Moq shipped a thing that quietly hashed your git email and phoned it home. MassTransit and AutoMapper announced commercial licenses within the same stretch. And nearly every time, the reaction across .NET shops is identical. It’s a mixture of

• let’s rip it out and write their own,

• search for a free alternative

• Cry to Microsoft to buy the lib or provide a replacement,

Essentially: a throw-the-baby-out-with-the-bathwater strategy.

And for me, that’s solving the wrong thing entirely. The source isn’t that the package started charging money, or pulled a rug. The source is that we took on a critical dependency without ever admitting to ourselves that it was critical, and never once thought about what we’d do if the terms changed. We didn’t consider the bus factor, and we didn’t do due diligence to ensure the work on it was sustainable and could continue. Pulling it out and hand-rolling a replacement fixes none of that. It just resets the same trap, this time with only the code we maintain.

The IdentityServer episode was the clearest version of it I’ve seen. People were upset that they had to pay suddenly. Then, in the next sentence, calling it a critical security component. Then, in the sentence after that, asking what the free alternatives were. A critical security component that you want for free and are ready to swap out overnight is, to my mind, asking for a security incident.

And there’s a bit of maths that quietly settles most of these arguments, if anyone bothers to do it. Take what the licence costs you per year. Then take into account what it would cost to have an engineer build and maintain your own version. Put the two side by side.

Almost every time, paying the maintainer comes out cheaper, and on top of that, you’ve lowered the bus factor on something you already lean on, which is its own kind of supply chain security. “We’d write it ourselves, but then we’d have to maintain it” is true. I just read it as the argument for paying the person who already does, not against it. If you depend on something, its survival is your problem too. That’s part of owning the decision.

I know this case too well.

LLM as a fork

Getting back to Mitchell’s thought. The part I find most interesting is because of the moment we’re in. I keep hearing that LLMs change all of this, that writing your own small things is suddenly trivial, so the whole dependency question softens. I don’t buy it. It’s never that easy. Writing the small thing was never the hard part anyway. Owning it, understanding it, maintaining it, being the one on the hook when it breaks at 2 am, that’s the hard part, and no model takes that off your plate.

I don’t see how LLMs can change the cost of owning code. They can (maybe) change the cost of producing it. That doesn’t fix the “install without deciding”. The old move was install and move on. The new move is “vibe it” and move on. Same missing decision, new flavour. The same lack of responsibility and ownership.

This trend isn’t new. It’s a classic Shadow IT. If you haven’t been around long enough to run into the term, Shadow IT refers to the tools and systems people build or adopt within a company without going through whoever is officially meant to approve them. The spreadsheet that quietly runs a whole department. The little script someone wrote on a Friday that half the team now depends on. The integration nobody in the platform group has ever heard of. It has always existed because people route around slow governance to get their job done, and most of the time, nobody notices until it breaks.

With LLMs, it’s more tempting than it has ever been. Someone in sales promises a customer a feature the team supposedly needs. The team has no time, so they cobble a tool together from the API and ship it. It doesn’t work. The customer says they’re not paying for this. It escalates. The thing was unowned from the moment it was conceived; nobody decided to take it on, it just appeared, and the blame game is starting.

And here’s where I think it all settles, because the corporate steamroller flattens everything in the end. Companies will dictate the allowed list, the way they always have. The cautious majority will stick to what’s known and popular: React, TypeScript, Python, Spring Boot. That’s what they did last time, and the time before. And the people who want to move faster will do it off the books, with an LLM, as Shadow IT. The declarative, standards-based frameworks that hide their complexity will do well in that world, because that style suits how these tools work, but it’s the same ceiling as before. You bet on React. You don’t own it. The small stuff you can hold; the big stuff stays as bet.

What to do then?

We cannot fix the entire software industry, but we can fix how our own engineering teams operate. Instead of waiting for automated audits to scream at us, or ripping out packages in an emotional panic, I suggest a simple, regular exercise for your organisation.

Sit down and explicitly define your dependency posture:

1. Inventory: List the dependencies you use (even without peer dependencies). Use tools npm-sbom to actually see what you are pulling in.

2. Criticality: Identify which of these packages are absolutely critical to your system.

3. Lifecycle: Define a clear strategy for upgrading and versioning them. Are you updating just for the sake of it, or are you looking for specific commits like Mitchell suggests?

4. The Bus Factor: Ask yourself: what happens if the author of a critical package gets hit by a bus, burns out, or the tool becomes paid?

5. Mitigation: Decide on a concrete backup plan for that exact scenario. Do you fork it? Do you pay the license fee? Maybe pay earlier for support or help in another way to maintain it.

6. Response Time: Estimate how quickly you can upgrade and deploy the application if a major security breach occurs in a dependency. Also, if the strategy is to use replacement, then how fast will you be able to replace this dependency?

Building reliable software requires intent. We don't have to write everything from scratch, but we must be precise about what we bring into our software. Architecture is not just about writing code; it is about choosing which liabilities you are willing to own.

Cheers,

Oskar

p.s.2. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

How soon is now? In PostgreSQL, it’s not always as soon as you’d think. I learned that the hard way recently, so you don’t have to.

It took me hours and wasn’t easy to reproduce, even though the fix is one line. I found it in a Cybertec post, as I quite often do when I’m staring at something odd in PostgreSQL. I’m supposed to know my way around the database, but I missed it, which is another reason I want to write this down.

I was working on distributed locking in Emmett. When you scale a service horizontally, you can easily end up with two instances of the same message processor running at once. That’s bad. Both instances would pull the same events, both would write to the same projection storage, and we’d get duplicated side effects, overwritten state and broken checkpoints. So we need to guarantee that exactly one instance of each processor is active at any time. Emmett does that using two things working hand in glove: PostgreSQL advisory locks and a row in the emt_processors table. The row keeps the durable side of ownership: which instance currently holds the processor (processor_instance_id), when it last checked in (last_updated), and what state it’s in (status). I described the full design in Rebuilding Event-Driven Read Models in a safe and resilient way, so I won’t bore you with the whole picture here.

For this story, the part that matters is what happens when an instance crashes. The crashed processor’s connection is gone, so its advisory lock has already been released. A new instance can grab the advisory lock without resistance. But the row in emt_processors still says status = 'running' and still points to the previous owner, because the crash didn’t give anyone a chance to clean it up.

From the outside, we can’t tell whether the previous owner has crashed or is just between heartbeats. So we wait. If the row’s last_updated is older than a configurable timeout, the new instance is allowed to claim ownership anyway. Anyone quiet for that long is treated as gone. To make this graceful, the lock acquisition runs inside a retry policy. A fresh instance starting just after a crash doesn’t fail straight away; it retries until the timeout window expires.

The bug

The takeover decision lives in the upsert against emt_processors. In the real function, that upsert sits inside a Common Table Expression (CTE) alongside a pg_try_advisory_xact_lock call.

For the record: the snippets below skip that wrapping (and trim a couple of unused parameters) to keep the focus on the upsert, where the bug lives. The full version is in the source.

CREATE OR REPLACE FUNCTION emt_try_acquire_processor_lock(
    p_processor_id           TEXT,
    p_processor_instance_id  TEXT,
    p_lock_timeout_seconds   INT
)
RETURNS BOOLEAN
LANGUAGE plpgsql
AS $$
BEGIN
  INSERT INTO emt_processors (processor_id, processor_instance_id, status, last_updated)
  VALUES (p_processor_id, p_processor_instance_id, 'running', now())
  ON CONFLICT (processor_id) DO UPDATE
  SET processor_instance_id = p_processor_instance_id,
      status                = 'running',
      last_updated          = now()
  WHERE   
     -- same instance reconnecting
     emt_processors.processor_instance_id = p_processor_instance_id                      
     -- previous owner stopped cleanly
     OR emt_processors.status = 'stopped'     
     -- previous owner timed out    
     OR emt_processors.last_updated
        < now() - (p_lock_timeout_seconds || ' seconds')::interval;
  RETURN FOUND;
END;
$$;

The last branch is the takeover. It reads naturally: if the previous owner hasn’t checked in for longer than the timeout, the new instance can replace them. All tests were green. Stop me if you think you’ve heard this one before. The problem surfaced through user feedback (thanks, Martin!), and it took me a long time to reproduce; none of the existing tests covered the scenario that triggered it. Once I had a new end-to-end test that pinpointed the symptom, the rest was the usual, long, boring, debugging loop.

To see why, open psql and run this:

BEGIN;
SELECT now() AS tx_now, clock_timestamp() AS wall_clock;
SELECT pg_sleep(2);
SELECT now() AS tx_now, clock_timestamp() AS wall_clock;
COMMIT;

You’ll get something like:

            tx_now             |          wall_clock
-------------------------------+-------------------------------
 2026-05-25 10:00:00.123456+00 | 2026-05-25 10:00:00.124012+00

            tx_now             |          wall_clock
-------------------------------+-------------------------------
 2026-05-25 10:00:00.123456+00 | 2026-05-25 10:00:02.131845+00

The first column is the same in both rows. The second one is two seconds apart. As it turns out, now() is a synonym for transaction_timestamp(): it returns the time the transaction began, and keeps returning that value for every statement inside the same transaction. A light that never goes out, in other words. clock_timestamp() reads the wall clock each time it’s called, so it advances as time does. Cybertec wrote a good walkthrough of the whole family of timestamp functions if you want the full picture.

What difference does it make? For a column like last_updated, the constancy of now() is usually what you want: every row touched in the same transaction shares a single timestamp, which keeps audit logs and write batches coherent. For asking “has enough time passed?” inside the same transaction, the same constancy works against us.

Now back to the retry. The consumer that calls tryAcquire looks roughly like this:

pool.withTransaction((tx) =>
  asyncRetry(
    () => tryAcquireProcessorLock(tx.execute, options),
    { retries: 10, minTimeout: 200, maxTimeout: 1000 },
  ),
);

pool.withTransaction opens a database transaction and passes its executor to the body. asyncRetry then repeatedly calls the stored procedure on that same executor, with a backoff between attempts. So even though the retries are spread out in real time, every call runs inside the same database transaction:

withTransaction        (transaction starts at T)
  └── asyncRetry
       ├── call lock function    → now() = T
       ├── call lock function    → now() = T   (200 ms later)
       ├── call lock function    → now() = T   (400 ms later)
       └── ...

last_updated < now() - timeout evaluates the same way every iteration. The predicate is effectively constant for the lifetime of that transaction. From the database’s perspective, no time was passing between attempts, even though the retries were spread across real seconds. (of course, the valid question is whether retries should happen inside a transaction, but let’s say that this is out of scope of today’s article, deal?).

So what was the fix? Change the time source inside the function. PL/pgSQL lets you declare local variables, so I added one at the top, initialised from clock_timestamp(), and used it everywhere the function previously called now():

CREATE OR REPLACE FUNCTION emt_try_acquire_processor_lock(
    p_processor_id           TEXT,
    p_processor_instance_id  TEXT,
    p_lock_timeout_seconds   INT
)
RETURNS BOOLEAN
LANGUAGE plpgsql
AS $$
DECLARE
  v_current_time TIMESTAMPTZ := clock_timestamp();
BEGIN
  INSERT INTO emt_processors (processor_id, processor_instance_id, status, last_updated)
  VALUES (p_processor_id, p_processor_instance_id, 'running', v_current_time)
  ON CONFLICT (processor_id) DO UPDATE
  SET processor_instance_id = p_processor_instance_id,
      status                = 'running',
      last_updated          = v_current_time
  WHERE emt_processors.processor_instance_id = p_processor_instance_id
     OR emt_processors.status = 'stopped'
     OR emt_processors.last_updated
        < v_current_time - (p_lock_timeout_seconds || ' seconds')::interval;
  RETURN FOUND;
END;
$$;

clock_timestamp() ignores transaction boundaries. Every call to the stored procedure reads the wall clock fresh, so each retry sees a slightly later value than the last. After enough retries inside the wrapping transaction, the takeover predicate flips, and the new instance wins.

The function uses the timestamp in two places: when setting last_updated on the new owner, and when comparing the previous owner’s last_updated to the timeout. I switched both to v_current_time, so the write and the check read from the same wall clock. Mixing clock_timestamp() on one side and now() on the other would leave a subtler version of the same bug.

A cleaner option for the future is to move the retry one layer up, so each attempt opens its own transaction. That would remove the trap entirely and let the function go back to plain now(). For now, the local variable does the job.

Why my tests didn’t catch it

Now to the testing side. I had a careful suite of integration tests for the stored procedure. Two instances racing for the lock. The same instance reconnects after a crash. Takeover after a custom timeout. They all passed, and they could not have caught this bug. Here’s the shape of a typical one:

await pool.withTransaction((connection) =>
  lock.tryAcquire({ execute: connection.execute }),
);

That’s the shape: set up some state, call tryAcquire once, check the result. A single call to the stored procedure works fine with now() in the WHERE clause. There is only one timestamp involved per call, and the predicate evaluates correctly against it. The bug only shows up when several calls share one transaction, which happens when the consumer’s withTransaction wraps a retry policy. The stored-procedure tests never put those two together.

The end-to-end consumer tests do go through the consumer’s withTransaction wrapper, but they covered the happy paths: clean start, clean stop, two consumers competing, and an instance reclaiming its own stale lock. None of them combined the three conditions that together expose the bug:

1. a previous owner whose row still says status = 'running' (a crash, not a graceful stop),

2. a new instance arriving with a different instance ID,

3. a retry acquisition policy with a timeout short enough for the retries to outlast it inside the test’s deadline.

Any one of those missing and the takeover predicate either succeeded on the first attempt (so the retry never fired), or the test finished before the retry’s failure to make progress was visible.

So why did this slip through both layers? The stored-procedure tests never combined a retry policy with the stale-row state, so they never produced multiple calls inside one transaction. The end-to-end tests did exercise the retry path, but none of them happened to combine all three conditions above at once. Both layers had blind spots, and the bug lived exactly where they overlapped.

The Pull Request that fixes the bug also adds a new family of tests that mount tryAcquire under the same transactional wrapper the real consumer uses, with the crash + new-instance + retry combination wired up on purpose. That’s the kind of test I should have had from the start.

What I’m taking away

Two things about now():

• now() is the right tool when you want every row touched in the same transaction to share a timestamp. created_at, last_updated, audit columns. That stability is a feature.

• now() is the wrong tool when you want to ask “has time moved on?” from inside a transaction. Use clock_timestamp() when you genuinely mean the wall clock.

And one harder thing about tests. Inner tests for stored procedures give me a tight feedback loop and pinpoint failures, but only inside the scaffolding I build for them. End-to-end tests run the real wiring, but I can’t enumerate every combination of timeouts, instance IDs and crash states without the suite collapsing under its own weight. The combination where this bug lived wasn’t reachable from either side by accident; it needed a deliberate setup.

I don’t have a clean rule for where to draw that line, and honestly, I don’t think there is one. My takeaway is to look at the seam: the spot where the inner test invokes the code differently from how the production caller invokes it. Here, it was a single-call test against a retry loop sharing one transaction. Wherever that gap sits, write a test there. Not at the unit level, not at the full end-to-end level, but in a setup that mirrors how the real caller actually drives the code and exercises the path you care about.

TLDR

now() returns the start of the current transaction, not the current moment. Inside a transaction it doesn’t change between statements. If you wrap a retry loop around a function that uses now() in a WHERE clause, and the retry loop runs inside one transaction, the predicate is frozen and the retries do nothing. Use clock_timestamp() when you mean “right now”. And pay extra attention to the seam between inner and end-to-end tests, because that’s where mismatches between how tests drive the code and how production drives it tend to hide.

The fix and the new tests live in Emmett Pull Request #339.

Uff. That bug was nasty.

Read also:

• Rebuilding Event-Driven Read Models in a safe and resilient way, with the locking design this bug lives inside,

• Distributed Locking: A Practical Guide,

• Consumers, projectors, reactors and all that messaging jazz in Emmett,

• Checkpointing the message processing,

• Cybertec: PostgreSQL now() vs now()::timestamp vs clock_timestamp(),

• Is keeping dates in UTC really the best solution?.

Or my other articles about PostgreSQL:

• Postgres Superpowers in Practice,

• PostgreSQL partitioning, logical replication and other Q&A

• PostgreSQL JSONB - Powerful Storage for Semi-Structured Data

• Push-based Outbox Pattern with Postgres Logical Replication

• How to get all messages through Postgres logical replication

• The Write-Ahead Log: The underrated Reliability Foundation for Databases and Distributed systems

• How Postgres sequences issues can impact your messaging guarantees

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

For instance, I believe there’s a strong synergy between C4 Model, Context Maps, and EventStorming. They all allow us to look at the system from a different angle and act as simulations, providing different feedback on whether our model will fly.

Look below for the diagram I prepared for my upcoming workshop.

It’s a C4 diagram showing containers in a hospitality system (so-called Property Management, where property means hotel).

A container gives quick feedback on how many pieces I’ll potentially need to manage and deploy, and it also shows me boundaries.

Context Maps help in understanding how they relate to each other, which module or team has bigger leverage, and can force more on others. It can also show me the information flow and simulate which module will expose the API.

I can then take some business process and have a look at how the message flow will look. This can happen as an early simulation or after EventStorming sessions.

Then C4 allow me to also look inside the container, show components if I need to understand more details. I can group containers if my bounded context has more of them.

How to start with it? Let me show you an example.

The starting point could be understanding the system's overall view; integrations with external systems usually introduce the most complexity. C4 System Diagram works well for that.

We see actors representing our system users and interactions with external systems. We can decide what integration is out of scope for now, and may come as the second step, but not for now. We can also show the current state as is, then in the second diagram show our vision of what we want it to be. Be creative, this is not a relational database, we don’t need to normalise it.

Then we may decide to dive into what we need to build to facilitate that, make a bet based on our current knowledge, zoom in and check the C4 Container view.

As mentioned, this can already be used both for static documentation and for discussions during a workshop, simulating the potential complexity and boundaries. Important to note: until it’s settled, it’s more than fine to show different versions, discuss them, and add notes and questions.

We may have concluded that C4 is fine, but it doesn’t capture all the important reasons we modelled it this way. Or during the workshop, it may not be clear how to discuss precisely how to break it down. Context maps can help in that.

They can give us information on which module is generic (Open Host), which module (or team) is more important (upstream), and which module or team has less leverage (Downstream). Which provides api or data (Supplier) and which one is using it (Consumer).

We could have a dedicated model only focusing, on that, but we could also capture it in the same one. It could look as follows:

We can even use colours, e.g., upstream as green and downstream as red. This can already give us feedback, e.g., that Cashiering is an important module for financial processing and that it is downstream of multiple modules. Maybe we can change it and redefine boundaries, or maybe live with it, but take some corrective actions.

We can also put more data and use a specific example of a process and have a look at the specific process, as shown in the original diagram:

This can also pinpoint some issues, e.g., a generic module, which is more likely to expose commands (see more on why in my article on Passive-Aggressive Events). We can see if we have all the needed ACLs defined, or which module acts as a coordinator.

Of course, keeping it all on the same diagram is not perfect; it’d be great if we had a tool that would enable zoom-in/zoom-out, link and enable/disable some of the details and notation. Then we can even better play with what we have. Maybe I should vibe such?

Nevertheless, I encourage you to try different techniques, experiment, and combine them. For instance, Example Mapping would, imho, play great here as an extension. I know that some use Wardley Maps or Domain Storytelling.

Start small, use existing techniques and tools, and try to see how you could benefit from mixing them. Show it to your friends and try to collaborate and have fun together.

I even learned from Richard Gross that this has an even name: Model Storming. So, crunching your design with different modelling techniques and coming up with some useful variations.

I have even more mashups like that. Tell me if you’d like me to expand on it or this part in the follow-up articles!

Also, if you did such experiments, please share them with me and others. Happy to see what you came up with.

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

I was toying with LLM-based domain research. It reminded me of the common mistake we make when we try to practice DDD: overreliance on what domain experts are telling us.

I wanted to remind myself of the domain I used to know: hospitality management, to adjust my upcoming workshop. I selected LLM (Claude Opus) as a sparing partner. And boy, I got a loooot of details. I was swamped by them. The LLM modelled everything: marketing consent, loyalty timing, inventory management, revenue posting, regulatory submissions, data retention policies, all at the same level of detail, which buried the core checkout flow under noise, the thing I asked about.

That’s not much different from initial work with domain experts. When we’re starting discovery, people tend to start by explaining everything they do and feel is important. Quite often, that means you’ll get more domain-expert pet peeves in the surroundings than in the process descriptions.

You also get a lot of jargon, words that sound familiar but mean something different.

That’s also what I got from LLM, I asked it to review reference sources like:

• popular tools documentation,

• open specifications from the hospitality organisations,

• some laws like GDPR related stuff,

I researched how different tools handle the guest checkout process. I knew it, and implemented it in past; it’s a surprisingly complex process, as you need to:

• verify if the stay is fully paid and close the financial account,

• generate invoice

• mark guest’s stay as completed,

• schedule full room cleanup,

• mark in the inventory that the room will be available soon (unless the maid finds that it’s broken),

• cleanup GDPR-related data that’s not needed to keep,

• adjust loyalty points and update CRM,

• etc.

I remembered how it works, but not all the details, and I would like to double-check whether anything in the industry has changed.

I wanted to get the overall vision first, then gradually dive deeper. As mentioned, I was overwhelmed with naming like:

• Folio,

• Drain Pending postings,

• Property,

• Account Receivables,

etc.

Don’t get me wrong, those were valid names in these domains. Sounds plausible if you’ve never worked in hospitality. Sounds plausible if you have, too. So what’s wrong with it?

They’re valid terms, as people actually use them, but weird, as they don’t tell “how the process works” but “how people do stuff,” and that’s not the same thing.

For instance, Property is a hotel, kind of makes sense. But Folio?

This name comes from the Oracle Opera. Yes, Oracle has a system in this domain, a dominating one. At some point, authors decided to name things this way, and now 30 years later, that vocabulary is baked into how thousands of hoteliers talk about their work. “Drain pending postings” is a phrase real cashiers say.

“Settle the folio” is a thing real cashiers do.

The problem: none of it explains how our system should work. It only tells what current systems do.

Also, when I asked about the business rules and policies, I got stuff like:

The system immediately posts room and tax charges for day-use reservations upon opening the Billing screen.

Why are transactions such as night stay charges or taxes added when we open the billing screen? That sounds counterintuitive, but maybe it was a fair tradeoff 30 or 20 years ago for an on-premises system installed in the specific hotel (ekhm “property|). Yes, Opera had (and maybe still had) consultants, similar to SAP. They’d go to the hotel, go to the back office, log in to the server and hack stored procedures in the Oracle database to fine-tune Opera behaviour.

Also, is there really a “Draining Pending Postings” option before doing checkout nowadays? Maybe it is, but in modern systems, we shouldn’t manually check all bills, etc., and explicitly pull them from integrated payment solutions. Nowadays, all the accommodation charges should already be recorded on the bill. The financial module continuously collects charges from payment gateways throughout the stay. There’s no separate moment of “draining”. The LLM invented a coordination command to match a phrase (“drain the interfaces”) that real cashiers use as shorthand for “let me check that nothing’s outstanding.”

As I asked, LLMs researched systems in this space: OPERA, Mews, Apaleo, and Cloudbeds. Each has its own vocabulary and mechanics, and the training data heavily favours OPERA because its documentation is everywhere. When I pushed back on terminology, the model would just swap in different OPERA jargon instead of actually thinking about what’s modern. The blending happened invisibly, mixing vocabulary from one system with mechanics from another, all delivered with equal confidence.

To be fair, the same happens quite often when talking to domain experts. They explain how it works now and what people do. Quite often, they bring us solutions instead of problems. Usually, solutions are based on their experience and how they see the updated version. This can be fine as a brain dump, but it’s not enough to translate it directly into the software design.

In Domain-Driven Design, finding and understanding the ubiquitous language is considered the most important aspect. Yet, Ubiquitous Language is not the source of truth. It’s the way to keep our heads from exploding due to the constant split-brain situation. It’s a tool to reduce cognitive load and the need for additional translation. That’s why we separate domain contexts and bind them to specific departments, people, and the language they use.

It’s fine to start with the current state of the art. Understanding how people do their job. We need to understand, though, that what we get is a mixture of habits (both good and bad), tribal knowledge, jargon, etc. If you ask different tribes, each will tell you something different.

Domain language is a cognitive tool, not gospel. LLMs compound this problem by reiterating competitor vocabulary without understanding the reasoning behind those systems, and they tend to align with whatever the prompter already believes.

Domain experts also struggle to define what they want and how it should work. That’s also why they hire us. It’s our job to help them and to transfer those sometimes contradicting visions into working software. That’s what we’re learning and what we do when modelling and step-by-step shaping the working software. That’s our work as engineers. We should work together to have a proper outcome. Collaborate.

And I’m not making the bold statement here that we’re smarter or we know better. We’re not; we have different expertise and different roles in the software development process.

Contrary to what many people believe in the DDD community, in my opinion, we’re not here to become domain experts; we’re here to build software.

The model we built doesn’t have to reflect the whole universe; it’s a way to take part of the business, understand, and automate it in the form of a software system. A software system is a tool for the business make more money. So software needs to be useful in a certain, defined way. Not all possible ways.

We need to learn how to communicate with business people, for instance:

• Don’t use jargon or acronyms or assume someone should know something.

• Understand that what someone wants to tell us might not be what they hear.

• Do not take others’ behaviour personally.

• Be assertive, critical and sceptical. Also, to our own judgments.

• Be curious about the business domain. Don’t assume too much.

• Don’t use “business won’t let me” as an easy excuse.

For some people, that’s too much. That’s probably why they try to ask LLM instead of Domain Experts, hoping that we won’t need to learn that, and we’ll get a solution for free.

We’re sometimes annoyed by being pushed hard by business people, I get that, but if we want to build something useful, that’s also where LLM will fail, as they will just agree eventually to what we believe. And that’ll probably be even worse than the skewed reality shown by the domain expert, since this will be Artificial Reality.

I keep hearing blank statements that “LLMs are great for research”. Blank because they usually are not followed up with what the author means by “great”. LLMs have many inherent limitations here. We’re anthropomorphising them too much. They don’t reason; they’re statistical machines. We should constantly remind ourselves of that.

I think people who claim that “LLMs are great at research” are just conflating their own skills and projecting them onto LLMs.

Hence, in my opinion, that’s why we’re getting those hot takes. Might be that they just have the skills to drill down, organise research, evaluate it, and model system design.

And I think this narrative is dangerous.

Because I, too, could sit down and say that I iteratively arrived at a solution thanks to LLMs. Write this article in a much different narrative, praising LLMs. I could say that if someone couldn’t get the same result, then that’s a skill issue.

But that wouldn’t be true, because someone without my experience in the domain and in modelling probably wouldn’t pick up on it.

Furthermore, it’s not necessarily true that I did it well; that would just be my perspective.

In practice, in this context, “LLM does it well” means “LLM does it the way I wanted it to.” And if the outcome is right, it is more likely that the “operator” had the necessary skill and used an LLM as a tool to speed it up.

The research I did looked at what our competition does and how they name things, but without internal knowledge of why those systems got to where they are. If our goal is to provide additional value to users, then just doing a blind copy won’t take us far. Instead of doing Lift and Shift, building something that has the same features but does better, we’ll get Lift and Shift with silent f.

Yes, LLMS can gather and compile multiple sources into a summary. That’s actually impressive and can spare us a lot of time. They’re also taught in the public documents, so some of their knowledge is built in. They look impressive on the surface, but not so if you look deeper. In the mentioned research, I got disconnected pieces of information, the techniques were randomly assembled with muddled technical and business language, and the whole thing didn’t tell a coherent story. Because of the randomness, you never know where the knowledge came from, what was omitted and what was skewed. With domain experts, you can at least understand the origin of their biases.

LLM also has limitations. The biggest is the context size, and being a yes-man. If we ask for the big picture, usually, we’ll end up with a swamp of information instead. If we don’t know how to sort this knowledge, what we’re looking for, and do a proper drill-down, then we won’t be able to untangle it. If we don’t have domain experts and don’t know the domain, how will we challenge issues like the ones I gave above?

I also keep hearing that LLMs are really useful in modelling. Sure, I tried that. I also asked LLM to try to model, and well, even after numerous iterations and using modelling tools, the results were mediocre. Mostly cliches and bad modelling practices.

The output muddied what we had before. The context was lost, and the process was oversimplified. When asked to apply specific tools like the C4 model, EventStorming, and Context Maps, it was forcing DDD patterns instead of describing actual processes. Even with precise instructions, it was building models with broken notation where rules floated outside the command-event chains, leaning on clichéd solutions, losing context between iterations. The domain was also presented as a big ball of mud. Concepts from room reservation bleed into the cashiering module. It was a recurring pattern across different versions of the same mistakes.

If you give too much context, they’ll mix and blend multiple conflicting vocabularies from different operational tribes trying to satisfy you. If you give it too little, it will come with simplistic hallucinations.

Is it hopeless then? Not at all, we can get help from LLM, but when we use it as a tool, not a replacement. We should not outsource thinking.

LLMs are useful for grasping the big picture and identifying known unknowns that we may miss as we get into the domain. We can learn about the language specifics, as mentioned in the Folio, Postings, Account Receivables, etc. LLMs can help us drill down into interesting aspects. We can get a brief understanding of domains we don’t know at all. But then we need to dive deeper and collaborate with our domain experts, real stakeholders. We need to understand what we want to build, organise our findings and focus on a certain context.

If we want to model our system, then LLMs can help us do boring work. It can organise our findings and create rapid text-based transitions with a defined format (including modelling practices). This can work. Yet it’s still our job to do domain discovery, refine the knowledge, evaluate and test it. We still need to focus on the feedback loop with real world.

We won’t hide from gathering modelling skills, we won’t skip the process of translating the domain into technical design, and we’ll still need to drive how and when to drill down. Those are engineering skills that were needed and will still be needed. And that’s fine, as if they wouldn’t, why would someone want to hire us?

I see that (for unknown reasons) collaboration is not discussed anymore. Working collaboratively with fellow humans has become something people dread rather than embrace. Maybe some people hope that they won’t need to talk to “domain experts” and “fight them”, because they “have everything here in LLM”.

And yes, they have all.

All the same issues they would have with domain experts.

As much as we shouldn’t trust domain experts blindly, but work with them, we shouldn’t blindly trust LLMs. We shouldn’t drop our engineering and design skills and outsource them.

If we want our software products to be better than the competition, simply blending their experience isn’t enough. We should focus on finding our special sauce, and I don’t see any other way to make it right than to work together and collaborate. We can use LLMs to help us do it faster, but as tools, not as solutions or replacements.

Check also:

• Bring me problems, not solutions!,

• A few words on communication,

• How to design software architecture pragmatically,

• Business Won’t Let Me and other lies we tell to ourselves

• Vibing, Harness and OODA loop,

• Interactive Rubber Ducking with GenAI,

• The End of Coding? Wrong Question,

• Requiem for a 10x Engineer Dream,

• Tech Debt doesn’t exist, but trade-offs do

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

Borys had the best dribbling, Cybor bent it like Beckham, there were a few little kids who could pull off 200 keepie-uppies. My class 5B always gave 5A a hiding after a tough match. For a while now, we’d only been playing by the new rules, no back-passing to the keeper. I didn’t do hundreds of tricks, I’d cap my keepie-upiess at 20–30. It didn’t entertain me. I didn’t need excessive ornaments. I thought that, at the end of the day, fundamentals are what matter most. Kicking the ball where you want it to go. What’s the difference if it slams into the top corner, curves in like a Brazilian wonder-goal, or barely rolls over the line? What matters is that it ends up in the net. That’s what counts in the end. Art for art’s sake? Not for me.

In FIFA, I never picked Brazil, not even in ‘98 did I play as France. That would’ve been too easy. Better to go against the grain. What’s the challenge in playing a guitar that’s already in tune? Better to lose to better players than to win against worse ones. Better to be a benchwarmer at Manchester United than the captain at Miedź Legnica.

I always read the instructions. You’ve got to prepare. You’ve got to assemble it the right way.

I don’t buy the first thing I see; I check price-comparison sites, read forums, and set up auction snipers on auction platforms.

Yesterday, after two years, I finally fixed the broken blinds.

• “What’s your biggest weakness?”

• “I’m a perfectionist.”

Sometimes I call myself the master of unfinished ideas. Like most of us, I dream of having my own idea, my own product, my own startup. I regularly throw myself at the next brilliant (in intent) idea, and I regularly fail to finish it. Every attempt ends the same way - “the same place, different girlfriend.”

Of course, every project of mine has to have solid foundations. Architecture first, then the framework. Everything has to be top-notch, the latest tech, best practices, and design patterns. I’ve finally got a free hand, after all, no nagging customer to worry about. Honestly, our industry is often a lot like Polish State Railways: everything would run brilliantly if it weren’t for the customers. We constantly laugh at customers: what do they know, idiots, don’t teach me how to do my job. Who cares about colours, a button shifted three pixels to the left, the wrong shade? What matters is that it’s SOLID. The most important thing is good architecture and beautifully functioning internals. Our projects are like icebergs, massive underneath, but somehow not much visible to show for it.

Last week, a book landed in my hands, well, an ebook: “Just Fucking Ship” by Amy Hoy. It hit my mood of brooding over what I’m doing wrong perfectly. Why is Borys the schoolyard footballer everyone remembers, and not me? Why does my dad say I’m all gas, no follow-through, that I burn hot and burn out fast? Why are Xamarin and SignalR thriving, while my unfinished framework with similar ambitions sits at the back of my hard drive, untouched for years? You’d think I’m doing everything right. Reason. Solid foundations. Knowledge. Experience. Mr. Prim and Proper.

Maybe it’s exactly because I make the same mistakes everyone (well, almost everyone) makes:

• I focus on chasing the rabbit, not catching it. I start a project from the framework, a website from the layout, and before any of that, from picking the domain name. So I burn through my peak motivation on bullshit that’s least important from a product perspective.

• I don’t take notes, I don’t write down plans, “why would I? I don’t need to, I’m exercising my memory.”

• I don’t focus on the goal, I don’t set deadlines.

• I don’t follow the “start small, grow big” principle.

Amy’s book isn’t groundbreaking; it doesn’t hand you a golden recipe — it doesn’t even aspire to. What it does give you is a breath of fresh air in your head, a few mental hatches popping open, and a handful of valuable, concrete, real-life advice. The most important: set yourself a goal, a deadline, and just fucking do it. First, the goal and the time, then the scope and the methods. Of course, we’ll all say, “yeah, but it doesn’t always work that way”. But look at it differently: when we want to invite friends over for dinner, do we plan, “well, maybe in two weeks instead, because I won’t have time to make a three-course meal, a cake, and homemade liqueur”? We don’t do that. We adjust the menu or buy something pre-made. Why can’t we apply the same approach to our projects?

Amy notices that we focus too much on the project as a whole. We don’t set ourselves small goals — the kind that, when achieved, would give us satisfaction and motivation to keep going. We get so fixated on the grand vision that we end up abandoning the idea without doing anything at all. We’re afraid it isn’t cool enough, it isn’t innovative enough. Why don’t we let other people judge that? We stay frozen at the starting line instead of just running and seeing how far we get. We forget that to get to the upper floor, you take the stairs one step at a time. Standing on the ground floor, wondering if we can make it up, gets us nowhere. We can even take the elevator, as long as we just fucking do it.

After this read, I’m not going to turn my life upside down, I’m not suddenly going to become a different person. I’m just going to try, this time, to finally fucking deliver on my plans.

A week ago, at almost 30 years old, I picked up my first football medal. There’s still hope.

Cheers!

Oskar

Hey, have a look at what I made during the weekend. I had some time, grabbed a beer, turned on the computer and tried to code this feature. If I could do so much during the weekend, how much could you and your team do with it in 2 weeks?

It’s almost a 1:1 quote of what I heard from the startup founder I worked with over 10 years ago. I’m sure that you’ve heard similar phrases from people you worked with. We all know the annoying type of person who doesn’t code anymore but thinks, “I still got it!”. Then they threw a piece of stuff at you to “just fine-tune it a bit and do final touches”. Then they’re the first ones to ask “Why so long?“.

Nowadays, the Internet is full of such people. They shout about what they did with Claude or how much progress LLM tools have made. Some even predict the end of coding. I already wrote that this is wrong perspective. I won’t repeat that, but I want to say that…

Vibing isn’t new and isn’t always an issue.

I’m saying that LLM tools are an appraisal for ignorance. The more ignorant we are of the topic we’re working with, the better we see the outcomes. And that, by itself, is not always bad, as there’s power in ignorance if we focus on getting it done with the simplest tools we have.

Still, this can be terrible if we fall in love too much with what we’ve vibed.

To understand why that “weekend beer” energy is both a superpower and a liability, we need to look at the OODA Loop.

Disclaimer, it’s not a competition for Ralph Wiggum Loop. It’s much older and generic.

Military strategist John Boyd developed the OODA loop (Observe, Orient, Decide, Act) for fighter pilots. In a dogfight, the pilot who cycles through these four stages the fastest and most accurately survives.

In software, the “dogfight” is the gap between your intent and the production-ready feature.

OODA loop is built from four steps:

1. Observe - This is the intake of raw, unfiltered information. In our world, this means looking at the state of the system.

2. Orient - This is the most critical and difficult stage. It’s where you filter your observations through your experience, culture, and technical knowledge.

3. Decide - Based on your orientation, you formulate a hypothesis.

4. Act - You execute.

Getting back to my favourite founder and LLM-based tools.

The reason founder could build a PoC in a weekend while the team needed more than two weeks is that he bypassed the Observe and Orient phases. He went straight from a vague idea to Act.

If we skip or brush past the observation step, it feels like lightning speed. If the fancy UI grid is there and it does something we wanted, we move on. We’ve outsourced Orientation to our own ego. It’s too easy to assume that because we wrote it, it works.

Observation is the intake of raw data. In a professional environment, our eyes aren’t enough. We need a Harness. If we don’t have automations, tests, integration tests, and pristine traces, we aren’t observing the system; we’re just looking at it. If the inputs are messy, our observation is clouded.

But real engineering, the kind that takes those “two weeks”, is about closing the loop properly. That’s also where we need different perspectives and knowledge sharing.

Orientation is where you process those observations. This is the part where LLMs make us feel smarter than we are. If we don’t understand how a database handles concurrent connections, our “orientation” of a generated script will be shallow. We’ll see code that “looks” right, decide it’s fine, and act by deploying it.

The “I still got it” crowd loves the Decide and Act phases because that’s where the visible progress happens. LLM tools have made these phases nearly instantaneous. We can decide to build a feature and have the code for it in ten seconds.

The problem is that the faster we Act, the faster we need to Observe. If our “Act” phase takes seconds but our “Observe” phase requires a manual weekend of clicking around and drinking beer, our OODA loop is broken. We’re just generating a pile of stuff that we haven’t actually verified.

That’s why the team usually needs more than an imaginary “two weeks”. They are not “fine-tuning” the single-brilliant-dude masterpiece. They are building the infrastructure required to make the OODA loop sustainable.

And to make that possible, they need to run the full loop: Observe, Orient, Decide, Act. And do it multiple times. That takes time, but it’s required to assess the direction, automate what needs to be automated, and ensure they can iterate further and run this loop sustainably. That’s critical for delivering the outcome at the expected pace.

Of course, there’s a danger here, overfocusing on the Orient and Decide can lead to overengineering, building stuff we don’t need. That’s where ignorance can be blissful, especially when we connect it with humility. Being humble about what we don’t know and trying things the easiest way, then learning and making enhancements. Still, humility fails under deadline pressure. The harness doesn’t.

Let me give you…

The example

I’m adding proper Observability and Open Telemetry to Emmett right now. I spent some time working on it and instrumented the first component: Command Handling.

Of course, I had tests to prove it works, but I don’t trust them enough, and I wanted to try it on a real sample, since you never know until you run it. Even the best test suite won’t tell you all.

So I decided to plug it into the sample. See if it works, how ergonomic the API is and how it fits conventions in this area.

To do it, I decided to use Grafana stack and set it up with Docker Compose. So, stable, boring stack. Not going to lie, I vibed the config. Not that there are no docs, but I intentionally wanted to see the typical config people use.

If someone says LLM-based tools are great at proof of concepts, they don’t run the stuff they vibed. If I made the observation based on the initial config, then an oriented decision would be that it won’t work. Of course, then I did the typical back-and-forth, with the LLM tool doing some Linux command Voodoo to make it work. Once. Then, if you try to repeat it, you won’t know how to do it without doing Voodoo again.

Again, that’s not much different from the other stuff we do. I’m sure that you had multiple cases, when someone didn’t use Continuous Deployment tools, but clicked through Azure, AWS, GCP portal, deployed the stack, and then there was no trace on how to set it up again (e.g. to have a different environment for testing or demos for customers).

So, we need a harness, we need a leash to keep our process on track.

How to do the harness? My advice is to start simple. We may ask LLMs to give us shell scripts, and we may ask them to run them multiple times. We also need experience and knowledge of what we want to achieve and the tools we use. It’s fine not to remember all the YAML config to set up the Grafana stack, but it’s not fine not to understand why you even use it, how it relates, and how to set it up.

Still, our first loop can close on the first working solution, even a manually vibed one. But that’s not even a PoC. We need to automate them.

I asked LLM to take notes on what issues it had, and it solved them. Then, based on that, I asked to research how to code it in TypeScript. And to use tools I know, used in past, validating if there are no new more modern ones. For instance, I was a big fan of Gulp.js and Bullseye in the past, but they’re mostly dead. I wanted to have something in the same spirit, using native, maintained tooling.

I ended up with the following tools:

• execa for running shell scripts,

• native fetch for calling http endpoints,

• native Node.js test tools for checking if the stack works as expected.

Then I asked it to create the script to automate the shell Voodoo they did to make Grafana stack and Docker Compose work.

Essentially, it should:

1. Run Docker Compose script starting up services (Grafana, Prometheus, Loki, Tempo, PostgreSQL, etc.).

2. Wait for them to check when they’re ready (it usually takes some time).

3. Start the application and make a request.

4. Check if the predefined dashboard with Emmett metrics appears, and shows expected traces and metrics.

Initial diagnostic tools looked like that

async function fetchWithDiag(label: string, url: string, init?: RequestInit) {
  const res = await fetch(url, init);
  if (!res.ok) {
    const body = await res.text().catch(() => '(could not read body)');
    console.error(`\n  ✗ ${label} → HTTP ${res.status}\n  body: ${body}\n`);
  }
  return res;
}

async function diagnoseCollector() {
  const text = await fetch(URLS.otelCollectorMetrics)
    .then((r) => r.text())
    .catch(() => 'unreachable');
  const emmett = text
    .split('\n')
    .filter((l) => l.startsWith('emmett_') && !l.startsWith('#'))
    .slice(0, 5);
  console.log(
    emmett.length
      ? `\n  collector /metrics (emmett lines):\n  ${emmett.join('\n  ')}`
      : '\n  collector /metrics: no emmett_* lines found',
  );
}

async function diagnosePrometheus() {
  const json = await fetch(
    `${URLS.prometheus}/api/v1/label/__name__/values`,
  )
    .then((r) => r.json() as Promise<{ data: string[] }>)
    .catch(() => ({ data: [] as string[] }));
  const emmett = json.data.filter((n) => n.startsWith('emmett_'));
  console.log(
    emmett.length
      ? `\n  Prometheus emmett_* metrics: ${emmett.join(', ')}`
      : '\n  Prometheus: no emmett_* metrics found yet',
  );
}

async function diagnoseLoki() {
  const labels = await fetch(`${URLS.loki}/loki/api/v1/labels`)
    .then((r) => r.json() as Promise<{ data?: string[] }>)
    .catch(() => ({ data: [] as string[] }));
  console.log(`\n  Loki labels: ${(labels.data ?? []).join(', ') || '(none)'}`);
}

async function diagnoseDockerLogs(service: string, lines = 10) {
  const { stdout } = await execa('docker', [
    ...COMPOSE,
    'logs',
    '--tail',
    String(lines),
    service,
  ]).catch(() => ({ stdout: '(could not get logs)' }));
  console.log(`\n  docker logs ${service} (last ${lines}):\n  ${stdout.split('\n').join('\n  ')}`);
}

Are they pretty? No. Can they be improved? Yes. Do they have to be improved at this specific moment? No.

The setup uses test infrastructure

const CLEANUP = process.env['CLEANUP'] === '1' || process.env['CLEANUP'] === 'true';
const CLEANUP_AFTER = process.env['CLEANUP_AFTER'] === '1' || process.env['CLEANUP_AFTER'] === 'true';
const NO_START = process.env['NO_START'] === '1' || process.env['NO_START'] === 'true';

// ─── configuration ───────────────────────────────────────────────────────────

const COMPOSE = ['compose', '-f', 'docker-compose.yml', '--profile', 'observability'];

const URLS = {
  app: 'http://localhost:3000',
  prometheus: 'http://localhost:9090',
  tempo: 'http://localhost:3200',
  loki: 'http://localhost:3100',
  grafana: 'http://localhost:3001',
  otelCollectorMetrics: 'http://localhost:8889/metrics',
};

// Fresh client per run — avoids stale cart state from previous runs.
const SERVICE_NAME = 'expressjs-with-postgresql';
const CLIENT_ID = randomUUID();
const CART_ENDPOINT = `${URLS.app}/clients/${CLIENT_ID}/shopping-carts/current/product-items`;
const CONFIRM_ENDPOINT = `${URLS.app}/clients/${CLIENT_ID}/shopping-carts/current/confirm`;

// Matches the .http file — unitPrice is resolved server-side.
const ADD_PRODUCT_BODY = JSON.stringify({ productId: randomUUID(), quantity: 10 });


before(async () => {
  console.log(`\n▶ client ID for this run: ${CLIENT_ID}\n`);

  if (NO_START) {
    console.log('▶ --no-start: skipping docker compose and app startup');
    return;
  }

  if (CLEANUP) {
    console.log('▶ --cleanup: killing port 3000 and tearing down stack (down -v)…');
    await execa('bash', ['-c', 'fuser -k 3000/tcp 2>/dev/null || true']).catch(() => {});
    await new Promise((r) => setTimeout(r, 500));
    await execa('docker', [...COMPOSE, 'down', '-v', '--remove-orphans'], {
      stdio: 'inherit',
    });
  }

  const stackReady = await fetch(`${URLS.prometheus}/-/ready`)
    .then((r) => r.ok)
    .catch(() => false);

  if (stackReady) {
    console.log('▶ observability stack already up — skipping docker compose up');
  } else {
    console.log('▶ starting observability stack…');
    await execa('docker', [...COMPOSE, 'up', '-d'], { stdio: 'inherit' });
  }

  console.log('▶ waiting for backends…');
  await waitFor(() => checkUrl('Prometheus', `${URLS.prometheus}/-/ready`), {
    timeout: 90_000, label: 'Prometheus',
  });
  await waitFor(() => checkUrl('Grafana', `${URLS.grafana}/api/health`), {
    timeout: 90_000, label: 'Grafana',
  });
  await waitFor(() => checkUrl('Tempo', `${URLS.tempo}/ready`), {
    timeout: 90_000, label: 'Tempo',
  });
  await waitFor(() => checkUrl('Loki', `${URLS.loki}/ready`), {
    timeout: 90_000, label: 'Loki',
  });

  // /health returns { status: 'ok', service: 'expressjs-with-postgresql' } —
  // checking service name lets us distinguish our app from other processes on :3000.
  const checkOurApp = () =>
    checkUrl('app /health', `${URLS.app}/health`, async (res) => {
      const json = (await res.json().catch(() => ({}))) as { service?: string };
      if (json.service !== SERVICE_NAME) {
        console.log(
          `    app /health: service="${json.service ?? '(missing)'}", expected="${SERVICE_NAME}"`,
        );
        return false;
      }
      return true;
    });

  const appIsOurs = stackReady && (await checkOurApp());

  if (appIsOurs) {
    console.log('▶ app already running and healthy — skipping npm start');
  } else {
    const portTaken = await fetch(URLS.app).then(() => true).catch(() => false);
    if (portTaken) {
      // Port is occupied but not by our app — stale process or unrelated service.
      console.error(
        '\n  ✗ Port 3000 is occupied by a process that is not this app.\n' +
          '  It may be a stale version of this app (connected to a wiped database)\n' +
          '  or a completely different service.\n' +
          '  Fix: run  npm run verify:observability:cleanup  to kill it and restart,\n' +
          '  or manually free port 3000.\n',
      );
      process.exit(1);
    }

    console.log('▶ starting app…');
    app = execa('npm', ['start'], { stdio: 'inherit' });

    await waitFor(checkOurApp, { timeout: 60_000, label: 'app /health' });
  }

  console.log('▶ setup complete\n');
});

As you see, nothing fancy, the cleanup is even simpler

after(async () => {
  if (app) {
    console.log('\n▶ stopping app…');
    app.kill('SIGTERM');
    await app.catch(() => {});
    console.log('▶ app stopped');
  }

  if (CLEANUP_AFTER) {
    console.log('▶ tearing down stack (down -v)…');
    await execa('docker', [...COMPOSE, 'down', '-v', '--remove-orphans'], {
      stdio: 'inherit',
    });
    console.log('▶ stack torn down');
  } else {
    console.log('▶ stack is still running');
    console.log('▶ to clean up: npm run verify:observability:cleanup');
  }
});

Having that we can run tests:

test('successful command returns x-trace-id header', async () => {
  const res = await fetchWithDiag('POST add product', CART_ENDPOINT, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: ADD_PRODUCT_BODY,
  });

  assert.equal(res.status, 204, `Expected 204 — body logged above`);

  const header = res.headers.get('x-trace-id');
  if (!header) {
    console.error(
      '  ✗ x-trace-id missing — verify the wrapper app in src/index.ts ' +
        'adds it via @opentelemetry/api before mounting the emmett app',
    );
  }
  assert.ok(header, 'x-trace-id header missing');
  assert.match(header, /^[0-9a-f]{32}$/, `"${header}" is not a 32-hex trace ID`);

  traceId = header;
  console.log(`  trace ID: ${traceId}`);
});

test('OTel collector exposes Emmett metrics on port 8889', async () => {
  // Send a few more requests so metrics are definitely recorded.
  for (let i = 0; i < 5; i++) {
    await fetch(CART_ENDPOINT, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: ADD_PRODUCT_BODY,
    });
  }

  try {
    await waitFor(
      async () => {
        let text: string;
        try {
          const res = await fetch(URLS.otelCollectorMetrics);
          text = await res.text();
        } catch {
          console.log('    collector :8889: connection refused');
          return false;
        }
        const emmettLines = text.split('\n').filter((l) => l.startsWith('emmett_') && !l.startsWith('#'));
        if (emmettLines.length === 0) {
          const allFamilies = [...new Set(text.split('\n').filter((l) => !l.startsWith('#') && l).map((l) => l.split('{')[0]))].slice(0, 5);
          console.log(`    collector :8889: no emmett_* metrics yet. Present: ${allFamilies.join(', ') || '(none)'}`);
          return false;
        }
        return true;
      },
      { timeout: 90_000, interval: 5_000, label: 'emmett metrics on collector :8889' },
    );
  } catch (err) {
    await diagnoseCollector();
    await diagnoseDockerLogs('otel-collector');
    throw err;
  }
});

I put it into a single file that can be run as a regular Node.js script.

It already showed me (and Claude) that what they initially did wasn’t working if you try to run it multiple times. It also showed that doing a full cleanup and rebuild, and making it reproducible, needs more work.

Is it done? Not yet; it takes too much time and resources to run it continuously throughout the pipeline. The code is a bit messy, so it needs to be organised. It’s segmented into blocks, includes basic automation and tests, and has already gone through some failures to get it done.

Could I do it better? Sure, and I will improve it, but that’s not the point. I wanted to show you my findings during weekend vibing (without beer tho), the real, not polished iteration, before I run the next one.

The main idea behind OODA loops is not to be perfect, but to iterate quickly, gather feedback as soon as possible, learn from it, develop another theory, and verify it through action.

It’s not about vibing, but it’s also not about analysis paralysis.

I hope you’re now better equipped to think about when vibing, with beer or without, with LLMs or without, actually helps, and when it doesn’t.

Vibe coding is just high-frequency steering. It only works if you have a Harness: a mechanical way to observe and orient, so you don’t steer the whole project into a wall.

Act takes seconds now. Observe takes as long as it always did. Without a harness, you’re not going faster; you’re just making more stuff you haven’t checked.

Harness is not magic, a new discipline, or the next buzzword; I hope I showed you that a bit in this article on what it may look like.

So iterate fast, but wisely remembering to do the full loop. It’s great that LLMs can help us make Acting faster, but we should not skip other steps. We should aim for a fast feedback loop to iterate in the right direction and achieve continuous improvement, to deliver proper value.

Just like Vibing isn’t new, we shouldn’t abandon “old” engineering practices. We should also not replace collaboration with solitary self-high fives.

Check also:

• Emmett Pull Request with mentioned changes

• Interactive Rubber Ducking with GenAI

• The End of Coding? Wrong Question

• A few tricks on how to set up related Docker images with docker-compose

• Docker Compose Profiles, one the most useful and underrated features

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

Try not. Do. Or do not. There is no try!

I’m calling this the Yoda Principle.

Master Yoda said that to Luke Skywalker a long time ago in a galaxy far, far away. He was teaching Luke how to name commands properly while trying to untangle some legacy enterprise mess.

I’m sure you’ve also seen a death star of weirdly-named stuff. Some of them have already tripped you hours of thinking whether someone named this thing badly, or there’s some hidden truth behind it.

Let’s discuss that by the example: E-Commerce order fulfilment.

The order is placed automatically once the customer confirms the items in the shopping cart. We’re not making any product reservations before the shopping cart is confirmed, as this would lock it for other customers, and, as you know, they tend to drop items from their carts.

Once we receive the event notification that the cart has been confirmed, we’ll start the order fulfilment process. It starts (as mentioned) with the order initiation, which acknowledges and initiates the multi-step fulfilment process.

The first step is checking product availability before confirming the order. We need to determine whether we can proceed with completing the shipment and initiating product payment. If the product is unavailable, we need to either ask the customer to wait until we have it again or cancel the order.

We have dedicated modules for order fulfilment and for inventory. Fulfilment is the orchestrator, and inventory is responsible for tracking the state in warehouses.

The ordering module would need to call the inventory module to verify product availability. We could send a command (through the messaging system or web api). How would we name it?

What about VerifyProductExists? We’d send the product id and quantity from the order information, and return true if we have enough products, false otherwise. Sounds fair, right?

Well, it may seem nice at first glance, but what happens if more than one order verifies the same product availability, and we’re running short?

Then we’re vulnerable to race conditions I described in Tell, don’t ask! Or, how to keep an eye on boiling milk. The information we get is only valid at the time of querying. If we don’t lock the product quantity, it can change before we get a response (think: Black Friday-like demand).

Naming our command like VerifyProductExists is a mistake.

VerifyProductExists is not even a command; it’s a query. Command is a request (intention) to run business logic. Query is a request to return data.

Of course, pragmatically, our command can return status information about the result of our operation. But the intention is different.

What’s our real intention here?

The real intention is that we’d like to reserve products so we can ship them and get payment for them, not to verify if they exist. It’d be better to name our command as ReserveProducts or LockProducts.

Why does it matter?

If we’re naming our commands with Verify/Validate/Check prefixes, we’re putting ourselves into the wrong mindset. We’re not focused on actions and integrations, but just brief checking. If we’re in such a mode, it’s easy to handwave the integration complexity.

Locking for shipment may require sending someone to double-check that the product is in the warehouse and hasn’t been stolen, damaged, or other steps. It may be an async process on its own. Still from the ordering module, we should not care, as we’re telling what our intention is and expect to get events informing us whether the reservation succeeded, failed, or timed out (we don’t want to lock products forever in case of order fulfilment issues, but only for some time).

Prefixes like Verify/Validate/Check, etc., are just synonyms for trying. And well, commands are always a form of trying. The handling module can reject the command, as its business rules and state are the source of truth.

We should always assume that the command processing can fail. We should not be discouraged by that, and we should double-check everything. We should not be intimidated by the potential failure, but prepared for it.

We should try not. Do or do not. There is no try.

What if we have both? So VerifyProductExists and LockProducts? It can work if the first one is a query used by the Shopping Cart module, without any guarantee that the data isn’t stale, on a best-effort basis.

If we’re always requiring VerifyProductExists from the handling module before LockProducts, we’re making our communication chatty. I described that in What does Mr Bean opening the car have to do with programming? that this is not only a bad developer experience, but also just redundancy. Locking should already verify whether the product exists, so why require someone to memorise those scenarios instead of checking it internally?

The same goes for cases like:

• verify payment has been made,

• check if the order wasn’t already fulfilled,

• validate if the shipment has been completed,

• etc.

All of them either hide a missing business concept or should be a business rule verified within the specific action (e.g., confirming an order).

I recognise this may seem nitpicky, but big things are built on small details.

If we don’t think about such things, we’ll not only end up with misnamed integrations but also fight race conditions and incorrect boundaries.

Then we’ll be doomed.

So better think twice and do or do not.

May the force be with you!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

Have you ever heard phrases like.

Just an update, the milk ran out. Someone finished it and put the empty carton back.

Or

So everyone is aware, the meeting started 15 minutes ago.

Or

Heads up: the coffee machine is empty again.

I’m sure you either heard or used such phrases.

We all know that there’s some hidden intention behind it.

The intention is not to inform, but to trigger a certain action.

Formally, we’re reporting on events to announce the facts, but in practice, we’re using passive-aggressive words. The real intention is to command someone.

We don’t want to inform that the trash bin is full, but we want someone to take it out. We don’t want to inform that the coffee machine requires coffee beans refill, but we want someone to do it.

Passive-aggressive tone is the worst. It’s toxic for both sides of the communication. Usually, it’s just better to ask someone to do it.

The same rule applies in event-driven modelling. We should avoid passive-aggressive communication at all costs.

We should watch out for Passive-Agressive Events. So events that should be commands.

I already warned you in past not to let Event-Driven Architecture buzzwords fool us.

Event-Driven Architecture is an integration architecture style. We’re trying to model our business processes to run smoothly. To achieve that, we prefer a non-blocking communication flow, with things happening in parallel at their own pace. The goal is to achieve autonomous components, reducing the time needed to understand them. That helps maintain, or even replace them as your business evolves.

And events are enablers for that. They notify of what has happened, allowing other components to interpret facts and take the next steps.

But… Let me show one more photo.

It’s parliament, per the official definition: a room full of angry, shouting people.

If we model our communication only in terms of events, our system will look just like that. We’d just announce new facts in a passive-aggressive style and not be interested in what happens next. Oh, wait, are we really not interested? Actually, we are. If someone won’t do what we expect with our information, we’ll be even angrier.

What’s the difference between a command and an event? Both are messages. They convey specific information: a command indicating intent to do something, an event describes what has happened. From the computer’s point of view, they are no different. Only the business logic and the interpretation of the message can distinguish between an event and a command.

And that’s the main difference: commands can be rejected by the command handler. Events can only be ignored.

If we publish an event, we expect one or more consumers to be interested in it. Yet, we don’t know which components will do it. We just broadcast information.

This can easily change into passive-aggressive:

I did my work, now it’s your turn.

And here’s the crucial part. If we always have a single consumer for an event that needs to run the specific logic and expect to get the particular event back, then it should be a command. It’s not an event, we don’t inform. We want some component to take the next specific step and let us know when it’s finished.

Aren’t we making our communication synchronous?

What does it even mean, synchronous or asynchronous?

That’s what Sam Newman discussed in his great talk. The main conclusion is that synchronous vs asynchronous discussion is actually about blocking or non-blocking processing. And that’s much broader topic than the technical solution (so whether we call something in-process via an HTTP endpoint or a messaging system).

It’s a common misconception that events are published asynchronously through a messaging system (e.g. Kafka, RabbitMQ, SQS, WhateverQueue) and commands are sent through synchronous WebAPI. That can be true for a specific solution, but not as the general rule. As said, both events and commands are messages; we can send them through a messaging system or via HTTP (e.g. events via webhooks).

This misleading split came out from our expectation about handling. We expected the command handler to give us the result. For event handler, we don’t expect a specific result. At least in theory.

If we publish a specific event to the messaging system and expect a specific critical path of follow-up events, then we’re not making our communication non-blocking. It’s still sequential. We cannot proceed until the expected sequence occurs.

Whether something is blocking or not is not established by the tools we use, but by how our business process looks.

Speaking about it.

Let’s get back to our favourite E-Commerce Order scenario (read more in Predictable Identifiers: Enabling True Module Autonomy in Distributed Systems).

We could model it so we just publish the OrderConfirmed event and passively-aggressively expect that others will take it from there. So:

• The payment module will initiate the payment.

• Inventory will start completing shipments.

• The notification module will send a confirmation e-mail.

• Fraud detection module will check if the order is not rigged.

Once we receive information about a successful shipment or payment registration, we can complete the order.

You may notice two paths for order processing:

1. Blocking - We need to wait for information about payments and shipments. This is our critical path.

2. Non-blocking- Order process shouldn’t stop if the notification wasn’t sent or the data warehouse wasn’t able to process events. We’d like that to happen, but it’s expected rather than critical.

Now, both payments may fail (if our customer doesn’t have enough money), and the shipment may not be completed (if it’s Black Friday, and multiple people are competing for the same product).

If that happens, ordering module needs to take action, for instance, do reimbursement if the shipment wasn’t completed, and eventually cancel the order.

If we don’t foresee that and stay in passive-aggressive mode, we tend to forget about “negative” scenarios. it’s too easy to stay in I-Alread-Did-My-Job mode. This will have severe consequences: blocked orders, missed communication, and dissatisfied customers.

We may lear too late that another module can actually say no:

1. Payment module can say: Man, that’s not going to happen, you’ve already run out of money.

2. Shipment module can say: Man, I’m sorry, but you weren’t fast enough and we’ve run out of product.

And both of those scenarios will block successful order completion.

How to find such cases? Doing Example Mapping during modelling can be a good option for that.

Most importantly, we need to embrace the fact that some processes require direct, blocking communication, and others don’t. Just like in real life, sometimes it’s just more effective to tell someone to do something. We should avoid micromanagement and aim for autonomy, but not end up with anarchy.

In our case, it’d be better to have a coordinator (workflow, saga, process manager, To-Do List etc.) that publishes the OrderConfirmed event for modules not on the critical path and sends commands like RecordPayment and InitiateShipment.

By that, we’re separating responsibilities and making explicit what should be explicit. This also helps in understanding the business process, as you have a central place to see the critical flow and get proper observability.

Lacking tracing and observability of the business process is one of the most common issues I see in my clients’ projects. As said, if we don’t want to end up with parliament instead of proper communication in our system, we need to be explicit about our intention.

Is that all? Not quite, there’s one more message type we model as events that should not be events.

Gregor Hohpe, in “Enterprise Integration Patterns”, besides Event and Command defines one more message type: Document.

What’s the Document? It’s a state. Or to be precise: self-contained data we have at a certain point in time. We can store it, but we can also publish information about its new value.

That’s probably why Martin Fowler frames it as Event-Carried State Transfer, and I don’t like that term. For me, it’s extremely misleading as, it doesn’t tell what has happened, but what has changed. It just gathers the new version of the state (or the diff).

In my opinion, it’s a variation of State Obsession anti-pattern. Many people fell into that and believe it’s fine to connect the messaging system to the database, use tools like Change Data Capture, and publish it automatically to others. They end up with passive-aggressive communication style:

You have all you need. The whole state is in the events, just interpret it.

How can you reason about what has happened if instead of OrderConfirmed you get OrderCreated, OrderUpdated, OrderDeleted? You’d need to do the diff, compare with previous values, and do the guess about the reason of the specific change.

You deal with Clickbait Events and have a leaking business abstraction. All consumers need to understand the internals of your processing to detect a specific type of change. I wrote about it in detail in Internal and external events, or how to design event-driven API.

Again, the loose coupling of the event-driven processing is only loose for producers; consumers need to adapt. This can lead to hidden coupling, where a change in the producer breaks consumer flows. And that’s the worst type of coupling you can get.

If we’re making commands explicit, we’re also making an explicit relationship between components. It’s no longer flattened to producer <=> consumer, where the producer always shapes the communication. Now, if the other component exposes a command, that’s the driving force behind its behaviour. This helps to shape autonomy. In our case, we could make the Payment Module a generic module with a stable public API for registering payments, and an ordering module that requests them, in accordance with the Shipment Module. Fraud Detection could continue subscribing to events, as it already does. Context Mapping can greatly help in finding those relationships.

TLDR

We tend to be all about events these days, but they’re not the only message types. In our systems, messages take various forms: Events, Commands, and Documents, each serving distinct purposes:

• Documents are all about state transitions, which are essential for syncing data across services but missing deeper business insights.

• Commands represent a clear intent to act, directed with an expectation of execution, and can be accepted or rejected.

• Events are immutable facts, announced without waiting for a response. They’re like broadcasting news, hoping it catches the right ears.

Event-Driven Architectures enable loose coupling, but only for producers. To make consumers loosely coupled, we need to take extra steps, embrace different message types, and have them participate in modelling business processes.

If we go too far with an event-all-the-things communication style, we’ll make our system a room full of shouting people, with a passive-aggressive communication style. Or just aggressive.

In consequence, we won’t know what’s happening in our system, will see only noise, and will have a hard time making it reliable, observable and predictable. We should treat our messages as a communication contract, API and model their flow in a way that shapes our regular communication.

So next time, ask yourself if your event shouldn’t be a command. If it has always had a single consumer and you expect a specific event back, then it’s probably so. It’s all about being clear about the intention, not lying to yourself and others.

I hope this article will equip you with the knowledge to fix that.

If you’re dealing with such issues, I’m happy to help you through consulting, training or mentoring. Contact me and we’ll find a way to unblock you!

See also more in series about event modelling anti-patterns:

• State Obsession,

• Property Sourcing,

• I’ll just add one more field.

• Clickbait event,

• Passive Aggressive Events,

• Should you record multiple events from business logic?,

• Stream ids, event types prefixes and other event data you might not want to slice off.

Check also more general considerations:

• Events should be as small as possible, right?,

• What’s the difference between a command and an event?,

• Internal and external events, or how to design event-driven API,

• Event Streaming is not Event Sourcing!,

• Don’t let Event-Driven Architecture buzzwords fool you,

• How to design software architecture pragmatically,

• How to deal with privacy and GDPR in Event-Driven systems.

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

One of the first indications of getting old(er) is when people stop getting your movie or music references. Of course, based on this rule, some people are always old. That happens.

Recently, I realised during my workshops that referencing Friends is not so cool anymore. It started to happen when I was explaining the Example Mapping technique.

It always starts with “The one where”.

Just like in Friends.

I started to notice a bit slower head nodding and a bit more awkward smiles from the attendees. I repeated

You know, like in titles of Friends episodes.

And head-nodding stopped; only totally awkwardly polite smiles remained. Definitely, it wasn’t The One Where Everybody Finds Out. So I finally asked:

You weren’t watching Friends, didn’t you?

Obviously, the answer was:

Emmm. No…

Okay, then, if you don’t know Friends or the Example Mapping technique, this will be The One Where You Find Out.

Let’s say that we’re working on the guest checkout feature for a hotel management system.

We could start by asking the business how it works. We could get an answer that:

The guest approaches the desk and requests checkout. The clerk inquires about the quality of the products and services, and after receiving an answer, requests the room key. After gathering, the key clerk checks whether the balance is settled. If it’s settled, then proceed with the checkout. Marking the stay as completed.

Sounds straightforward, but we should already have several questions popping up, e.g. what does it mean that “balance is settled”? We could get quick feedback that:

This means that the difference between the sums of all charges and payments is equal to zero.

Then we could try to come up with an example:

Ah, so for instance, when guests haven’t paid upfront for their stay, right?

Right.

Oh, then we need to charge them, right?

Right.

We could visualise what we discovered in the following way:

Now, this generated another flow for us. We have a new feature we weren’t aware of: the guest’s stay payment registration. Let’s try to start this time from the visualisation.

It’s the one Oskar pays for his stay, because he wants to check out but didn’t pay upfront. The payment is registered, and we can try checking out again. Sounds fine, but we should ask whether there are any rules for payments. It may appear that:

Yes, there are some, for instance:

• Only guests with a valid credit card can pay with it for their stay,

• Guests paying in cash need to hand it over before accepting the payment.

And hey, we just found out new business rules, let’s put them on the board and update our flow to be more precise and reflect our scenario by adding a note that this scenario represents a guest paying with a credit card.

Now, what if the payment fails? Can it fail? Let’s ask the business!

Payment may fail if the payment gateway is unavailable or the issuer rejects it.

What if our internet connection is down for the whole day?

The clerk should ask the guest for cash.

What if the guest doesn’t have cash? 🤔

Then the shift manager can authorise unsettled balance checkout and register a charge with a delayed due date.

Here’s the updated flow. The one where Oskar pays for his entire stay with a credit card, but the Internet is down, and he doesn’t have cash.

Now, we found out:

• A new outcome, failed payment,

• A new rule, that we need Internet access to authorise credit card payment,

• A new feature, the shift manager can authorise unsettled balance checkout and register a charge with a delayed due date.

How would the authorisation look? How should we register a delayed charge?

We should follow double-entry bookkeeping and register the authorisation (type of payment) with the unbalanced amount to settle the balance for today, and register an additional delayed charge for the same amount

The flow will look like:

And that’s precisely how the Example Mapping session looks like. It’s a structured conversation format created by Matt Wynne. You take a user story, gather a small group (usually a developer, tester, and someone from the business side), and spend around 25-30 minutes breaking it down together.

You don’t need a big setup, a huge ceremon, you don’t need sticky notes, you can just use plain text like:

Given: Example

When: We use specific features

Then: Based on business rules, we get a specific outcome.

Business people don’t need to give them to you in such form. You can use the interview as I showed above and note it on your own, while you’re discussing stuff. It’s also a nice way to collaborate and visualise your discussions.

You don’t even need to start with an interview; you can use the Example Mapping as a brainstorming tool to generate as many examples of your (part of the) system. Then, try to model it as you see fit and ask the business for clarifications in the preferred form. It can help facilitate discussion with your team, not only with business stakeholders.

It’s super helpful, as misunderstandings are expensive. When multiple people try to describe the same rule using real examples, you’ll quickly see where assumptions don’t match. Better to find that out in a 30-minute chat than after two weeks of coding the wrong thing.

It also works as a readiness check. Too many red cards? The story isn’t ready. Too many blue cards? The story is probably too big. If examples come easily and everyone nods along - you’re good to go.

There’s also more, as you can try to distil business rules based on the examples and outcomes the business describes.

It’s worth noting that I cheated you in colours. The original looks like this:

Why did I change them?

Example Mapping plays nicely with other collaboration techniques like Event Storming. And if you’re familiar with the Event Storming colour scheme, that’s also the reason why I used it. I aligned them. See:

I’m typically using it during modelling sessions to:

• brainstorm (read more in Start Alone, Then Together: Why Software Modelling Needs Solitary Brainstorming),

• challenging existing models with real-world examples,

• expanding the model with uncovered (through examples) use cases,

• finding business rules,

• helping with facilitation by looking at the model from a different perspective.

And hot spots and notes, as known from EventStorming, are super helpful here. Read also more in The Underestimated Power of Hot Spots and Notes in EventStorming.

What’s more, if you look at the Given/When/Then pattern, you may notice that it works nicely with Behaviour-Driven Design. I already wrote that Behaviour-Driven Design is more than tests. How to do it? Check here.

I’ll also expand on it in the next articles. I’m doing the extreme Example Mapping with events, so stay tuned, the more will come.

For now, check also those materials:

• Seb Rose - short, practical and actionable intro to Example Mapping

• Kenny Baas-Schwegler - showing how to use Example Mapping with EventStorming

• An introduction by Matt Wynne himself,

• Other quick intro by Gojko Adzic.

And most importantly, try it. Take one feature from your system, try to crunch it, or start your design session with this technique.

Most of the teams I’m working with are enjoying this technique, as it’s a fun way to get quick, actionable outcomes.

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

You may already know that I’m a GenAI sceptic. And a general sceptic.

Do you know that scepticism comes from the Greek σκέπτομαι (skeptomai), meaning ‘to search, to think about, or look for’? So my intention is not to say no to everything new, but more to think about it first, and understand before I say yes.

There’s a lot of stuff about GenAI that makes me smile, but I still understand that my way is my way, and I won’t stop the world. I won’t even try. Thus, I want to research and consider how those tools can help me. I already wrote that I don’t feel like 10x Dev, but I’m finding more ways to get help from it.

One of the ways that helped me is something I call “Interactive Rubber-Ducking”.

Initially, I called it just brainstorming, but that wouldn’t be precise, as I’m not using it to brainstorm ideas, more to challenge and clarify them.

Most of the code I write nowadays is done in my OSS projects. I’m grateful to have a great community with people actively contributing in different ways; still, the canonical design and code work is on my side. As I work in an event-driven niche, I’m often alone with my own thoughts. I try to use the RFC process and discuss it with other fellow humans, but they are not always available. Even if they do, to avoid wasting their time, I need to know what to tell or ask them. I need to give some proposals (with alternatives) to have an effective discussion. I may seem organised, but that’s not always the thing. Sitting in your own head is not a great place to be in general. If you’re a technical leader or an architect, I’m sure that you know that solitude too well.

GenAI tools are not great sparing partners. They’re Yes men. If they read this article, they’d for sure confirm it. They’d probably do it even without reading it. Of course, you can ask them not to be sycophant. You can ask numerous MUSTS with capital letters and bolded NEVER here and there, and it can help, but it won’t fully beat the way they were trained.

And talking to yourself is a similar experience: you’ll find numerous ways to justify your own decisions, and looking at the same place for too long will make you miss obvious blind spots.

Ok, so why would we take those two blind “people” and try to make them help each other?

That’s kinda what “Interactive Rubber-Ducking” is. It takes a blind human with an idea, and another blind not-so-human asking questions. It starts with such a prompt:

Ask me one question at a time so we can develop a thorough, step-by-step spec for this idea. Each question should build on my previous answers, and our end goal is to have a detailed specification I can hand off to a developer. Let’s do this iteratively and dig into every relevant detail. Remember, only one question at a time.

Once we are done, save the spec as spec.md

Before asking another question, store the previous one with the answer in qa.md. Write literally the question and answer, not just a summary.

Here’s the idea:

Don’t treat it as “one magical prompt that will change your life”. Most important is why we’re doing it, what happens next, and who’s actually doing the work. Spoiler alert: it’s not LLM.

I’m using it as a command in Claude Code, and, most importantly, with beefier models like Opus, which can better reason and ask better questions. Doing it with lower-level models always gave me much worse results.

I’m using it with Claude Code, not Claude Chat, because I want the model to scan my codebase. I can ask it to look in certain areas or to reference my answers. I can even ask to search the web or MCPs like Context 7 to check documentation and APIs for popular libraries. Then it’s getting more into brainstorming sometimes, than rubber-ducking, but that’s fine.

As a result, we’ll get two artefacts:

• qa.md - with the log of the back-and-forth discussion,

• spec.md - in theory spec built by LLM, but imho it’s more of a concise summary.

It may look like Specification-Driven Design, but it’s not.

My goal for this exercise is not to get an actionable specification.

The goal is to get our LLM-based Rubber Duck to ask us hard questions and make us think, not to make the LLM think for us. Find blind spots, and challenge our thinking.

But we’re drivers, we need to know what we want to do, we need to know all the WHYs, and we also need to know HOW. LLM is here to help, but not to do creative work for us. It just pulls it out from our heads.

It also helps to see how our design may be seen by others, especially such mediocre thinkers as LLMs.

I don’t expect the Agent to be able to start implementing the spec. I expect it to reflect all considerations and summarise findings. I’m always double-checking to make sure it includes all the important points. If not, I’ll keep doing Q&A until I’m satisfied.

Having both of those files will allow us to keep a full discussion without losing important details, and a shorter version. We can feed that to another model for review, or try to work on tasks and develop a more detailed plan. Sometimes I plan on my own; for simpler tasks, I may ask the LLM to do it fully. Usually, I’m driving the LLM step by step, passing just specific asks.

The example? Why not.

I recently did such an exercise, trying to narrow down how to introduce the Second-level cache to Pongo and Emmett. What’s Second-level cache? A second-level cache is a local store of data managed by the persistence provider to improve application performance.

Why do I want to introduce it? Because I got an issue from the user that rebuilding projections with a lot events can take too long. One of the reasons is that applying an event on a projection takes:

• loading the current state,

• updating it,

• storing the result.

Those are two operations per event. If we have a batch of 100 events, that’d mean 200 operations; for 1000 events, this would be 2000, so the classical N+1 problem. We could do it differently, and within a batch:

• group events by the target documents,

• load all of them in one operation by finding all documents within an array of ids (taken from events),

• caching them,

• applying events in memory,

• storing updated documents in one operation.

Then we’ll get at worst 102 operations for 100 events and 1002 for 1000 events, so a linear increase.

Still, I decided to add batching and introduce 2nd-level cache as a solution. I and my experience were a driving force behind that. I could, of course, rubber duck it, but I had that already planned, not just detailed. I also did some initial research before doing the session.

Let me show you the result below. Not too bad if you ask me. Of course, the task wasn’t very innovative; it’s a standard way for mature solutions. Still questions were surprisingly good and helped me.

I hope this log will show you (besides how bad I’m at typos) some insight into how I think, talk, and consider the design.

Tell me your thoughts, if that matches your way, if you haven’t done it yet, try it and tell me how it went!

Interactive Rubber Ducking example

My idea

I’d like to provide 2nd-level cache to Pongo. I want to either use keyv (https://www.npmjs.com/package/keyv) or lru-cache (https://www.npmjs.com/package/lru-cache) packages.

For now, I’d like to do it by id.

I probably need to either extend pongoCollection src/packages/pongo/src/core/collection/pongoCollection.ts with findeOneById and findManyByIds methods, or check if _id is passed in and then try to access the cache.

Cache should be set up either on the pongo collection, pongo db or pongoClient.

Optimistic Concurrency errors should invalidate cache (as that means someone else updated the record in db, e.g. through a different node).

InsertMany and insertOne should put it into the cache.

I’d also like to expand the handle method to allow passing either an id or an array of IDs to support batch loading of entities and improve performance, and to support batch update/insert/delete.

Thoughts?

Which defaults should I set? (Of course, allowing users to override it).

Check the related code I pasted. I want to keep it aligned and as non-invasive as possible.

Q1: Cache library choice — lru-cache, keyv, or custom interface?

Question: You mentioned both keyv and lru-cache. They pull in different directions. Which trade-off do you prefer?

• A) Start with lru-cache, optionally let users swap adapters later.

• B) Start with keyv so cross-node sharing works out of the box.

• C) Define a thin cache interface (get/set/delete/clear) and ship lru-cache as the default adapter.

Answer: Custom cache interface (option C). No dependency on external APIs — keep it pluggable. Start with an in-memory provider only. Considered writing a custom LRU with me, but wants to ship fast. Key point: don’t stick to any third-party API surface, allow pluggability.

Q2: Where should the cache live in Pongo’s hierarchy?

Question: Three attachment points: A) PongoClient level (shared across all), B) PongoDb level (per database), C) PongoCollection level (per collection). Or configurable at multiple levels?

Answer: Allow defining it at different levels — cascading configuration. Set a default higher up, override per collection if needed.

Q3: Inheritance/override behavior for cascading cache config?

Question: For the cascading config — what should the inheritance/override behavior be?

Two sub-questions:

1. Should a lower level be able to disable caching even if a parent enables it?

2. Should cache settings (like max size, TTL) also cascade, or only the cache provider instance itself?

Answer:

1. Yes. Cache should be an object with type, general params and specific for this type, or 'disabled'. That’s a proper devex.

2. Yes, as above. But if one selects a different cache type then general params should be inherited but type-specific should not be inherited and should fallback to defaults.

Claude’s thoughts: The CacheConfig | 'disabled' pattern is clean — discriminated unions play nicely with TypeScript. The inheritance rule makes sense: general params (TTL, max size) cascade, but type-specific params reset to defaults when you switch providers, avoiding leaking nonsensical config (e.g., a Redis connection string bleeding into an in-memory cache). Also worth distinguishing between “I didn’t specify anything” (inherit from parent via undefined) and “I explicitly want defaults” (reset) — undefined vs 'disabled' vs CacheConfig gives a clean three-state: inherit / off / explicit.

Q4: Cache interface shape — async, batch ops, scoping?

Question: Settling the cache interface details:

1. Async or sync? Making the interface Promise-based from the start keeps it pluggable. Sound right?

2. Batch methods (getMany, setMany, deleteMany) — should the cache interface support batch ops natively, or loop internally?

3. Should clear() be scoped per collection, or always clear the entire cache?

Answer:

1. Maybe we could return either promise or no promise, then if we await it, it would support both? Also ensure that settings are close to what lru-cache and keyv are having.

2. Having them (even as syntactic sugar) would be a good devex.

3. We could pass down not only settings, but also the cache provider instance, then if settings are the same we can reuse cache without recreating it. If set had ttl provided, then we could do it by overriding those TTLs on item level. ClearAll probably would be something we should expose only for advanced, typically user should not be aware of cache.

Claude’s thoughts: MaybePromise<T> (i.e. T | PromiseLike<T>) works — await on a non-Promise just resolves immediately. No overhead for sync providers, no breaking change for async ones. Passing a cache instance is smart: config becomes either settings (Pongo creates the cache) or a pre-built instance (user brings their own). Multiple collections can explicitly share one cache instance. clear() should be internal/advanced — scope by collection prefix internally so a collection-level clear doesn’t nuke another collection’s entries when sharing an instance. For aligning with lru-cache/keyv settings: both support max (max entries) and ttl (TTL in ms). Start with max and ttl as the general params.

Q5: Cache key strategy

Question: Since cache instances might be shared across collections, how to avoid key collisions?

• A) Pongo manages key prefixing internally ({dbName}:{collectionName}:{id}). The cache interface works with plain string keys, namespacing is Pongo’s concern.

• B) Key is just the document _id. Simpler but only safe with per-collection instances.

Answer: A, definitely.

Q6: Cache invalidation and consistency with optimistic concurrency

Question: How should cache interact with different operations?

1. Successful writes → update/evict cache. Clear.

2. Concurrency errors → evict stale entry. Clear.

3. Reads with _id filter → cache hit or miss-then-populate. Clear.

4. handle method — should the read inside handle use the cache (risking more concurrency errors on stale data) or always go to DB?

Answer: Definitely use cache. We could consider disabling cache through handle options.

Claude’s thoughts: Use cache by default — optimistic concurrency is the safety net. If cached version is stale, write fails, we evict, caller can retry. Adding skipCache?: boolean to HandleOptions is a clean escape hatch for high-contention cases.

Q7: Should skipCache be available on read methods too?

Question: Should skipCache also be available on findOne and other read methods for symmetry, or keep it handle-only?

Answer: Good point, having skipCache on find methods (also findMany etc.) would be good.

Claude’s thoughts: Consistent API surface. Add skipCache?: boolean on the shared options type (e.g. CollectionOperationOptions) so it’s available everywhere without repeating yourself.

Q8: Batch operations on handle — signature and handler shape

Question: You want handle to accept an id or array of ids. Two design choices:

1. Overloaded signature (handle(id: string | string[], ...)) vs separate method (handleMany)?

2. Handler shape for batch: A) still receives/returns one document at a time (Pongo batches DB ops internally) or B) receives/returns full array?

Also noted: since PongoDocuments always have _version, if someone provides expectedVersion we could compare it against the cached version and decide if it’s stale without hitting the DB.

Answer: Option A for handler shape. handleMany would be consistent with other APIs, but single id or array (string | string[]) is clean enough.

Claude’s thoughts: Version comparison in cache is a cheap short-circuit — if cached _version doesn’t match expectedVersion, we know it’s stale without a round trip. On signature: TypeScript overloads can narrow the return type (string → single result, string[] → array). handle is a higher-level orchestration method, not direct CRUD, so a union input feels natural. Go with handle(id: string | string[], handler, options?).

Q9: Defaults — max entries, TTL, enabled by default?

Question: What defaults to set? Proposed: max: 1000, ttl: 60_000 (1 min), and should cache be enabled by default or opt-in?

Answer: Stick to recommended lru-cache defaults for now. Enable cache by default.

Claude’s thoughts: Enabling by default works because: (1) optimistic concurrency is always the safety net, (2) it’s document-by-id only so invalidation is straightforward, (3) users can set 'disabled'. lru-cache requires max to be set (no default) and has no default TTL. Suggested: max: 1000, no TTL (rely on LRU eviction + invalidation on writes/concurrency errors) — most lru-cache-aligned.

Q10: TTL or no TTL as staleness backstop?

Question: No TTL means cached entries live until LRU-evicted or invalidated by a Pongo write. If something updates the DB outside this Pongo instance (another service, direct SQL), the cache won’t know. Options:

• A) No TTL, accept the trade-off — OC catches it on next write.

• B) Set a conservative default TTL (e.g., 5 minutes) as a staleness bound.

Answer: Fine with using TTL and sometimes hitting OC issue. OC assumes rare conflicts.

Q11: Where does cache integration hook into existing code?

Question: Two approaches for minimal invasiveness:

1. Wrap at the pongoCollection level — intercept operations inside the factory function, driver/SQL layers untouched.

2. Decorator/wrapper pattern — cachedPongoCollection wraps PongoCollection, adds cache logic around delegation.

Answer: Leaning more to option 1. Thinks it’ll be easier to make observability precise eventually.

Claude’s assessment: Agreed with option 1. Cache decisions depend on internal state (whether filter targets _id, document _version, write success vs concurrency error). A wrapper would need to re-inspect all of that from the outside. For observability, cache hit/miss logic next to the actual DB call enables precise metrics in context. Pongo’s factory functions already compose behavior internally (transactions, schema migration) — cache is another concern at the same level. A decorator would duplicate filter-inspection logic and add indirection that doesn’t carry its weight.

Q12: Cache interaction with transactions

Question: How should cache interact with transactions?

1. Should writes within a transaction update the cache immediately, or only after commit?

2. Should reads within a transaction check the cache?

Answer:

1. After commit.

2. We should use cache, but allow skipping it on transaction or session options. We can also allow setting cache settings per session.

Claude’s thoughts: Reads within a transaction benefiting from cache is more aggressive but logical — most transactions are short-lived. The narrow case: insertOne then findOne same _id within a transaction — cache won’t have the uncommitted insert, falls through to DB, gets the right answer anyway. No correctness issue, just no perf benefit for that case. Per-session cache settings fit the cascading pattern (e.g., startSession({ cache: 'disabled' }) for batch migrations). Proposed cascade priority: client → db → collection → session → per-operation skipCache.

Q13: Cascade priority — does session override collection?

Question: Does the cascade priority client → db → collection → session → per-operation skipCache feel right? Session overriding collection is the non-obvious part — the argument being that session is the most immediate runtime context, while collection is structural config. Or would you prefer collection always wins over session?

Answer: That’s fine. Session should precede collection, as normally you create session to e.g. share connection, and join the number of operations logically, so it’d make sense actually to also do it to e.g. override settings like caching.

Claude’s thoughts: Session as logical grouping of operations with shared context (connection, transaction) — overriding cache settings at that level is extending the same concept. Bulk import in a session? Set cache disabled once on the session instead of skipCache on every call. Final cascade: client → db → collection → session → per-operation.

Q14: Cache provider error handling

Question: If the cache provider throws (e.g., Redis connection drops), should Pongo:

• A) Treat it as a cache miss — fall through to DB silently, maybe log a warning. Cache failures never break the application.

• B) Propagate the error — let the caller deal with it.

Answer: A — yes, when we add observability to Pongo, we need to make it also a metric and trace event/attribute/log entry.

Claude’s thoughts: Cache is an optimization, not a correctness requirement. Silent degradation now, observable degradation when the observability layer lands.

Q15: Cache warming, events/hooks, and delete operations

Question: Three remaining areas:

1. Cache warming — should Pongo support pre-populating the cache on startup?

2. Cache events/hooks — should users be able to listen to cache hits/misses/evictions?

3. Delete operations — deleteOne/deleteMany should evict from cache. Anything non-obvious there?

Answer:

1. For now keep it as a future concept, or define pluggable strategy for users.

2. You can add the basic hook to the Pongo Cache wrapper. Shouldn’t be that hard to just call some function.

3. Yes, they should evict if they can, we can delete (for now) when deleteMany deletes by ids.

Claude’s thoughts: Cache warming deferred — can add a strategy hook later. Basic event callbacks (onHit, onMiss, onEvict, onError) on the cache wrapper are low-cost and useful for debugging even before full observability. Delete eviction scoped to id-based deletes for now — filter-based deleteMany doesn’t evict (no way to know which ids were affected without querying first).

Interactive Rubber Ducking result

Pongo 2nd Level Cache — Specification

Overview

Add a document-level (by _id) 2nd level cache to Pongo with a pluggable provider interface. Ships with an in-memory provider backed by lru-cache. Enabled by default. Designed to be minimally invasive to existing code.

Cache Interface

PongoCacheProvider<T>

A thin, Pongo-owned interface. No coupling to any third-party API.

type MaybePromise<T> = T | PromiseLike<T>;

interface PongoCacheProvider {
  get(key: string): MaybePromise<PongoDocument | null | undefined>;
  set(key: string, value: PongoDocument, options?: { ttl?: number }): MaybePromise<void>;
  delete(key: string): MaybePromise<void>;
  getMany(keys: string[]): MaybePromise<(PongoDocument | null | undefined)[]>;
  setMany(entries: { key: string; value: PongoDocument; ttl?: number }[]): MaybePromise<void>;
  deleteMany(keys: string[]): MaybePromise<void>;
  clear(): MaybePromise<void>;
}

• MaybePromise return types: sync providers (in-memory) return values directly, async providers (Redis) return Promises. await handles both transparently.

• Batch methods (getMany, setMany, deleteMany) are first-class. Default in-memory implementation may loop internally, but the interface allows optimized batch ops for external providers.

• clear() is internal/advanced — not exposed to typical users. When sharing a cache instance across collections, scoping is handled via key prefixing by Pongo, not by the provider.

Cache key strategy

Pongo manages key prefixing internally: {dbName}:{collectionName}:{documentId}.

The cache provider works with plain string keys — namespacing is Pongo’s concern, not the provider’s.

Event hooks

The Pongo cache wrapper supports basic callbacks:

• onHit?(key: string): void

• onMiss?(key: string): void

• onEvict?(key: string): void

• onError?(error: unknown, operation: string): void

These are optional and intended for debugging and future observability integration.

Configuration

CacheConfig

type CacheConfig = {
 type: string;               // e.g., 'in-memory', 'redis', etc.
 max?: number;               // max entries (general param, cascades)
 ttl?: number;               // TTL in ms (general param, cascades)
  // type-specific options live here too, keyed by type
 [key: string]: unknown;
} | 'disabled';

Three states:

• undefined — inherit from parent level

• 'disabled' — explicitly turn off caching at this level

• CacheConfig object — explicit configuration

Cascading configuration

Cache config can be set at multiple levels. Each level inherits from its parent unless explicitly overridden:

client → db → collection → session → per-operation

Inheritance rules:

• General params (max, ttl) cascade down.

• If a lower level switches type, type-specific params reset to defaults (not inherited from parent).

• Session overrides collection — session is a logical grouping of operations, natural place to override runtime behavior (e.g., disable cache for a bulk import).

• Per-operation skipCache?: boolean is the most granular escape hatch.

Passing a cache instance

Users can provide either:

• Settings — Pongo creates and manages the cache provider.

• A pre-built cache provider instance — Pongo uses it directly.

If settings are the same across multiple collections, Pongo can reuse the same provider instance internally. When a user passes an instance, multiple collections can explicitly share one cache.

Defaults

• Enabled by default

• max: follow lru-cache recommended defaults (1000)

• ttl: follow lru-cache recommended defaults

• Default provider: in-memory (lru-cache)

Integration points

Where: pongoCollection factory function

Cache logic is added directly inside pongoCollection, not as an external decorator/wrapper. This gives cache operations access to internal state (filter inspection, _version, write outcomes) and keeps observability precise.

Read operations

findOne:

• If the filter targets _id, check cache first.

• Cache hit → return cached document.

• Cache miss → query DB, populate cache, return.

• skipCache?: boolean option available.

findMany / other query methods:

• If the filter is a list of _id values, check cache for each.

• Return cached hits, query DB for misses, populate cache with DB results.

• Non-_id filters bypass cache entirely (cache is by-id only).

Write operations

insertOne:

• After successful insert, put the document into cache.

insertMany:

• After successful insert, put all documents into cache.

updateOne / updateMany / replaceOne:

• After successful write, update the cache entry with the new document state.

deleteOne:

• After successful delete, evict from cache.

deleteMany:

• If deleting by ids, evict those ids from cache.

• Filter-based deleteMany does not evict (no way to know affected ids without extra query). Future improvement possible.

Optimistic concurrency

• On concurrency error (version mismatch), evict the stale entry from cache. This is critical — a concurrency error means someone else updated the record (e.g., from another node), so the cached version is stale.

• If a caller provides expectedVersion and the cached document has _version, compare them in memory. If they don’t match, we know it’s stale without hitting the DB — cheap short-circuit.

handle method

• Accepts id: string | string[] (overloaded: string returns single result, string[] returns array).

• Read phase uses cache by default.

• Write phase updates/evicts cache based on outcome.

• skipCache?: boolean available in handle options.

• For batch: loads all ids from cache, fetches misses from DB in one query, processes handler per document, batch writes to DB, updates cache after commit.

Transactions

• Writes update cache only after commit. Not during the transaction — uncommitted data should not enter the cache.

• Reads within a transaction check cache by default (most transactions are short-lived, cached data is probably valid). If the transaction modified a document and then reads it, cache miss falls through to DB which returns the correct uncommitted state.

• Cache can be disabled per-session: startSession({ cache: 'disabled' }).

• Per-session cache settings follow the cascade (session overrides collection).

Error handling

Cache provider errors are swallowed — treated as cache misses. The application never fails because of a cache failure.

• Cache is an optimization, not a correctness requirement.

• Errors trigger onError hook for debugging.

• Future: errors will become metrics and trace events when Pongo’s observability layer lands.

Future considerations (out of scope for v1)

• Cache warming: pluggable strategy for pre-populating cache on startup. Interface hook defined but no default implementation shipped.

• Query-result caching: caching results of non-_id queries. Complex invalidation — deferred.

• Cross-node cache sharing: Redis/Memcached providers. The interface supports it; no provider shipped in v1.

• Filter-based deleteMany eviction: would require querying affected ids before delete.

• Observability integration: metrics (hit ratio, latency saved, eviction counts), trace attributes, log entries for cache operations.

Implementation approach

1. Define PongoCacheProvider interface and CacheConfig types.

2. Implement in-memory provider using lru-cache.

3. Build Pongo cache wrapper (key prefixing, event hooks, error swallowing).

4. Wire cascading config resolution (client → db → collection → session → per-op).

5. Integrate into pongoCollection: intercept _id-based reads, update cache on writes, evict on deletes and concurrency errors.

6. Extend handle to accept string | string[] with batch support.

7. Add skipCache to operation options.

8. Tests: unit tests for cache provider, integration tests for cache + collection operations, e2e tests for concurrency scenarios.

Cheers!

Oskar

p.s, and yes, Nano Banana also did pretty well generating the cover of this post!

p.s.2. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

Be careful what you wish for, because your wish may come true.

What LLMs revealed is how many people in our industry don't like to code.

It's intriguing that now they claim and showcase what they "built with Claude", whereas usually that means they generated a PoC.

It's funny, as people still focus on how they're building, so it's all about the code. And if that's the message sent outside, together with the thought that LLMs are already better than "average coder Joe", then the logical follow-up question is: why do we need those humans in the loop?

I think that most people look at the forest and see trees. The current way of working with LLMs is not scalable. It's transition phase. I can't imagine calling myself an engineer and doing ONLY stringly-typed development with chat or markdown.

What I can imagine is getting help from it, and using those tools as a help for research, for generating the OUTPUT, still keeping me responsible for the OUTCOME.

Why am I saying that we're in the transition phase? As prompting in our natural language is not precise, it's verbose, and adding a translation layer between our freeform prompts and programming languages is a waste of time (and tokens, which LLM vendors love).

I think that we'll still be coding, but with some other layer, as LLMs are good with structured input, like programming languages. So we might need other programming languages than we have atm. Might we need different tools to evaluate LLMs' output to make it deterministic? Might we need a different approach for engineering to make it scalable? Might we need more?

Still, I don't see those discussions.

I mostly see: noise and celebrities who don't code showing their beautiful PoCs. And doing a mic drop about the end of coding. Just like PoC represents the whole Software Development Life Cycle.

So, will we code or not?

If the answer is yes, let's talk about what's next, then let's discuss how and when.

If the answer is not, then let's talk about how our job will change, ask if it's still engineering, etc.

Let's try to make our discussions more precise, more focused on the essence, and avoid them from becoming just a bunch of anecdotal evidence.

Noone outside our industry cares how we code unless it changes the cost, quality, or delivery time.

Let's discuss the impact that matters, rather than just the amount of code we produce (or not).

Now, guess who I am quoting:

Imagine you’re a software application developer. Your programming language of choice (or the language that’s been foisted on you) is Java or Typescript . You’ve been at this for quite a while and your job doesn’t seem to be getting any easier.

These past few years you’ve seen the growth of multiple incompatible architectures. Now you’re supposed to cope with all this and make your applications work in a distributed client-server environment. The growth of the Internet, the World-Wide Web, and “electronic commerce” haveintroduced new dimensions of complexity into the development process.

The tools you use to develop applications don’t seem to help you much. You’re still coping with the same old problems; the fashionable new object-oriented techniques seem to have added new problems without solving the old ones.

You say to yourself and your friends, “There has to be a better way”!

The Better Way is Here Now

Now there is a better way—it’s our new model. Imagine, if you will, this development world…

• It’s still dead simple.

• Your development cycle is much faster.

• Your applications can be created across multiple platforms. Write your spec once, and you never need to port them—they will be recreated if you without your hand-rolled modification on multiple operating systems and hardware architectures.

• Your applications are adaptable to changing environments.

• Your end users can trust that your applications are secure, and you can use protection against viruses and tampering through security scans.

You don’t need to dream about these features. They’re here now.

And also this from another source:

When I started interviewing programmers in 2005, I would generally let them use any language or tool they wanted to solve the coding problems I gave them. 99% of the time, they chose Java.

Nowadays, they tend to choose LLMs.

Now, don’t get me wrong: there’s nothing wrong with LLM as an implementation tool.

Wait a minute, I want to modify that statement. I’m not claiming, in this particular article, that there’s anything wrong with LLM as an implementation tool. There are lots of things wrong with it but those will have to wait for a different article.

Instead what I’d like to claim is that LLM is not, generally, a hard enough programming tool that it can be used to discriminate between great programmers and mediocre programmers. It may be a fine tool to work in, but that’s not today’s topic. I would even go so far as to say that the fact that LLMs aere not hard enough is a feature, not a bug, but it does have this one problem.

Well, I cheated you, but only a bit. I changed “Java” to “LLM ” and cut some phrases.

The first one is from “The Java Language Environment” by Sun Microsystems, introducing Java in 1995.

The second one was from Joel Spolsky’s “The Perils of JavaSchools” article, written in 2001.

Let me be clear. I’m not trying to do grandpa talk on the old days and claim that it’s the same old thing.

What I’m trying to say is that we were continuously introducing new abstractions into our development cycle to scale it. By scale, I mean: getting more people to deliver more code. Even Java was invented for precisely this goal. Yes, the one that’s together with craftsmanship madness presented as “the enterprisy complex environment”. In the early days, it was just said that it’ll make us dumber.

The goal of abstraction is not to gatekeep but to allow us to reduce cognitive load. We invented new languages to help us, but then added more components like a distributed environment, multiregion, because of globalisation. For the same reasons, we use cloud-native tooling so we don't have to deal with it.

Some of the architecture and security tools were commoditised by the cloud. We don’t need to think about much stuff we had to do before.

Will Claude-native do the same?

We don’t need to learn Assembler, C++, Lisp anymore; we have lost a lot of mechanical sympathy. We deal with higher abstractions, but we still engineer solution, we still code, is it a different coding? It is. Is it better or worse? Well, it is how it is. As Gerald Weinberg said:

Things are the way they are because they got that way

And now the question is whether we’re fine with the way we’re doing stuff, and where we're getting to.

Personally, I don’t think that stringly-typed markdown or chat-based design will be the thing in the future.

Knowing how skilled we were always with:
- breaking down tasks into smaller chunks,
- writing precisely what we had in mind,
- thinking before doing,
- waging tradeoffs,

I'm optimistic about the Spec-Driven Design idea. It's going to be great.

Not.

If we want to call ourselves engineers, we need to put more structure and determinism.

I agree that reviewing all code generated by the GenAI is not sustainable.

But I also don’t think that generating tons of code is, in general, sustainable.

With the current state of the art we have, sure, that’s some solution to just generate based on the tools we have.

But let’s start to think what’s next.

Simon Wardley brought an interesting point:

That said, it is fine for an entire culture to decide that producing outputs matters more than understanding mechanisms. You only have to compare the practical engineering of the Roman Empire and the loss of inquiry from science in the Hellenistic age to see this. When the Roman Empire collapsed, the practical knowledge embedded in those institutions (how to maintain aqueducts, how to produce certain grades of concrete) was lost remarkably quickly. Not because people decided to forget it, but because the knowledge was procedural, embedded in chains of practice rather than recorded as transferable understanding. When the chains of practice broke, that embedded knowledge went with them. We had to rediscover the art of inquiry (i.e. Science) to bring them back.

So if we don’t code, how do we hone our skills? How will newcomers from LLMSchools (using Joel’s terms) be able to decide whether something is wrong or right? I don’t think that you can be good on something without doing it.

The paradox with code is that it’s not an asset; it’s a liability. Some say that code doesn’t matter, only proper design. But how do you define “proper design”? Yes, code style doesn’t matter as long as it works as expected. But…

But code eventually matters, as that’s the source of truth for what’s on production. As Alberto Brandolini said:

It's developers' (mis)understanding, not domain experts' knowledge, that gets released in production.

Now, it’s the developers’ and LLMs’ misunderstandings that are deployed to production, not the expert’s knowledge. Neither the markdown spec.

And coding is just one danger.

Outsourcing thinking is an even more dangerous path, as:

• If LLMs are doing everything, then again, what are humans for? Aren’t we cutting the branch on which we’re sitting?

• LLMs are statistical parrots. They repeat the most possible answer. Which means mediocre. This can still be fine enough for many cases, but for those we want to make a difference for? Definitely not.

• Just like we’re losing our coding skills by not doing them, we’re losing design skills by not practising them.

Of course, whatever happens, LLMs will stay with us. How and where it’s hard to say. Unless you have a Magic 8 Ball of 100% correct predictions. I don’t.

That’s why I’d like our industry to finally start mature discussions on the real impact. I would like us to stop acting like children, bragging about generating code and then claiming that code doesn’t matter.

I’d like to think about how to reshape our SDLC process and make it sustainable.

I’d like us to think about what tools we need, and how to change what we have.

If we won’t finally start to do it, then things will be the way they are because they got that way.

And it might not be what we wished for.

Cheers

Oskar

p.s. to kinda prove that I’m more sceptic and pragmatic than hater, I recently started playing with building an Agent with Emmett to better understand those tools. If you’d like to read about the findings and honest thoughts I have while doing it, please comment!

p.s.2. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

Last time, I shared with you how sneaky I was on transaction handling.. Today, the opposite: I’ll tell you how I fixed the issue when I tried to be too sneaky. I already told you that Sneaky Code Bites Back. The moral? Do as I tell, not how I do.

In some environments, we’re spoiled. We’re getting a lot from a Base Class Library or standard frameworks, so we stop thinking that those issues can exist. For instance, serialisation. Do you know how many data types JSON has? 6. Six. Sześć.

Exactly those:

• string,

• number,

• boolean,

• object,

• array,

• and (TADA!) null.

What about number precision and size? It is. That’s what I can tell you, but it’s not enough, e.g., to keep big int/long, etc. What about Dates? Also, there are none. I wrote about it longer here or how much fun that brings.

If you use statically typed languages and runtimes like C#, Java, etc., your serialiser can, in addition to parsing, perform mapping and, sometimes, validation. And it can also be tricky, as nicely Alexis King put in her “Parse, don’t validate”.

If you’re in a dynamic environment, like JavaScript, then you’re left with parsing and explicit mapping afterwards. What about TypeScript? Same case, types are only used during compilation, then erased and not visible at runtime. So, the place where we do parsing.

Because JSON was defined a long time ago, JavaScript moved on and now supports bigints (Big Integers) and Dates natively (what an achievement!), which created a gap I wanted to fill.

As you know from my previous articles (e.g. this one), big integers are quite important in distributed processing. You can represent the position in log with them. Since your log may be quite long, regular numbers aren’t enough. Or they’re long enough, until they overflow, then they’re not anymore.

I’m using those bigint types extensively in internals in Emmett and Pongo. And I store them in JSONs. I store them as alphanumeric strings, because strings don’t have a max length (or at least I don’t know such).

So, for instance, an event payload can look like:

{
  "type": "InvoiceIssued",
  "data": {
    "invoiceNumber": "123",
    "version": 1,
    "issuer": "John Doe",
    "issuedAt": "2026-02-23T14:07:20Z"
  },
  "metadata": {
    "streamPosition": "3",
    "globalPosition": "928391"
  }
}

As you can see in the metadata stream and global positions, the values are bigints (even if they’re smaller than the maximum value), and data can also use bigints if the user decides to (e.g., invoice number).

And encoding data is simple: you convert it to a string, call it a day. But how to get it back?

And here’s where my struggles started. How do you know that someone intentionally used bigint when they just wanted to store a number as a string?

There are several options. The first one is: encode value.

We could store it, for instance, as:

• prefixed value: “_bigint:928391”. But then you need to find a prefix that will be unique enough not to cause conflicts,

• nested object, e.g. { “_kind”: “bigint”, value: “928391” }.

Then, either based on the prefix or the object structure, we could automatically decode the value. Still, ther creates other issues, as the structure no longer matches the original value. If we’re just storing and retrieving, that shouldn’t be so bad, but… But remember that in Pongo I’m allowing the use of PostgreSQL and SQLite as document databases, supporting such queries:

const invoices = pongoDb.collection<Invoice>("invoices");

const invoiceNumber = 123n;
const invoice = await invoices.findOne({ invoiceNumber });

That gets translated into a fancy JSONB SQL query.

Of course, I could work around it by encoding the value, but… But I was lazy!

I decided to use a Get Out of Jail Free Card and just treat all strings with numbers as bigints. Sneaky. And it will get even sneakier.

In JavaScript, JSON.parse accepts a parameter that allows you to provide custom mapping logic. I decided to use it and check if the string is alphanumeric, and gulp, I’ve used a regular expression to parse it:

const bigIntReviver: JSONReviver = (_key, value) => {
  if (typeof value === 'string' && /^[+-]?\d+n?$/.test(value)) {
    return BigInt(value);
  }

  return value;
};

Yes, it’s either DNS or Regex. Or both.

I explained in another article that JavaScript runtime doesn’t like where you do CPU-heavy computations.

Small Regex isn’t CPU-heavy, but if you consider that ther will be done for each string in each document or event you try to deserialise, and multiply that by the number of concurrent requests? That can cause the JavaScript event loop to freeze.

What’s more, I plugged that automatically into node-postgres driver custom type handling, so each JSONB deserialization goes through it.

Again, not shit, Sherlock. I should have known it wasn’t the best choice, but I was busy trying to be sneaky at that moment.

Fortunately, a user, Dawid, benchmarked and noticed CPU freezes. It wasn’t catastrophic, but clearly needed a fix.

The Shift

And here I had several options. I could keep hacking on the same idea- maybe replace the Regex with a simpler string check, still globally. Or I could just ignore bigints during deserialisation entirely, let them stay strings, call it a day, move on. Or I could apply the encoding I mentioned earlier, prefixed values, and nested objects. All of those would fix the performance issue. And all of those would be the same kind of wrong choice I already made: trying to solve a schema problem without the schema.

Because that’s the actual mistake here, not the Regex. The pg driver has no idea what your schema looks like. It doesn’t know that “928391” is a bigint and “John Doe” is a name. It doesn’t know that “123” is an invoice number (bigint!) and “90210” is a zip code (string!). I asked it to guess, and it guessed wrong, because there is no right guess at that level.

Enough is enough. I had been planning to do ther properly for a while, and the performance issue gave me the push I needed.

Old rule says: “Make it work, make it right, make it pretty”. I had “make it work” covered for a long time. Now it was time for “make it right”.

And honestly? It wasn’t that hard. Maybe because “make it work” came first, I already understood the problem well enough to see the shape of the solution.

In Pongo, I dropped the automatic bigint parsing from the driver entirely. If you want bigint or date parsing, you say so at the client level:

const client = pongoClient({
  driver: databaseDriver,
  connectionString: postgresConnectionString,
  serialization: {
    options: {
      parseBigInts: true,
      parseDates: true,
    }
  },
});

By default, strings stay strings. You opt in. I didn’t want to break things for users who don’t care about bigint precision or don’t have performance-sensitive workloads. The serializer became an explicit parameter passed down to each query, each collection, each operation- instead of a global that silently changed everything.

That was the “make it right” part for Pongo. But disabling alone isn’t a solution, it’s a band-aid. Users who need bigints and dates still need a way to get them back after deserialisation. The question is: where does that conversion happen?

And that’s where upcasting comes in. Let me start with a simple example in Pongo, then build up.

Say you have a user document. In the database, dates are stored as ISO strings, and the version counter is a numeric string (because JSON). But in your application, you want proper Date objects and bigints:

type UserDocStored = {
  name: string;
  createdAt: string;
  lastLogin: string;
};

type UserDoc = {
  name: string;
  createdAt: Date;
  lastLogin: Date;
};

The upcast function does the conversion:

const upcast = (doc: UserDocStored): UserDoc => ({
  name: doc.name,
  createdAt: new Date(doc.createdAt),
  lastLogin: new Date(doc.lastLogin),
});

You wire it into the collection, and every read goes through it:

const collection = pongoDb.collection<UserDoc, UserDocStored>(
  'users',
  {
    schema: { versioning: { upcast } },
  },
);

What’s in the database:

{ "name": "Alice", "createdAt": "2024-01-15T10:30:00.000Z", ... }

What you get back:

{ name: 'Alice', createdAt: Date, ... }

That’s all. new Date(str) is cheap. Running a Regex against every string in the document is not. The CPU freeze Dawid spotted came from that check running millions of times at the driver level for every field on every concurrent request. With upcasting, the conversion runs only for the fields you declared, in a plain function, no Regex.

But ther is just type mapping - the simplest case. As I wrote about in my serialisation article, the explicit mapping pattern is useful for much more than just fixing types. It’s the same pattern you need for schema versioning. It defines the stored schema and the application schema separately together with function to transform one into the other.

Let’s say business requirements changed. You now need to group user data differently: a profile object for identity data, and a timestamps object for temporal data. The V1 documents are flat. The new V2 shape is nested:

type UserDocV1 = {
  name: string;
  createdAt: string;
  lastLogin: string;
};

type UserDocV2 = {
  profile: {
    name: string;
  };
  timestamps: {
    createdAt: Date;
    lastLogin: Date;
  };
};

Ther isn’t just a type change like string-to-Date anymore. The structure itself is different. Flat fields became nested objects, and field names moved into sub-objects. And you have thousands of V1 documents already stored. You can’t migrate them all at once (or don’t want to, because it’s risky, and some consumers might still expect V1). But your application now expects V2.

Compatibility FTW

Ther is where backward and forward compatibility come in.

Backward compatibility means: old data still works. V1 documents stored months ago need to be readable by the V2 code. The upcast handles ther. It reads the document in whatever shape it has and transforms it into V2.

Forward compatibility means: new data doesn’t break old consumers. If you have another service or an older deployment that still reads the V1 format, it needs to keep working. The downcast handles ther. When storing V2 documents, it writes the V1 fields alongside the V2 fields, so older readers can still find what they expect.

Together:

type StoredPayload = UserDocV1 & UserDocV2;

const upcast = (doc: StoredPayload): UserDocV2 => ({
  profile: doc.profile ?? { name: doc.name },
  timestamps: {
    createdAt: new Date(doc.timestamps?.createdAt ?? doc.createdAt),
    lastLogin: new Date(doc.timestamps?.lastLogin ?? doc.lastLogin),
  },
});

const downcast = (doc: UserDocV2): StoredPayload => ({
  name: doc.profile.name,
  createdAt: doc.timestamps.createdAt.toISOString(),
  lastLogin: doc.timestamps.lastLogin.toISOString(),
  profile: doc.profile,
  timestamps: doc.timestamps,
});

Look at the upcast: if the nested profile or timestamps fields exist (document was written by V2 code), it uses them. If they don’t exist (for the old V1 document), it falls back to the flat fields. One function handles both old and new documents: that’s backward compatibility.

And look at the downcast: it writes name, createdAt, lastLogin as flat string fields (V1 shape) alongside profile and timestamps (V2 shape). A service still reading V1 sees the flat fields and works fine. A service reading V2 sees the nested ones. That’s forward compatibility.

You wire both into the collection:

const collection = pongoDb.collection<UserDocV2, StoredPayload>(
  'users',
  {
    schema: { versioning: { upcast, downcast } },
  },
);

From here, your application code only deals with V2. The collection handles the translation in both directions:

const v2Doc: UserDocV2 = {
  profile: { name: 'Alice' },
  timestamps: {
    createdAt: new Date('2024-01-15T10:30:00.000Z'),
    lastLogin: new Date('2024-06-20T14:45:00.000Z'),
  },
};

await collection.insertOne(v2Doc);

What’s stored (downcasted, both shapes for compatibility):

{
  "name": "Alice", 
  "createdAt": "2024-01-15T10:30:00.000Z",
  "lastLogin": "2024-06-20T14:45:00.000Z",
  "profile": { 
    "name": "Alice" 
  },
  "timestamps": { 
    "createdAt": "2024-01-15T10:30:00.000Z",
    "lastLogin": "2024-06-20T14:45:00.000Z" 
  } 
}

Then you can read it back with:

const doc = await collection.findOne({ ... });

And get upcasted to V2 data in your application code:

{
  "profile": { 
    "name": "Alice" 
  },
  "timestamps": { 
    "createdAt": "2024-01-15T10:30:00.000Z",
    "lastLogin": "2024-06-20T14:45:00.000Z" 
  } 
}

Same collection, V1 and V2 documents coexisting. insertMany, replaceOne, findOne: all go through the upcast/downcast. No batch migration needed. You roll out the new code, and old documents are handled transparently.

There’s another thing the downcast gives you: querying remains backwards-compatible. Because the downcast writes the flat V1 fields alongside the nested V2 ones, a query like collection.findOne({ name: ‘Alice’ }) still works even though V2 code doesn’t use name directly anymore. The V1 field is there in the stored document. That matters if you have queries or indexes built against the old shape. They don’t break.

Now, for events, ther matters even more. In event sourcing, stored events are immutable- the log is append-only, and you don’t modify what was already written. I wrote about versioning patterns in more detail. The core idea is: your business evolves, your code evolves, your event schemas evolve, but the events in the store stay as they were. You can’t go back and rewrite them (well, you can, but you really shouldn’t). Upcasting is how you bridge the gap.

For Emmett, the same pattern works at the event store level. You define the stored shape (what JSON gives you from the database) and the application shape (what your code works with):

type ShoppingCartOpenedFromDB = Event<
  'ShoppingCartOpened',
  { openedAt: string; loyaltyPoints: string }
>;

type ShoppingCartOpened = Event<
  'ShoppingCartOpened',
  { openedAt: Date; loyaltyPoints: bigint }
>;

And an upcast that handles each event type:

const upcast = (event: Event): ShoppingCartEventWithDatesAndBigInt => {
  switch (event.type) {
    case 'ShoppingCartOpened': {
      const e = event as ShoppingCartOpenedFromDB;
      return {
        ...e,
        data: {
          openedAt: new Date(e.data.openedAt),
          loyaltyPoints: BigInt(e.data.loyaltyPoints),
        },
      };
    }
    case 'ShoppingCartConfirmed': {
      const e = event as ShoppingCartConfirmedFromDB;
      return {
        ...e,
        data: {
          confirmedAt: new Date(e.data.confirmedAt),
          totalCents: BigInt(e.data.totalCents),
        },
      };
    }
    default:
      return event as ShoppingCartEventWithDatesAndBigInt;
  }
};

You pass it when reading a stream:

const { state } = await eventStore.aggregateStream<
  ShoppingCartState,
  ShoppingCartEventWithDatesAndBigInt
>(shoppingCartId, {
  evolve: evolveState,
  initialState,
  read: { schema: { versioning: { upcast } } },
});

Or in a command handler:

const handle = CommandHandler<ShoppingCart, ShoppingCartEvent>({
  evolve,
  initialState: () => ({ ... }),
  schema: { versioning: { upcast: upcastDatesAndBigInt } },
});

The difference with events is that you can’t update them in place. For documents, you have both directions: upcast on read, downcast on write. For events, upcasting is the main tool because the event store is append-only. Old events stay as they were written. But downcasting has its place too.

Consider ther: you have a projection or a subscriber that was built months ago against the old event schema. Maybe it’s a read model that listens to ShoppingCartOpened and expects clientId as a flat string. But your current code evolved. Now ShoppingCartOpened carries a client object with id and name:

// What old subscribers expect
type ShoppingCartOpenedV1 = Event<
  'ShoppingCartOpened',
  { clientId: string; openedAt: string }
>;

// What current code produces
type ShoppingCartOpenedV2 = Event<
  'ShoppingCartOpened',
  { client: { id: string; name: string }; openedAt: Date }
>;

Upcasting enables the current code to read older events. The ones stored with just clientId. Downcasting helps old subscribers consume new events. It transforms the new client object back into the flat clientId they expect. Same principle as with documents, but especially important here because event subscribers often live in separate services or deployments that you can’t update all at once.

And the same upcast function that started as simple type mapping string → Date, string → bigint handles ther structural change too. You just add another case to the switch:

case 'ShoppingCartOpened': {
  const e = event as ShoppingCartOpenedV1;
  return {
    ...e,
    data: {
      client: { id: e.data.clientId, name: 'Unknown' },
      openedAt: new Date(e.data.openedAt),
    },
  };
}

Old events get the client object synthesised from the flat clientId. New events already have it. The evolve function only deals with the V2 shape.

And here’s where things started to click for me. I added upcasting to fix the bigint problem: explicit type mapping instead of a Regex. But the same mechanism, without any changes, also handles structural versioning. The simple string → Date mapping from the first example is the same code path as the clientId → client migration above. It’s one function, one place, one pattern for all of it: type coercion, field restructuring, schema migration.

Right decisions stack

Right decisions stack. The Regex hack was blocking the slot where upcasting should have been all along. Once I removed it, the performance got fixed, and I got schema versioning on top. One fix created room for the next one, which created room for the next. That doesn’t happen when you keep patching around the same bad decision.

Looking back, maybe the Regex wasn’t the wrong first move. The rule is “make it work, make it right, make it pretty”, in that order. The Regex made it work. It had performance problems, but it still let me ship and learn where the real problem was. If I had tried to design the upcast/downcast system from scratch, without having lived with the Regex for a while, I might have over-engineered it or missed the connection to schema versioning entirely. The understanding came from living with the shortcut.

Dawid raised a performance issue with Pongo projections, but the same Regex was running in Emmett as well. I could have fixed it in one place and called it a day. Instead, I used it as a push to do the thing I’d been planning anyway and applied it to both Pongo and Emmett to keep things consistent. Because I already understood the problem well enough, “make it right” turned out easier than I expected.

You can recover from shortcuts. You should. But you also shouldn’t be afraid to take them in the first place, as long as you come back and do it properly.

Full changes:

• Pongo PR #149,

• Emmett PR #292.

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

Not that it’s easy to teach tradeoffs, it’s a subtle art of explanation. You need to provide enough context and be precise so others don’t treat tradeoffs as a general best practice. Because they’re usually not such, they typically come from the muddy banks of the Wishkah River.

I think that a decent way is to tell the story. Not the fairy tale, but the specific tradeoff applied in practice. That’s what I’m going to do today: tell you how I cheated on database transactions.

Dumbo

Do you know Dumbo? Of course, flying elephant, such a sweetie. It’s also a codename of my Open Source project. I didn’t tell you about it so far, as it’s a shared dependency for Pongo and Emmett, responsible for:

• connection pooling,

• safe connection lifetime management,

• handling transactions,

• migrations,

• SQL execution.

Not that small a scope for a Dumb tool, aye? Still, the intention is to make usage dumb, hiding the weirdness of a specific SQL database, so I solve it once and don’t need to be constantly distracted thinking about it, but focus on:

• How to append and process events the best way in Emmett,

• How to make the best JSON handling and translation into specific SQL dialects in Pongo.

Dumbo usage is quite simple:

import { dumbo } from '@event-driven-io/dumbo';
import { pgDumboDriver } from '@event-driven-io/dumbo/pg';

const pool = dumbo({ connectionString, driver: pgDatabaseDriver });

You need to set up a specific database driver (e.g. pg for PostgreSQL or sqlite3 for SQLite), as Dumbo now supports multiple relational databases.

Having that, you can do stuff like:

import { SQL } from '@event-driven-io/dumbo';

await pool.execute.batchCommand([
  SQL`CREATE TABLE test_users (id SERIAL PRIMARY KEY, name TEXT)`,
  SQL`INSERT INTO test_users (name) VALUES ('Alice'), ('Bob')`,
]);

And also do queries:

const count = pool.execute.query<{count: number}>(
  SQL`SELECT COUNT(*) as count 
      FROM test_users 
      WHERE ${SQL.in('id', userIds)}`,
);

It’ll handle query parameterisation, data escaping, etc.

It can also handle transactions:

const users = await pool.withTransaction(async (tx) => {
  await tx.execute.command(
    SQL`INSERT INTO test_users (name) 
        VALUES (${firstUserName}), (${secondUserName})`,
  );

  return execute.query<User>(
    SQL`SELECT *
        FROM test_users 
        WHERE ${SQL.in('id', userIds)}`,
   );
});

As simple as this looks, you should already have the question about the first tradeoff I made in your head.

Why on earth I think that’s a sane move to write my own multi-database driver?!

Well, I agree, that’s not the best move at the first glance, but let me explain to you why in MY context this actually makes more sense:

1. Yes, there are tools like Knex, Kysely, Drizzle, etc. in Node.js land that handle similar stuff. They’re nice, I really like them, I really do, but… But they are all big and are bringing a lot of their conventions, and when I’m building storage tools like Emmett and Pongo, I need to have more control. I don’t want those tools and their limitations drive my architecture decisions. I also don’t want to be surprised when the creator decides to drop working on it, or when I become a victim of a supply chain attack. Still, when I want a new kitchen table, I don’t start by going to the forest with a saw. I’m still using existing database drivers, what’s more I’m allowing people to choose the one they prefer. I just don’t want to be driven (and also my users be driven) by some other higher abstraction tool.

2. Even with that, this decision can seem like a bold “how hard can it be?!” statement. And it is, but… But I’ve built, or was co-authoring, such tools in the past. Some were proprietary, some were Open Sourced (see Weasel). And they served me well.

3. And last but not least, the last point. Well, I only had those two above. OK, I can add that I didn’t plan to make it a general-usage tool, just a small one for my needs.

So I did.

And here we’re at the moment when I had to cheat on transactions because of those decisions.

Cloudflare D1

All relational databases seem similar, but only until you start using them extensively, or until you need to write your own storage library. Then you learn stuff that not necessarily the stuff you’d like to spend time on. For instance:

• What’s the difference between databases like PostgreSQL and single-threaded databases like SQLite or DuckDB, and how concurrent processing can be surprising

• or that sqlite3 only calls the first query, but the next silently ignores,

• etc.

Still, well, abstractions like Dumbo give the possibility to massage such cases behind the scenes.

And then Cloud Databases and SaaS Databases came into play, like Cloudlfare D1.

I was motivated by the generous sponsorship from Sam Hatoum to make supporting Emmett and Pongo a top priority. Thanks Sam, appreciate that! And I made it, but I had to cheat a bit.

Databases like Cloudflare D1 and Supabase expose databases as pay-as-you-go services. They optimise deployment, making it highly scalable, so you don’t need to care about it. It’s cost-effective for start-ups and scenarios when you’re not under a huge load. If you are, then you’ll need to pay more, but they also give you autoscaling at least.

They do it by exposing the database API through HTTP API, for instance PostgREST. This gives them easier management around throughput, security, etc., as each call is made as an HTTP request through the exposed API.

Yet, that kills some options like: ekhm, transactions. The challenge with transactions is that they don’t scale (that’s why MongoDB is WebScale). They don’t scale, as you’d need to open a transaction, do some freehand operations, and then commit or rollback. That means (typically) you need to keep a connection open during that time. If you’re building SaaS, it’s a no-go, because sneaky users would open it for a few hours, do crazy stuff, and kill your SaaS resources’ utilisation.

But, boy, aren’t transactions one of the selling points of relational databases? They do, so how to proceed?

Then you need to cheat, as I did.

Repeatable reads, batches, etc.

Cloudflare D1 doesn’t provide transaction data for the reasons I outlined, but it doesn’t leave us without other tools. Those two tools to let me do the smoke and mirrors trick are:

• sessions,

• SQL batches.

What are sessions? Per Cloudflare Docs:

A session encapsulates all the queries from one logical session for your application. For example, a session may correspond to all queries coming from a particular web browser session. All queries within a session read from a database instance which is as up-to-date as your query needs it to be. Sessions API ensures sequential consistency for all queries in a session.

Essentially, that means that we’re getting repeatable reads. So when we’re starting specific sessions, they will be handled sequentially.

And now, the second ingredient: Batches. Per Cloudflare docs

Sends multiple SQL statements inside a single call to the database. This can have a huge performance impact as it reduces latency from network round trips to D1. D1 operates in auto-commit. Our implementation guarantees that each statement in the list will execute and commit, sequentially, non-concurrently.

Batched statements are SQL transactions. If a statement in the sequence fails, then an error is returned for that specific statement, and it aborts or rolls back the entire sequence.

To send batch statements, provide D1Database::batch a list of prepared statements and get the results in the same order.

So Cloudflare didn’t allow us to do the full, freehand transaction, but they allowed us to send multiple statements that internally will be executed as a SQLite Transaction. When the request is handled, it’ll open a transaction, run the statements, and return the results.

Cool, let’s mix this soup together, as having that, I decided to:

1. Fail automatically if someone tries to create a transaction on Cloudflare D1 with an error:

D1 does not support SQL transactions (BEGIN/COMMIT/ROLLBACK/SAVEPOINT). Use { mode: “session_based” } to opt-in to session+batch semantics, or use ‘connection.execute.batchCommand() for atomic multi-statement execution.

Then they get clear information on the first try.

2. Allow users to explicitly open a session-based transaction by providing mode:

const users = await pool.withTransaction(
  async (tx) => {
    await tx.execute.command(
      SQL`INSERT INTO test_users (name) 
      VALUES (${firstUserName}), (${secondUserName})`,
    );

    return tx.execute.query<User>(
      SQL`SELECT *
              FROM test_users 
              WHERE ${SQL.in('id', userIds)}`,
    );
  },  
  { mode: 'session_based' }
);

When they do it, they will need to be aware of the limitations of the tool they have. So that this will internally create a D1 session, and only handle a single batch of operations properly. We’re mimicking the sequential processing by the session-based repeatable reads capability. Still, we need to remember that we won’t be able to roll back changes across multiple statements. Only a single command or batch command is an atomic operation.

We can’t, for instance, run a batch of updates, and fail the whole batch if one update didn’t change any record. The batch will only fail if the database throws an exception. An exception can be in SQLite only called by a table constraint or trigger.

By making this choice to require explicit mode and naming it explicitly, I didn’t manage to cover all cases, but at least made it safe, so people need to learn about this non-default behaviour and be more careful about it. When designing an API, it’s usually better to start with a more strict option and do a bit of “scarification” sometimes.

Still, when I implemented that in Emmett and Pongo, I intentionally used it to enable event appends, but document operations, etc.

If you’d like to try it, you can check Emmett’s or Pongo’s beta versions.

For Pongo, you can install it with:

npm install @event-driven-io/pongo@0.17.0-beta.21

And use it as:

import { d1PongoDriver } from '@event-driven-io/pongo/cloudflare';

const client = pongoClient({
  driver: d1PongoDriver,
  database,
  transactionOptions: { mode: 'session_based' },
});

Or in Emmett by installing:

npm install @event-driven-io/emmett-sqlite@0.43.0-beta.1

And use it as:

import { getSQLiteEventStore } from '@event-driven-io/emmett-sqlite';
import { d1EventStoreDriver } from '@event-driven-io/emmett-sqlite/cloudflare';

const eventStore = getSQLiteEventStore({
  driver: d1EventStoreDriver,
  database,
});

Still, even if you don’t care about Emmett, Pongo, and my Open Source project, I hope that this will give you decent inspiration for your own tradeoffs analysis.

I hope that you learned a bit about how design APIs work, how to check the guarantees of your tools and learn to walkaround them when you have to.

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

In the last article, I explained how to rebuild Event-Driven Read Models in a safe and resilient way. I asked readers to let me know if they find any blind spots in my design.

Well, I found one myself.

This article is about that edge case, but more importantly, it’s about the rabbit hole I went down thinking about how to “fix” it. At the end of this hole, there is a nice learning about dealing with distributed systems. Sometimes the best engineering isn’t about preventing all failures. It’s about recognising your blind spots and making sure they don’t catch you off guard.

Everyone has a plan until they get punched in the mouth

Let me recap the situation described in the previous article. We’re using PostgreSQL to store events and read models. We want to rebuild an inline projection, one that’s applied in the same transaction as the appended event.

In the design from my previous article, during a rebuild, we:

1. Mark the projection as “rebuilding”.

2. Skip inline projections (the rebuild process will catch up anyway).

3. Process all historical events.

4. Mark the projection as “active” again.

The hybrid locking strategy with advisory locks and status checks ensures that inline projections know when to skip and that only one rebuild runs at a time.

Sounds solid. Here’s where it breaks:

The rebuild process checks for new events, sees none (because event 1000 is still in an uncommitted transaction), declares victory, and sets the projection to active. Meanwhile, the append transaction has already decided to skip the inline projection. When it finally commits, the event exists but was never projected. The rebuild process is done, and only new events will update the projection during the next update to the new event append.

We could handwave it with it won’t ever happen!. But, well, under load, this will happen eventually. You might not even notice it, but it will. The timing window is small, but with enough throughput, it becomes a certainty.

The Rabbit Hole of “Fixes”

My first instinct was to engineer my way out of this. Surely with enough clever coordination, we can close this gap?

Let me walk you through the rabbit hole.

Attempt 1: Wait for In-Flight Transactions

PostgreSQL provides pg_snapshot_xmin(pg_current_snapshot()) which tells us the oldest transaction that’s still running. The idea: after processing events, wait until all potentially-skipping transactions have completed before marking the projection as active.

Why it fails: We don’t know what we’re waiting for. While we wait for existing transactions to complete, new transactions start and make their own skip decisions. The target keeps moving. We can never “catch up” because new skips happen while we’re waiting for old ones to become visible.

Attempt 2: Use Transaction IDs as Boundaries

PostgreSQL assigns monotonic transaction IDs to each transaction when it starts. This seems useful—what if we record a “sealing” transaction ID when we’re ready to complete the rebuild, and use it to make decisions?

The logic would be:

When rebuild is ready to complete, record sealing_txid = current_transaction_id

Any event from a transaction with ID lower than the sealing point was “in flight” during rebuild, so it should be handled by async Events from transactions with higher IDs started after sealing, so they can safely use inline projections (the read model will be ready)

Sounds reasonable?

Here’s why it doesn’t work.

The core problem: transaction ID order ≠ commit order.

When a transaction starts, PostgreSQL assigns it the next available transaction ID. But transactions don’t commit in the order they started. A transaction that started earlier (lower ID) might take longer to complete and commit after a transaction that started later (higher ID).

Let’s trace through a concrete scenario:

Here’s what happened:

1. Tx 99 started first and got the lowest transaction ID. It inserted an event and decided to skip the inline projection (status was ‘rebuilding’). But then it got slow—maybe network latency, maybe the application did other work.

2. Tx 100 (the rebuild) started second, recorded sealing_txid = 100, and prepared to complete.

3. Tx 101 started third. It checked: “Is my transaction ID (101) >= sealing_txid (100)?” Yes, so it assumed the read model was ready and processed its inline projection. It committed successfully.

4. Tx 100 marked the projection as active and committed.

5. Tx 99 finally committed. But it had already decided to skip the projection back when status was ‘rebuilding’. That decision was made, the skip happened, and the event is now missing from the read model.

The fundamental issue: we can’t see uncommitted transactions. When Tx 100 set the sealing point, it had no way to know that Tx 99 was still out there, holding an event that would skip projection. Transaction 99 is invisible until it commits, but by then it’s too late.

You might think: “Just wait until all transactions before the sealing point have committed!” PostgreSQL even provides pg_snapshot_xmin(pg_current_snapshot()) which tells you the oldest active transaction. But this leads us back to Attempt 1—while we wait, new transactions start and make their own skip decisions. The target keeps moving.

I wrote about this exact problem in How Postgres sequences issues can impact your messaging guarantees. The same visibility challenges that affect outbox ordering apply here. Transaction IDs are useful for ordering within committed data, but they can’t help us coordinate with transactions that haven’t committed yet.

Attempt 3: Lock Appends During Transition

What if we use advisory locks more aggressively? The idea: when rebuild is ready to complete, acquire an exclusive lock that blocks the entire append path. While holding this lock, flip the status to ‘active’. No appends can be in progress during the flip, so no race condition, right?

Here’s the proposed flow:

• Rebuild finishes processing historical events

• Acquire exclusive advisory lock (all new appends must wait)

• Set status = ‘active’

• Release lock

• Waiting appends resume and sees ‘active’, processes inline normally

It should work. In theory. We’re creating a synchronisation point where nothing can slip through. But there’s a subtle problem: the skip decision was already made inside the append transaction.

An append transaction doesn’t just check the lock and status at one instant. It does several things:

1. BEGIN transaction

2. INSERT the event

3. Try to acquire a shared advisory lock

4. Check projection status

5. Decide: process inline or skip?

6. Execute that decision

7. COMMIT

The decision in step 5 happens inside the transaction. If the transaction saw ‘rebuilding’ status, it was skipped. That decision is now part of the transaction’s pending work. The transaction might be waiting to commit or doing other work, but the skip decision has already been made.

Our exclusive lock in the completion flow blocks step 3 - new transactions can’t acquire the shared lock. But what about transactions that already passed step 5 and are just waiting to commit?

The timing here is tricky. Advisory locks in PostgreSQL can be either:

• Transaction-scoped (pg_advisory_xact_lock): released automatically when transaction commits

• Session-scoped (pg_advisory_lock): held until explicitly released or session ends

If we use transaction-scoped locks for inline projections (which makes sense—you want the lock tied to the transaction lifetime), then the append transaction might have already released its shared lock by the time we try to acquire an exclusive lock. The lock protected the status check, but the transaction is still running with its skip decision already made.

Even if we could perfectly synchronise the lock acquisition, there’s another problem: advisory locks are session-scoped, not durable.

If the connection dies while holding the exclusive lock:

• The lock releases immediately (that’s how advisory locks work)

• We might have partially updated the status

• The system is now in an unknown state

• Other transactions resume with potentially inconsistent data

We can’t build reliable coordination on something that vanishes when connections fail. This is exactly why the original article combined advisory locks with persistent status checks—but that combination doesn’t solve this particular race condition.

TLDR on Attempts

Every “fix” follows the same pattern:

1. Identify a coordination point.

2. Discover there’s a window before that point we can’t see.

3. Try to close that window.

4. Create a new window somewhere else.

5. Repeat.

We’re not solving the problem. We’re relocating it.

So is it a bug in the initial design? Depending how you look on that. I see it as an example of fundamental property of concurrent systems. PostgreSQL’s isolation guarantees mean uncommitted transactions are invisible to other transactions. That’s a feature, not a bug - but it means there’s always a window we can’t see into.

Stop Fighting, Start Tracking

After chasing my tail through various “solutions,” I stepped back and asked a different question.

Instead of: “How do we prevent events from being skipped?” (which requires blocking appends or seeing into uncommitted transactions—both unacceptable because of performance, guarantees etc.)

I started to think on: “How do we know when an event was skipped?” and “How do we ensure skipped events get processed eventually?”

Both of these are solvable.

If we can’t prevent skips from happening, let’s make them visible. When an inline projection skips, it could record that it skipped in the same transaction as the event append.

If the append transaction rolls back, the skip record also rolls back (there’s no event to worry about). If it commits, we have a durable record of exactly what was skipped.

In Emmett, I’m using a dedicated emt_system_messages table rather than reusing the regular emt_messages table or creating a simple “skipped events” table. This might seem like over-engineering—why not just create a simple table with projection ID and event position? Or why not just reuse the existing messages table?

Why a dedicated system messages table?

The regular emt_messages table is for business events—the actual domain events that drive your application. Mixing system-level concerns (like “this projection skipped this event during rebuild”) with business events pollutes the event log and makes it harder to reason about. A simple “skipped_events” table with just (projection_id, event_position) could be enough, but in the long term, it may be hard to maintain. As systems evolve, we’ll have more cases where we might want to skip events. Plus, having system events recorded with all metadata gives us full observability of the internals of our system. We could even make tables for projections and processors’ status, built as read models from system events! Still, let’s hold our horses and get back to our use case.

The system messages table could mirror the structure of regular messages with:

• Global position sequencing (with all the transaction visibility handling from the outbox ordering article)

• Transaction ID tracking for proper visibility checks

• Archiving support via is_archived flag

• Partitioning for performance

If we created a throwaway “skipped events” table, we’d need to solve all these problems again. We’d essentially be building a second event log with the same guarantees.

The skip could be stored as a system message where:

• Stream ID = the projection identifier (name + version), so we can query all skips for a specific projection

• Message data = reference to the original event (sequence ID from the event log)

• Message metadata = processor ID, reason for skip, timestamp

• Message type = indicates this is a “skip during rebuild” system event

Won’t this table grow too much? It may, but we can archive skip events when they’re no longer needed.

When the rebuild processor handles a skipped event, we could archive it. In Emmett, this means setting is_archived = true on the record. The table is partitioned by this flag. Archived records automatically move to a separate partition, which can even be on a different disk drive.

Why archive instead of delete?

• Audit trail: You can see what was skipped and when it was processed

• Debugging: If something goes wrong, you have history to investigate

• Idempotency: If a processor crashes and restarts, it won’t reprocess already - archived skips

Later, we could define retention policies and clean up old archived records. We could do it more aggressively than for business events, since these are operational records, not business history. You might keep business events forever, but archived skip records only need to stick around long enough for debugging (days or weeks, not years).

The same processor that handles regular events for the projection could also process its skipped records. When it reads an event from the main event log and applies the projection, it also checks for and archives any corresponding skip record for that event position. This keeps everything in sync.

As mentioned, this also opens the door for other system events in the future - not just rebuild skips, but potentially failed projections, poison messages, or audit records. The infrastructure would already be in place, separated from the business event log.

The Final Flow

Let me walk through how this works in practice, bringing together all the pieces.

During rebuild

The projection status is ‘rebuilding’. The async rebuild processor works through historical events from the beginning. Meanwhile, normal operations continue. Events are appended, and inline projections check the status. When they see ‘rebuilding’, they skip the projection but record a skip message in emt_system_messages within the same transaction.

When rebuild catches up

The rebuild processor eventually reaches the current position where no new events are visible. At this point, it sets the status to ‘active’. From now on, new appends will process their inline projections normally.

But what about those skip records? The events they reference exist, but their projections were never applied.

Draining the skipped events

The rebuild processor (or a dedicated processor, or a manual trigger—your choice) now queries emt_system_messages for skip records belonging to this projection. Using the same transaction visibility rules from the outbox pattern (transaction_id < pg_snapshot_xmin), it only sees skip records from committed transactions.

For each skip record:

1. Find the referenced event in the event log (or use its data if stored in skipped event)

2. Apply the projection for that event

3. Archive the skip record (set is_archived = true)

The skip record and the event are committed in the same transaction. This gives us a simple invariant: if an event exists, either its projection was applied inline (status was ‘active’) or a skip record exists (status was ‘rebuilding’). There’s no third option where an event exists without a trace.

The drain process might find new skip records appearing. Transactions that were in flight during the status change, committed after we started draining. That’s fine. We keep querying until no more visible skip records exist. They will stop appearing as we alread stopped rebuilding processor, so no more inline projections should be skipped.

The “drain skipped events” phase can be automatically triggered by finishing projection rebuild. It could be handled as the 2nd phase of rebuild processor, or triggering a dedicated one. It could be also just handled by a human initiating the drain.

This flexibility lets user choose the approach based on the specific operational requirements.

The Dead Letter Queue Pattern

What we’ve built is essentially a Dead Letter Queue (DLQ)—a place where messages that couldn’t be processed normally are stored for later handling.

This pattern exists in every serious messaging system:

• Apache Kafka: Dead Letter Topics for messages that fail consumer processing

• RabbitMQ: Dead letter exchanges for rejected or expired messages

• AWS SQS: Redrive policies that move messages to a DLQ after N failures

• Azure Service Bus: Built-in dead-letter sub-queues for each queue

The pattern is universal because all messaging systems face the same fundamental problem: sometimes messages can’t be processed immediately, and you need a place to store them without blocking the rest of the system.

It’s a topic in its own right, as it’s not perfect and should be used cautiously. Many teams fail to apply them correctly. The DLQ becomes like a car alarm in a parking lot, technically signalling a problem, practically ignored by everyone. And that’s what I see in many systems: teams set up DLQs and do nothing about them.

It starts innocently. You configure the DLQ “just in case.” A few messages end up there during a deployment. Someone says, “We’ll look at it later.”

More messages accumulate. The DLQ becomes background noise—a number on a dashboard that nobody checks. Eventually, something critical lands there, and nobody notices until a customer complains.

That’s why in the discussed design, skip records aren’t meant to accumulate indefinitely. The rebuild processor drains them during completion.

Retention policies clean up archived records after a reasonable period. If skip records exist for too long, that’s a signal that something is wrong with the rebuild process - and users should know about it.

A DLQ is only helpful if it’s monitored, processed, and understood why messages end up there. Otherwise, it’s just a fancy way to lose data slowly rather than immediately.

The Broader Lesson

This article isn’t really about PostgreSQL advisory locks or projection rebuilding. It’s about how we approach problems in distributed systems.

When we find a race condition, the instinct is to fix it. Add a lock. Add a check. Add a coordination phase. But each “fix” often just moves the race condition somewhere else. We went through three attempts—waiting for transactions, using transaction ID boundaries, locking appends—and each one failed for a different reason. We weren’t solving the problem; we were relocating it.

At some point, I had to ask myself: am I making this system more reliable, or just more complicated?

The answer came when I changed the question. Instead of asking “how do I prevent events from being skipped?” I asked, “How do I know when an event was skipped, and how do I make sure it gets processed eventually?”

The first question has no good answer, not without blocking appends, which defeats the purpose. The second question is straightforward: record the skip in the same transaction as the event, and process it later.

A system isn’t trustworthy because it never fails. That’s impossible for anything sufficiently complex. A system is trustworthy because you know when it can fail, how it will fail, and how to recover. The skip tracking approach doesn’t prevent failures during the transition period. It makes them visible and recoverable. That’s a stronger guarantee than complex coordination machinery with hidden edge cases.

Sometimes the best engineering decision is to accept what you can’t control and focus on what you can. We can’t control PostgreSQL’s transaction visibility rules. We can’t see into uncommitted transactions. We can’t make the append decision and the rebuild completion atomic without stopping the world.

What we can control is recording skipped items, ensuring those skips are processed, and making the whole thing observable. That’s enough.

The blind spot I found wasn’t really about the specific race condition. It was a reminder that distributed systems have fundamental constraints we can’t engineer around, at least not without trade-offs worse than the original problem.

Transaction isolation and concurrent systems mean we can’t have zero-downtime rebuilds with perfect inline projection consistency and no coordination overhead, all at the same time.

Something has to give.

What we get instead is explicit tracking of what was skipped, guaranteed eventual processing via the system messages table, observability into the transition period, and crash recovery that doesn’t lose data. The read model might be briefly inconsistent during the transition, but we know exactly what’s missing, and we have a clear path to fix it.

That’s the kind of system I can reason about and trust.

If you’re dealing with similar challenges, I’m happy to help through consulting or mentoring. You can also join the discussion in our Emmett Discord—we have a nice community working through these exact problems.

Or check also other related resources:

• Rebuilding Event-Driven Read Models in a safe and resilient way

• How Postgres sequences issues can impact your messaging guarantees

• Guide to Projections and Read Models in Event-Driven Architecture,

• Distributed Locking: A Practical Guide,

• Consumers, projectors, reactors and all that messaging jazz in Emmett,

• How to scale projections in the event-driven systems?,

• Checkpointing the message processing,

• Let’s talk about positions in event stores.

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

Let’s make a soup today: a blog soup. We’ll mix multiple ingredients like:

• events (obviously),

• read models,

• inline and async projections,

• rebuilding read models,

• backfilling new ones with data from existing events,

• scaling async processing horizontally,

• distributed locking,

• PostgreSQL and its Advisory Locks.

Sounds a lot? Well, the soup should be nutritious.

In an event-driven way, after handling business logic, we record new facts and call them events. They gather information about what has happened. That brings many benefits, such as business observability by keeping a log of them. Especially if we’re doing Event Sourcing, we can make the next decision based on them.

Typically, we’re using events in two ways:

• reacting to them, triggering and integrating the steps of our business workflow,

• projecting them, and getting the flattened interpretation of our system state inside read models.

Such processing can happen asynchronously, but doesn’t have to. If we’re using a transactional database (like PostgreSQL, SQLite, or even MongoDB), we can update our read models in the same atomic transaction that stores new events. Such a process is typically called inline projection. Event stores like Emmett and Marten allows that. Still, you can do the same if you’re using outbox pattern, then you can achieve the same without Event Sourcing.

Inline processing is tempting because we’re getting immediate consistency. Yet, there’s no free lunch here. We’re slowing down our event append as we need to process more, and our transactions will be open longer, which can cause deadlocks, etc. We also can’t take advantage of batching event processing.

Also, as Bastian Waidelich rightfully pointed out to me, using inline projections also increases the coupling and fragility of our business logic. If the inline projection fails (due to a bug in its projection logic, a database constraint or other random issue), then we won’t be able to append our event, which is counterintuitive, as why would read model block our business logic (e.g. confirming shopping cart).

My thumb rule is that for single stream, simple projections, I prefer inline projections, but for more complex or workflow processing, I’d go with async.

The big benefit of a durable event log is that we can correct past mistakes and gain more insights from existing data.

How does it look in practice? Let’s say we initially had a basic read model that showed a summary of the specific shopping cart. In TypeScript this could look as follows:

type ShoppingCartSummary = {
  _id?: string;
  productItemsCount: number;
  totalAmount: number;
};

Besides the data, we also need a method that represents how we apply events on top of the existing state to get the next, evolved state.

const evolve = (
  document: ShoppingCartSummary | null,
  event: ProductItemAdded | ProductItemRemoved,
): ShoppingCartSummary => {
  document = document ?? { totalAmount: 0, productItemsCount: 0 };

  switch (type) {
    case 'ProductItemAdded':
      return withAdjustedTotals({
        document,
        productItem: event.data.productItem,
        by: 'adding',
      });
    case 'ProductItemRemoved':
      return withAdjustedTotals({
        document,
        productItem: event.data.productItem,
        by: 'removing',
      });
  }
};

const withAdjustedTotals = (options: {
  document: ShoppingCartSummary;
  productItem: PricedProductItem;
  by: 'adding' | 'removing';
}) => {
  const { document, productItem, by } = options;
  const plusOrMinus = by === 'adding' ? 1 : -1;

  return {
    ...document,
    totalAmount:
      document.totalAmount +
      productItem.unitPrice * productItem.quantity * plusOrMinus,
    productItemsCount:
      document.productItemsCount + productItem.quantity * plusOrMinus,
  };
};

Sounds fine, but well, it may appear that either we, implementing it, or our business, through requirements, forgot about some requirements. Or we didn’t forget anything, but just requirements evolved as they tend to always do.

What if we need to handle now also:

• Show the shopping cart status and show whether the shopping cart is open, confirmed or cancelled.

• not only show totals, but also a list of product items with their details.

Our new read model would look like:

type ShoppingCartSummary = {
  _id?: string;
  productItemsCount: number;
  totalAmount: number;
  status: 'Opened' | 'Confirmed' | 'Cancelled';
  productItems: ProductItem[];
};

type ProductItem = {
  productId: string;
  name: string;
  quantity: number;
  unitPrice: number;
}

And here we should ask ourselves the following questions:

• Is it really the same read model or another one that’ll be used in another place in our UI? Maybe the initial just shows basic data in the menu bar, and this will be used as the summary before confirmation?

• If the read model is the same, then are you fine with downtime where you clean the old data, and reprocess events?

• If it’s the new read model, do we need to backfill it with the old data?

There’s no best practice here; we need to do the drill and be prepared for multiple options.

Add new vs update existing.

In our case, my initial guess would be that this should be a new read model. We need to add significantly more data and new event handling for confirmation and cancellation.

You could say that:

Why add a new read model with similar data? Can’t we just do a subquery?

Of course, you can. I wrote about that more in How to create projections of events for nested object structures?. This can make sense if those read models will always evolve together.

If you need to run multiple views on the same read models, you’re increasing coupling. As such, when you’re rebuilding the read model, then potential downtime will impact both. Also, each time you adjust one view, ensure you haven’t broken the others.

You’re getting a smaller storage size, and potentially don’t need to remember multiple read models, but are doing it just one.

My experience shows that optimising for the end storage until we check that it’s too big isn’t a good driver. Nowadays, storage is cheap, so my default is to keep read models separated, and also not reuse the same evolve logic. A bit of code duplication won’t harm us, but we’ll see benefits as our models evolve. We’ll decrease the cognitive load.

Still, if those models are indeed the same, or we bet they’ll constantly evolve together, then it could be fine to reuse them.

In place update, blue/green rebuild and backfilling data

Ok, I was already using rebuilding word multiple times. But how do we actually do it?

If you’ve read articles on checkpointing or positions in event stores, you already know that each event in the event store/outbox can have its unique, monotonic position. We can subscribe to notifications about new events and process them one by one. That’s how consumers and processors work in Emmett. Once we process a specific event, we can store the checkpoint in the end storage. This enables resilient failover when our processor dies for some reason. When we restart, we’ll read the last processed checkpoint first and start listening for events from that point.

We could reuse this not only for failover but also for rebuilds. Instead of starting processing from the last known checkpoint, we could reset the checkpoint in our database to a specific point (e.g., the beginning of the log). Then we’ll get recorded events again.

And this is the moment we need to decide whether to do an in-place update or a blue/green rebuild.

In-place means that we’re the same storage target. We need to truncate it and then apply data from scratch, typically. The downside is that when we clear it, our read models won’t be up to date with the current state of our event store. We already have newer events recorded, but our read models are empty. We need to fill them in.

The same happens when we create a new read model based on the existing events. When we create a new read model, it won’t magically contain data from existing events.

Typically, we need to spin up a background worker (e.g., Emmett consumer with a projector plugged in) that pulls all events since the beginning and runs projection logic. Obviously, that means we need to run it asynchronously, and depending on the data size, we’ll need to wait until it catches up. During that time, querying for the data will return outdated information - eventual consistency in practice.

That’s why there’s another option. Instead of truncating existing data, we can keep it as it is, or even keep updating it in the old form. Having that, we can build a new read model in parallel. This read model could have a different name, or just a suffix, e.g. V2, V2.0.1, whatever we find helpful.

Then, once this new read model catches up (so it has processed all events, or is close enough to the latest event with a defined threshold), we can switch queries from the old to the new read model storage.

This is actually the preferred way, but it’s a bit more challenging when it comes to dynamically switching the query target. If you’re using Pongo, that’s not that hard, since you just switch the text-based collection name, which is just another table. But if you’re using an ORM, adding a new table dynamically and mapping it can be much more challenging.

Concurrency issues while (re)building read models

To make it harder, eventual consistency is not the only challenge we’ll face. We also need to deal with concurrency and parallel processing.

What should we do when a new event is appended while we’re rebuilding/backfilling the inline projection? If we try to update the end storage in the middle of the processing, we can end up with an inconsistent or erroneous state.

Another issue is what to do if we’re in a Kubernetes-like setup and don’t have full control over the number of instances of the same service? Or what happens if we accidentally spin up multiple rebuild workers?

Then we’re doomed, or at least the consistency of our read model data.

How do we solve it? Let me explain my plan for Emmett.

Distributed Locking and PostgreSQL advisory locks

I encourage you to check my other article: Distributed Locking: A Practical Guide. Yet, don’t worry, I won’t leave you with Read-The-Fucking-Manual type of answer.

Distributed locks are a fundamental tool for coordinating concurrency across systems. We can use a central place, typically scalable on its own, that’ll be used in multiple instances of our service, to ensure that exactly one can request a lock and run specific code.

In the mentioned article, I described multiple options and popular tools for handling that (e.g., Redis, Zookeeper, Kubernetes Replica Sets), as well as PostgreSQL and its Advisory Locks. Let me focus today on the last one and describe how I want to use it in Emmett’s PostgreSQL projection handling.

PostgreSQL gives us two options for coordination.

Row-level locks lock individual table rows. You do SELECT … FOR UPDATE, and anyone else trying to modify that row waits. The lock is tied to the specific row in a table.

BEGIN;
SELECT * FROM some_table WHERE id = @key FOR UPDATE;
-- make changes
COMMIT;

Such locks are straightforward, as we explicitly state what we want to lock. We could define the following table for projections:

CREATE TABLE IF NOT EXISTS emt_projections(
      version                       INT                    NOT NULL DEFAULT 1,  
      type                          VARCHAR(1)             NOT NULL,
      name                          TEXT                   NOT NULL,
      status                        TEXT                   NOT NULL,  
      PRIMARY KEY (name, version)
);

The background worker could lock the projection by a specific name and version and set the status to “rebuilding”. Then, during handling the inline projection, we could use the same lock and check whether the status is “active”; if not, skip processing. Using a lock-in inline projection would also prevent the rebuilding process from starting, as they wouldn’t acquire the lock.

And that’s a nuke option, as it would work but could create a performance problem. If every inline projection needs to grab a row lock on a coordination row, they will all be processed sequentially, one at a time. That’s a throughput killer when you’re appending thousands of events per second. Each of these append would require access to the lock for the specific projection type (e.g. our shopping cart summary), making not only updates but also event appends sequential. That’s not acceptable for most cases.

Advisory locks are different. They’re locks on arbitrary integers. PostgreSQL doesn’t care what the integer means—it just manages who holds the lock. No table rows involved. It also allows two modes: exclusive or shared.

-- Shared lock: many can hold simultaneously
SELECT pg_try_advisory_xact_lock_shared(12345);

-- Exclusive lock: only one holder, blocks shared locks
SELECT pg_advisory_lock(12345);

Those locks are either scoped to an open connection session or an opened transaction (with xact with name) and released automatically once they end.

Shared locks allow multiple sessions to access the lock with the same value. Exclusive lock blocks both other exclusive locks and new shared locks.

This allows a design where shared locks are used for readers, and exclusive locks for writers and maps directly to our problem:

• Inline projections take shared locks as they run concurrently, and just need to check if there’s no async job updating these projections

• Rebuilds take exclusive locks - they block inlines and other instances of async processing.

We just need to do one more thing: since advisory locks take integers, we need to map our projection name and version to them. We can do it by a consistent hash algorithm, either in the application code or in PostgreSQL.

PostgreSQL provides a built-in MD5 hash function. It’s not perfect, as it’s not a sophisticated hash, but it’s fast enough and predictable. In our case, we won’t have thousands of projections in our application, so the risk of a hash collision is negligible. If you’re still worried it’s too high, we could store id in our projections table and use it instead of hash-mapping. Still, if we used md5 function, it could look as follows:

// shared for inline projections
SELECT pg_try_advisory_xact_lock_shared(
        ('x' || substr(md5(?), 1, 16))::bit(64)::bigint
) AS acquired;

// exclusive for inline projections
SELECT pg_try_advisory_xact_lock(
        ('x' || substr(md5(?), 1, 16))::bit(64)::bigint
) AS acquired;

Where as query param, we’d pass the joined projection name and its version.

Thanks to that, multiple inline projections can access the lock if it’s not held exclusively by the async (re)building worker. Thanks to that, we’re not blocking event appends because of the lock on the inline projections.

An exclusive lock can be held only when there’s no single inline projection being applied at the moment.

What’s also cool is that advisory locks can be used with two strategies: fail fast when the lock is held, or wait for the lock to be released. The first option would be useful for inline projections, and the second for rebuilds.

Is that all? Well, not quite.

Why advisory locks alone aren’t enough

Advisory locks have a gap: they’re session-scoped. If the connection holding the lock dies, the lock releases automatically. For most cases, that’s fine and expected, but think through this scenario:

1. We have 1000 events in our event store.

2. Rebuild starts, acquires exclusive lock.

3. It truncates the projection and processes events 1-500.

4. Connection dies (network blip, out of memory kill, whatever).

5. Lock releases automatically.

6. Projection contains data only for events 1-500.

7. Inlines see no lock, start processing.

8. Inline applies event 1001, ending up with a potentially corrupted state (as there may be some events between 500 and 1001 that would impact the state).

Advisory locks can’t persist across connection failures. We need to add something that does.

The hybrid-locking approach

Emmett already maintains tables for tracking processors and projections. The relevant ones:

CREATE TABLE IF NOT EXISTS emt_projections(
    version         INT         NOT NULL DEFAULT 1,  
    type            VARCHAR(1)  NOT NULL,
    name            TEXT        NOT NULL,
    partition       TEXT        NOT NULL DEFAULT 'emt:default',
    kind            TEXT        NOT NULL, 
    status          TEXT        NOT NULL, 
    definition      JSONB       NOT NULL DEFAULT '{}'::jsonb, 
    PRIMARY KEY (name, partition, version)
) PARTITION BY LIST (partition);

CREATE TABLE IF NOT EXISTS emt_processors(
    last_processed_transaction_id XID8    NOT NULL,
    version                       INT     NOT NULL DEFAULT 1,
    processor_id                  TEXT    NOT NULL,
    partition                     TEXT    NOT NULL DEFAULT 'emt:default',
    status                        TEXT    NOT NULL DEFAULT 'stopped', 
    last_processed_checkpoint     TEXT    NOT NULL,    
    processor_instance_id         TEXT    DEFAULT 'emt:unknown',
    PRIMARY KEY (processor_id, partition, version)
) PARTITION BY LIST (partition);

The status column in the projections table can do what advisory locks can’t: persist status between connection crashes. Even if the connection dies, status = ‘rebuilding’ stays in the table. Inlines could check this and skip processing.

The processor’s table tracks checkpoint progress. When a rebuild runs, it updates last_processed_checkpoint as it goes. If it crashes and restarts, it can resume from where it left off rather than starting over.

Potentially, we could reuse the processor’s table, but having a dedicated projection table could also be useful for diagnostics (e.g., storing the projection definition) and for switching queries when a new projection version catches up. We could select the highest active projection version.

Having that, the query checking if the inline projection should be processed or skipped could look as follows:

WITH lock_check AS (
    SELECT pg_try_advisory_xact_lock_shared(
        ('x' || substr(md5($1), 1, 16))::bit(64)::bigint
    ) AS acquired
),
status_check AS (
    SELECT status = 'active' AS is_active
    FROM emt_projections
    WHERE partition = $2 AND name = $3 AND version = $4
)
SELECT
    COALESCE((SELECT acquired FROM lock_check), false) AS acquired,
    COALESCE((SELECT is_active FROM status_check), true) AS is_active;

Two checks, both must pass:

1. Can we get a shared advisory lock? Fails if a rebuild holds exclusive.

2. Is the projection status ‘active’? Fails if it’s ‘rebuilding’.

Then for async (re)building worker:

WITH lock_check AS (
    SELECT pg_try_advisory_lock(
        ('x' || substr(md5($1), 1, 16))::bit(64)::bigint
    ) AS acquired
),
ownership_check AS (
    INSERT INTO emt_processors (
        processor_id,
        partition,
        version,
        processor_instance_id,
        status,
        last_processed_checkpoint,
        last_processed_transaction_id
    )
    VALUES ($2, $3, $4, $5, 'running', '0', '0'::xid8)
    ON CONFLICT (processor_id, partition, version) DO UPDATE
    SET processor_instance_id = $5,
        status = 'running'
    WHERE 
       -- We already own it
       emt_processors.processor_instance_id = $5 
       -- Unclaimed
       OR emt_processors.processor_instance_id = 'emt:unknown' 
       -- Previous instance finished or crashed
       OR emt_processors.status = 'stopped
    RETURNING last_processed_checkpoint
)
SELECT
    COALESCE((SELECT acquired FROM lock_check), false) AS acquired,
    (SELECT last_processed_checkpoint FROM ownership_check) AS checkpoint;

Together Advisory locks prevent the race at the transition point where the connection was closed, and the rebuilding job is restarting. The status check handles crash recovery

The rebuild acquires the exclusive lock before updating the status. It waits for in-flight inlines (which hold shared locks) to finish.

If the checkpoint is null, the UPDATE matched no rows, then another instance owns the processor and is actively running. Back off.

If both succeed, you own the lock and the processor. The checkpoint tells you where to resume from (with a fallback to the beginning). Only then does it flip the status and start async processing.

If the rebuild crashes, the lock is released, but the status remains ‘rebuilding’. Inlines check status and skip.

The scenarios walkthrough

As a single good image speaks more than thousands of words, let me give you some diagrams to summarise how it works.

1. During inline projection while event appends:

• Each inline projection grabs a shared lock before applying,

• Multiple inlines can run concurrently (shared locks are compatible),

• Projection updates happen normally.

2. When a rebuild starts:

• Rebuild grabs an exclusive lock (waits for any in-flight inlines to finish),

• Marks projection as “rebuilding”,

• New inlines see the lock or status and skip,

• Rebuild processes all historical events,

• Marks the projection as “active” and releases the lock,

• Inlines resume.

3. If rebuild crashes:

• Lock releases automatically

• Status stays “rebuilding”

• Inlines keep skipping until another rebuild completes

4. Multiple async processors:

• Each processor tries to acquire the exclusive lock,

• First one wins, others wait or skip,

• Guarantees that only one processor handles a projection at a time.

5. When you add a new projection:

• No status row exists yet,

• Inlines skip automatically,

• First rebuild creates the row and backfills data.

TLDR

Just like making delicious soup, designing robust, fault-tolerant and performant distributed systems is not that easy. Building an event store is not that hard. But only if we exclude the async processing part.

I hope this walkthrough covers both the conceptual and practical aspects of handling projection rebuilds.

We used PostgreSQL and Advisory Locks, as PostgreSQL is cool and is a driving force in Emmett. But all the same principles apply to other tools and storage (with their specifics).

I explained why advisory locks and status columns complement each other:

• Advisory locks handle the fast path (in-memory, no disk I/O for normal operations) and prevent races at transition points (rebuild can’t start until in-flight inlines finish)

• Status column handles crash recovery (persists across connection failures) and new projection bootstrapping (inlines skip until first rebuild completes)

Neither alone is sufficient. Together, they provide the guarantees we need without external infrastructure. Just PostgreSQL doing what PostgreSQL does.

The cost on the hot path is microseconds: one in-memory lock check, one indexed read on a tiny cached table. For most systems, that’s an acceptable tradeoff.

I hope this article also shows you how to use distributed locking in practice.

Please tell me your thoughts and concerns, especially if you see any blind spots in this design! You can do that in our Emmett Discord, come on in, we have a nice community!

If you’re dealing with such issues, I’m happy to help you through consulting or mentoring. Contact me and we’ll find a way to unblock you!

Or check also other related resources:

• Emmett’s Pull Request implementing described approach

• Guide to Projections and Read Models in Event-Driven Architecture,

• Distributed Locking: A Practical Guide,

• Consumers, projectors, reactors and all that messaging jazz in Emmett,

• How to scale projections in the event-driven systems?,

• Checkpointing the message processing,

• Let’s talk about positions in event stores.

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.