My events came out of order! What should I do? Are you familiar with the term "phantom record" and its benefits? No? Let me explain it to you today. Let's discuss how to embrace the chaos and learn to deal with it.
Do you store the intermediate state in the storage or in the application's memory only?
What if the app restarts in the middle of processing the payment and some information that already have been asked will disappear (since we acked the message in the queue) and the other ones will just arrive after that?
Hey Hubert, it depends on the exact use case, but my preference would be to use durable storage, then I can have proper guarantees, and use e.g. optimistic concurrency to double check that somehow there wasn't another process trying to update the same document. That shouldn't happen, but in a distributed system like Kubernetes, with multiple consumer groups, it could happen due to misconfiguration.
If I were using Event Sourcing and putting external events first to the event stream in the local event store, then I could be building this in memory from all stream events. Having that, I could subscribe to such a stream and store new events in the new, internal stream when reaching some condition.
I'd consider storing it in memory also for some performance optimisations, together with durable storage.
Hey!
Do you store the intermediate state in the storage or in the application's memory only?
What if the app restarts in the middle of processing the payment and some information that already have been asked will disappear (since we acked the message in the queue) and the other ones will just arrive after that?
Hey Hubert, it depends on the exact use case, but my preference would be to use durable storage, then I can have proper guarantees, and use e.g. optimistic concurrency to double check that somehow there wasn't another process trying to update the same document. That shouldn't happen, but in a distributed system like Kubernetes, with multiple consumer groups, it could happen due to misconfiguration.
If I were using Event Sourcing and putting external events first to the event stream in the local event store, then I could be building this in memory from all stream events. Having that, I could subscribe to such a stream and store new events in the new, internal stream when reaching some condition.
I'd consider storing it in memory also for some performance optimisations, together with durable storage.
Thank you Oskar!
Happy to help! 🫡