I’m always saying that there’s a thin line between good and bad practice, and this thin line is named “Context”. That’s also true for the (anti-)pattern I’m calling “Requeuing Roulette”.What’s the Requeuing Roulette? As the name suggests, this technique involves putting a message back into the queue. It also (correctly) suggests that we’re hoping for the best. And sometimes we may be lucky to be true. Let's discuss pros and cons of it!
The concept of causal corelation really helped clarify when parallelism is safe versus risky. Your bank acount examples made it click for me how withdrawals and deposits need ordering but cross-account operations don't. I've seen teams burn hours debugging issues that basically boiled down to unintentional requeuing roulette without realizng it. The insight about CPU getting stuck processing the same 10 failing messages while thousands sit behind is such a practical concern that often gets overlooked when focusing purely on the ordering problem.
> The requeueing roulette is seductive because it promises something impossible: maintaining strict order in a distributed, concurrent system without sacrificing throughput. It’s trying to cheat the fundamental trade-offs of distributed systems.
Well said! I've got same problem in one of my current projects and still tried to overcome these fundamentals (CAP theorem). But the thing is that even if you adhere these rules you can still have problems with consistency
The concept of causal corelation really helped clarify when parallelism is safe versus risky. Your bank acount examples made it click for me how withdrawals and deposits need ordering but cross-account operations don't. I've seen teams burn hours debugging issues that basically boiled down to unintentional requeuing roulette without realizng it. The insight about CPU getting stuck processing the same 10 failing messages while thousands sit behind is such a practical concern that often gets overlooked when focusing purely on the ordering problem.
Great post as always! I like reading your takes on things I thought I’m familiar with, never fail to learn something new 🫡
> The requeueing roulette is seductive because it promises something impossible: maintaining strict order in a distributed, concurrent system without sacrificing throughput. It’s trying to cheat the fundamental trade-offs of distributed systems.
Well said! I've got same problem in one of my current projects and still tried to overcome these fundamentals (CAP theorem). But the thing is that even if you adhere these rules you can still have problems with consistency