This time, I discussed how to setup and scale message processing when you have multi-tenant environment or other dynamic setup. You'll see considerations where to scale horizontally, where vertically and why that matters to you
Really solid walkthrough of the tenant isolation tradeoffs. The consistent hashing explainer tied to Kafka's partitioning model is super helpful for teams trying to wrap thier heads around dynamic distribution. Hit a similar problem last year where we tried spinning up per-tenant workers and coordination overhead killed us at around 300 tenants. The parttionBy approach with internal worker threads sounds way more practical than jumping straight to Raft/Paxos. Curious how checkpointing behaves when a thread crashes mid-batch tho.
Really solid walkthrough of the tenant isolation tradeoffs. The consistent hashing explainer tied to Kafka's partitioning model is super helpful for teams trying to wrap thier heads around dynamic distribution. Hit a similar problem last year where we tried spinning up per-tenant workers and coordination overhead killed us at around 300 tenants. The parttionBy approach with internal worker threads sounds way more practical than jumping straight to Raft/Paxos. Curious how checkpointing behaves when a thread crashes mid-batch tho.