Queues & Streams
Learn when to decouple producers from consumers with a queue or a stream — and how that choice changes delivery guarantees, ordering, and how many consumers can read the same event.
Problem it solves
Some work shouldn't happen synchronously inside a request (sending an email, resizing an image, updating a search index). Putting that work on a queue or stream lets the request return immediately, absorbs traffic spikes, and lets slow or failing consumers retry independently — without cascading failure back to the caller.
When not to use it
- The caller genuinely needs the result before responding (e.g. an auth check) — that's a synchronous call, not an async one.
- Ultra-low-latency, small-scale work where the operational overhead of running a broker outweighs the benefit.
- Strict cross-record transactional guarantees that a message broker can't provide as cleanly as a database transaction.
Failure modes to call out
- Poison messages: a malformed message crashes every consumer that tries to process it, blocking the whole queue — needs a dead-letter queue.
- Assuming exactly-once delivery when the broker only guarantees at-least-once — consumers must be idempotent.
- Unbounded queue growth when consumers fall behind producers, with no backpressure or alerting.
- Losing ordering guarantees by scaling out consumers on a queue where relative order matters (use partition keys/streams instead).
Alternatives & trade-offs
| Alternative | Trade-off |
|---|---|
| Message queue (SQS, RabbitMQ) | One active consumer handles each delivery, and it's removed once acknowledged — great for task distribution, but redelivery after a timeout/failure means a handler can run more than once, so it must be idempotent. Fan-out to a second consumer type is possible (publish to separate queues), but each queue only gets its own copy — there's no independent replay of one shared log. |
| Event stream / log (Kafka, Kinesis) | Events are retained and multiple independent consumer groups can each read the full stream at their own pace — better for fan-out and replay, but consumers must track their own offset and handle at-least-once delivery. |
| Direct synchronous call | Simplest to reason about, but couples the caller's latency and availability to the callee's. |
| Fire-and-forget HTTP callback | Avoids a broker, but loses retries, backpressure handling, and ordering guarantees that a real queue/stream gives you. |
Diagram
One producer, two independent consumer groups — only a stream supports both reading the full history at their own pace.
Scroll the diagrams sideways to see all of them.
Estimation example
A photo-upload service handling 200 uploads/sec, where each upload needs thumbnailing, a virus scan, and a search-index update: putting that fan-out on a stream lets three independent consumer groups process the same upload event at their own pace, instead of the upload request blocking on all three synchronously.
Say this out loud in the interview
- "Does this work need to block the caller's response, or can it happen asynchronously?"
- "Do I need one consumer per message (queue) or multiple independent consumer groups reading the same events (stream)?"
- "What's my delivery guarantee, and how do I make the consumer idempotent given at-least-once delivery?"
Practice it in a case study
Design a URL Shortener
Even a simple redirect service can offload click-analytics recording to a queue so the redirect itself stays fast.
Done with this lesson?
Mark it complete once you've worked through the content and practice above.
Next lesson
Sharding & Replication
