A team I worked with ran Kafka for eighteen months to move about four thousand messages a day between two services. Three brokers, a Zookeeper ensemble at the time, a schema registry, and one engineer who understood the consumer group rebalancing behaviour. When he left, nobody would touch it.
Four thousand messages a day is roughly three a minute. A Postgres table with SELECT ... FOR UPDATE SKIP LOCKED would have done it, and any of them could have debugged that.
The choice between these tools is much less about throughput than the comparison posts suggest. It is about what shape of delivery you need, and who has to operate it at 3am.
Two Different Things Wearing the Same Name
Most confusion here comes from treating these as interchangeable. They are two categories.
A message queue (RabbitMQ, SQS) hands each message to one consumer, which acknowledges it, and then the message is gone. It is a work distribution mechanism. Think of a task list: someone takes the task, does it, crosses it off.
A log (Kafka) appends messages to an ordered, durable sequence that consumers read at their own position. Nothing is removed when read. Many independent consumers can read the same messages, and any of them can rewind.
That difference decides almost everything. If you want three separate systems to react to "order placed" — billing, email, analytics — a queue makes that awkward (you need fan-out to three queues) and a log makes it natural (three consumer groups, same topic). If you want one worker to resize an image, a queue is the obvious fit and a log is overkill.
SQS: The Default I Recommend Most Often
If you are on AWS and you need a queue, start here. There is no cluster, no capacity planning, no upgrades, and no on-call. You pay per request and it scales without you thinking about it.
What it gives you: at-least-once delivery, dead-letter queues after N failures, visibility timeouts so a crashed worker's message returns to the queue, and long polling. FIFO queues exist when you need strict ordering and exactly-once processing within a message group, at lower throughput.
The limits worth knowing: 256KB per message (put large payloads in S3 and send the key), no server-side routing or filtering, and standard queues can deliver duplicates and reorder. That last point is not a defect — it is the trade for the operational simplicity, and it is fine as long as your consumers are idempotent, which they should be regardless.
// Idempotent consumer. Assume redelivery, always.
async function handle(msg: Message) {
const id = msg.MessageAttributes?.eventId?.StringValue;
const fresh = await store.claim(`processed:${id}`, { ttl: "7d" });
if (!fresh) return ack(msg); // already handled; drop it quietly
await doWork(JSON.parse(msg.Body!));
await ack(msg);
}
RabbitMQ: When Routing Is the Requirement
RabbitMQ earns its place when the interesting part is deciding which consumer gets a message. Topic exchanges with routing keys, header-based routing, priority queues, per-message TTLs, delayed delivery. If your requirement sounds like "messages tagged eu.*.urgent go to this pool and everything else to that one," that is RabbitMQ's shape.
It is also the reasonable choice when you are not on AWS, or you need to run on your own infrastructure. Latency is low, and it handles tens of thousands of messages a second on modest hardware.
What you take on: it is a server you operate. Memory pressure when queues back up, disk alarms, cluster partition behaviour that has genuine sharp edges, and upgrades. Manageable, but it is a thing with a heartbeat.
Kafka: For Streams, Not for Tasks
Kafka is excellent and it is chosen wrongly more than any other component in this space. It is the right answer when you need:
Multiple independent consumers of the same event stream, each at their own pace.
Replay. A new service can read the last thirty days of events and build its own state. This is genuinely powerful and no queue does it.
Ordering within a key at high volume — all events for one account, in order, guaranteed.
Very high throughput — hundreds of thousands of messages a second, sustained.
Event sourcing or stream processing where the log itself is the source of truth.
What it costs: real operational weight. Partition counts you must plan for because increasing them later breaks key ordering. Consumer group rebalancing that stalls processing when instances come and go. Retention and disk sizing. Client libraries with a lot of configuration that matters. It is not a thing you run casually, and managed offerings reduce but do not remove that.
The honest test: if you cannot name a second consumer of the same stream, or a scenario where you would replay, you probably want a queue.
The Option Everyone Skips
Before any of them — can a database table do it?
UPDATE jobs SET status = 'running', started_at = now(), attempts = attempts + 1
WHERE id = (
SELECT id FROM jobs
WHERE status = 'pending' AND run_after <= now()
ORDER BY priority DESC, created_at
FOR UPDATE SKIP LOCKED
LIMIT 1
)
RETURNING *;
SKIP LOCKED makes this safe with many concurrent workers — each grabs a different row without blocking. You get retries, scheduling, priority and visibility into the queue with a normal SELECT, which is worth more during an incident than people expect.
The decisive advantage is transactional enqueue: the job is created in the same transaction as the data change that caused it. With an external broker, the classic bug is publishing a message and then having the transaction roll back — now there is a job for a row that does not exist. Solving that properly needs an outbox pattern, which is more machinery than the table you were avoiding.
This works comfortably into the low thousands of jobs per minute. Well past where most teams think they need Kafka.
How I Choose
Working down, taking the first that fits:
A database table if it is background work, volume is moderate, and you already run Postgres. Fewest moving parts, transactional enqueue, easy to inspect.
SQS if you are on AWS and want a queue you never operate. This covers most task-processing needs at most companies.
RabbitMQ if routing logic is genuinely the requirement, or you are not on AWS.
Kafka if you need replay, multiple independent consumers, or throughput that rules the others out — and you have someone who will own it.
Whatever You Pick
Three things matter more than the choice itself.
Consumers must be idempotent. Every one of these systems can deliver twice. Design for it rather than hoping.
Dead-letter handling from day one. A message that fails forever will retry forever and consume your throughput. Cap attempts, move it aside, and — the part people skip — put an alert on the dead-letter queue being non-empty. An unwatched DLQ is a silent data-loss channel.
Events describe what happened, not what to do. Publish OrderPlaced, not SendConfirmationEmail. The moment a message names an action, the publisher is coupled to the consumer again and you have a distributed monolith.
That team eventually moved to SQS in about a week. Four thousand messages a day, and now anyone on the team can debug it.
