📘 SQS vs Kafka: Correctness Tradeoffs

Choosing between “event delivery” and “event history”

SQS and Kafka solve different correctness problems.
Throughput is secondary.

1️⃣ The Wrong Question People Ask

❌ Wrong

“Which is faster? SQS or Kafka?”

✅ Right

“What correctness guarantees does my system actually need?”

Because once you pick:

• SQS → you accept lossy history

• Kafka → you accept coordination cost

You cannot escape the tradeoff.

2️⃣ Core Mental Model (This Matters)

🟦 SQS Mental Model

A distributed work queue

• Message exists until processed

• After deletion → gone forever

• Queue represents pending work

🟥 Kafka Mental Model

An append-only distributed log

• Messages are immutable

• Consumers track position

• Log represents event history

This difference explains every correctness tradeoff.

3️⃣ Delivery Semantics (The Foundation)

Property	SQS	Kafka
Delivery	At-least-once	At-least-once
Exactly-once	❌	⚠️ (within constraints)
Ordering	FIFO only	Partition-level
Message retention	Until consumed	Time/size-based
Replay	❌	✅

Kafka remembers.
SQS forgets.

4️⃣ Duplicate Processing (Both Have It, Differently)

SQS Duplicates

• Visibility timeout expires

• Consumer crashes

• Message reappears

Process → crash → process again

Kafka Duplicates

• Consumer crashes before commit

• Offset not committed

• Message replayed

Process → crash → replay from offset

Both are at-least-once.
But Kafka lets you replay intentionally.

5️⃣ Correctness Implication #1 — Idempotency

In SQS

Idempotency is mandatory.

No replay = no recovery

If you mess up:

• Data is wrong forever

In Kafka

Idempotency is strongly recommended.

But you have:

• Rewind

• Reprocess

• Fix-forward

Kafka tolerates mistakes.
SQS does not.

6️⃣ Ordering Guarantees (Subtle but Huge)

SQS

• Standard: ❌ no ordering

• FIFO: ordering per MessageGroupId

If one message is slow:

• Entire group blocks

Kafka

• Ordering per partition

• You control partitioning key

orderId → same partition

Parallelism + ordering = possible.

7️⃣ Correctness Implication #2 — Causality

SQS

Once message is deleted:

• Causality is lost

• History is gone

You cannot answer:

“What happened before this?”

Kafka

Log preserves causality.

You can:

• Reconstruct timelines

• Debug bugs

• Validate invariants

Kafka is debuggable.
SQS is operational.

8️⃣ Failure Recovery (Huge Difference)

SQS Failure Recovery

Failure	Outcome
Buggy consumer	Data corrupted
Bad deploy	Messages gone
Wrong logic	No rewind

Only option:

• Manual repair

• Re-run upstream jobs

Kafka Failure Recovery

Failure	Outcome
Buggy consumer	Reset offset
Bad deploy	Replay
New logic	Reprocess

Kafka gives you time travel.

9️⃣ DLQ vs Replay (Philosophical Difference)

SQS DLQ

• “This message is broken”

• Isolate and move on

Correctness boundary

Kafka Replay

• “Processing was wrong”

• Fix logic

• Re-run history

Correctness recovery

10️⃣ Backpressure & Load (Correctness Angle)

SQS

• Producers keep producing

• Queue depth grows

• Processing delay grows silently

Correctness risk:

• Time-sensitive events become meaningless

Kafka

• Consumers lag

• Lag is observable

• Replay window bounded

Correctness risk:

• Lag-based staleness

• But visible and measurable

1️⃣1️⃣ Exactly-Once Semantics (Reality Check)

SQS

❌ Impossible by design

Kafka

⚠️ Possible only if:

• Idempotent producers

• Transactional writes

• Single Kafka cluster

• Controlled sinks

Even then:

Exactly-once is contextual, not absolute.

1️⃣2️⃣ Multi-Consumer Correctness

SQS

Each message → one consumer

Bad for:

• Fan-out

• Independent consumers

You must duplicate queues.

Kafka

Many consumers can read same log.

Good for:

• Analytics

• Auditing

• Side effects

Kafka separates event storage from event consumption.

1️⃣3️⃣ D2 — Mental Model Comparison

SQS

Kafka

1️⃣4️⃣ When SQS Is the Correct Choice

Use SQS when:

• You want work distribution

• You don’t care about history

• You want minimal ops

• You can tolerate duplicates

• You want managed simplicity

Examples:

• Email sending

• Image processing

• Background jobs

• Async side effects

1️⃣5️⃣ When Kafka Is the Correct Choice

Use Kafka when:

• You need event history

• Replay matters

• Debuggability matters

• Multiple consumers exist

• Correctness > simplicity

Examples:

• Payments

• Order lifecycle

• Analytics

• CDC

• Audit logs

1️⃣6️⃣ The Hidden Cost Tradeoff

Dimension	SQS	Kafka
Ops cost	Low	High
Correctness recovery	❌	✅
Debuggability	❌	✅
Time travel	❌	✅
Simplicity	✅	❌

Kafka charges operational complexity
in exchange for correctness leverage.

1️⃣7️⃣ Design Rule (Hard-Won)

If you cannot afford to lose history, do not use SQS.
If you cannot afford operational complexity, do not use Kafka.