Choosing between “event delivery” and “event history”

SQS and Kafka solve different correctness problems.
Throughput is secondary.

1️⃣ The Wrong Question People Ask

❌ Wrong

“Which is faster? SQS or Kafka?”

✅ Right

“What correctness guarantees does my system actually need?”

Because once you pick:

SQS → you accept lossy history
Kafka → you accept coordination cost

You cannot escape the tradeoff.

2️⃣ Core Mental Model (This Matters)

🟦 SQS Mental Model

A distributed work queue

Message exists until processed
After deletion → gone forever
Queue represents pending work

🟥 Kafka Mental Model

An append-only distributed log

Messages are immutable
Consumers track position
Log represents event history

This difference explains every correctness tradeoff.

3️⃣ Delivery Semantics (The Foundation)

Property	SQS	Kafka
Delivery	At-least-once	At-least-once
Exactly-once	❌	⚠️ (within constraints)
Ordering	FIFO only	Partition-level
Message retention	Until consumed	Time/size-based
Replay	❌	✅

Kafka remembers.
SQS forgets.

4️⃣ Duplicate Processing (Both Have It, Differently)

SQS Duplicates

Visibility timeout expires
Consumer crashes
Message reappears

Process → crash → process again

Kafka Duplicates

Consumer crashes before commit
Offset not committed
Message replayed

Process → crash → replay from offset

Both are at-least-once.
But Kafka lets you replay intentionally.

5️⃣ Correctness Implication #1 — Idempotency

In SQS

Idempotency is mandatory.

No replay = no recovery

If you mess up:

Data is wrong forever

In Kafka

Idempotency is strongly recommended.

But you have:

Rewind
Reprocess
Fix-forward

Kafka tolerates mistakes.
SQS does not.

6️⃣ Ordering Guarantees (Subtle but Huge)

SQS

Standard: ❌ no ordering
FIFO: ordering per MessageGroupId

If one message is slow:

Entire group blocks

Kafka

Ordering per partition
You control partitioning key

orderId → same partition

Parallelism + ordering = possible.

7️⃣ Correctness Implication #2 — Causality

SQS

Once message is deleted:

Causality is lost
History is gone

You cannot answer:

“What happened before this?”

Kafka

Log preserves causality.

You can:

Reconstruct timelines
Debug bugs
Validate invariants

Kafka is debuggable.
SQS is operational.

8️⃣ Failure Recovery (Huge Difference)

SQS Failure Recovery

Failure	Outcome
Buggy consumer	Data corrupted
Bad deploy	Messages gone
Wrong logic	No rewind

Only option:

Manual repair
Re-run upstream jobs

Kafka Failure Recovery

Failure	Outcome
Buggy consumer	Reset offset
Bad deploy	Replay
New logic	Reprocess

Kafka gives you time travel.

9️⃣ DLQ vs Replay (Philosophical Difference)

SQS DLQ

“This message is broken”
Isolate and move on

Correctness boundary

Kafka Replay

“Processing was wrong”
Fix logic
Re-run history

Correctness recovery

10️⃣ Backpressure & Load (Correctness Angle)

SQS

Producers keep producing
Queue depth grows
Processing delay grows silently

Correctness risk:

Time-sensitive events become meaningless

Kafka

Consumers lag
Lag is observable
Replay window bounded

Correctness risk:

Lag-based staleness
But visible and measurable

1️⃣1️⃣ Exactly-Once Semantics (Reality Check)

SQS

❌ Impossible by design

Kafka

⚠️ Possible only if:

Idempotent producers
Transactional writes
Single Kafka cluster
Controlled sinks

Even then:

Exactly-once is contextual, not absolute.

1️⃣2️⃣ Multi-Consumer Correctness

SQS

Each message → one consumer

Bad for:

Fan-out
Independent consumers

You must duplicate queues.

Kafka

Many consumers can read same log.

Good for:

Analytics
Auditing
Side effects

Kafka separates event storage from event consumption.

1️⃣3️⃣ D2 — Mental Model Comparison

SQS

Kafka

1️⃣4️⃣ When SQS Is the Correct Choice

Use SQS when:

You want work distribution
You don’t care about history
You want minimal ops
You can tolerate duplicates
You want managed simplicity

Examples:

Email sending
Image processing
Background jobs
Async side effects

1️⃣5️⃣ When Kafka Is the Correct Choice

Use Kafka when:

You need event history
Replay matters
Debuggability matters
Multiple consumers exist
Correctness > simplicity

Examples:

Payments
Order lifecycle
Analytics
CDC
Audit logs

1️⃣6️⃣ The Hidden Cost Tradeoff

Dimension	SQS	Kafka
Ops cost	Low	High
Correctness recovery	❌	✅
Debuggability	❌	✅
Time travel	❌	✅
Simplicity	✅	❌

Kafka charges operational complexity
in exchange for correctness leverage.

1️⃣7️⃣ Design Rule (Hard-Won)

If you cannot afford to lose history, do not use SQS.
If you cannot afford operational complexity, do not use Kafka.

📘 SQS vs Kafka: Correctness Tradeoffs