📘 Correctness Under Concurrency

When your system is fast, scalable — and still wrong

Most production bugs are not caused by slowness.
They are caused by multiple things happening at the same time.

Concurrency bugs exist:

• At low traffic

• On one machine

• Even with perfect hardware

They are logic bugs, not capacity bugs.

1️⃣ What “Correctness” Actually Means

A system is correct if it behaves as expected under all valid interleavings of operations.

Not:

• “Works on my machine”

• “Passes unit tests”

• “Works at low traffic”

But:

Works when operations overlap in unpredictable ways.

2️⃣ Why Concurrency Breaks Correctness

Because time is not linear in distributed systems.

You think this happens:

A → B → C

Reality:

A ↘
     B
A ↗     C

Operations interleave.

3️⃣ The Simplest Concurrency Bug: Lost Update

❌ Naive Code

async function likePost(postId) {
  const post = await db.get(postId);
  post.likes += 1;
  await db.save(post);
}

What you expect

2 likes → likes + 2

What actually happens

2 likes → likes + 1 ❌

Concurrency overwrote correctness.

4️⃣ Why This Happens (Key Insight)

The bug is not “missing a lock”.
The bug is read-modify-write without protection.

This pattern is always unsafe under concurrency.

5️⃣ Naive Fix #1 — Application Locks ❌

await mutex.lock();
const post = await db.get(postId);
post.likes += 1;
await db.save(post);
await mutex.unlock();

Why this fails

• Only works in one process

• Fails with multiple instances

• Kills throughput

6️⃣ Correct Fix #1 — Atomic Operations (Best)

UPDATE posts
SET likes = likes + 1
WHERE id = ?

Why this works

• Atomic at DB level

• No read-modify-write

• Scales correctly

Push correctness down to the lowest layer possible.

7️⃣ Correct Fix #2 — Optimistic Concurrency Control (OCC)

Version-based update

const post = await db.get(postId);

await db.update(
  { id: postId, version: post.version },
  { likes: post.likes + 1, version: post.version + 1 }
);

If version mismatches → retry.

8️⃣ Correctness Bug #2 — Check-Then-Act

❌ Broken Logic

if (balance >= amount) {
  balance -= amount;
}

Why it fails

• Condition can change

• Assumption becomes false

9️⃣ Correct Fix — Combine Check + Act

UPDATE accounts
SET balance = balance - 100
WHERE id = ? AND balance >= 100

Rows affected:

• 1 → success

• 0 → insufficient funds

🔟 Correctness Bug #3 — Idempotency

❌ Duplicate Request Bug

createOrder(order);
createOrder(order); // retry

Result:

• Two orders ❌

1️⃣1️⃣ Correct Fix — Idempotency Keys

if (seen(idempotencyKey)) return cachedResult;

const result = createOrder(order);
store(idempotencyKey, result);

Correctness is about recognizing repeated intent.

1️⃣2️⃣ Why Correctness Is Harder Than Performance

Performance	Correctness
Gradual failure	Binary failure
Can degrade	Cannot be “almost right”
Measurable	Often invisible
Testable	Needs reasoning

1️⃣3️⃣ Why Load Doesn’t Matter Here

Concurrency bugs:

• Happen at 2 requests

• Reproduce rarely

• Pass load tests

• Fail in prod

This is why senior engineers fear them.

🎯 Golden Rules for Correctness Under Concurrency

1. Avoid read-modify-write

2. Use atomic primitives

3. Prefer database-enforced correctness

4. Make operations idempotent

5. Assume retries happen

6. Design for reordering