INC-0003 Race Condition

03 / race condition

02:13 AM. Andy’s phone buzzed once, then again. Then it didn’t stop.

He opened his laptop in bed. The monitoring dashboard was a wall of red.

[ALERT] INC-0003 — Shared-state conflict detected on account cust-9174
[ALERT] Subscription status mismatch: ACTIVE vs SUSPENDED
[ALERT] Malformed telemetry fragment in audit replay log

Two workers had updated the same customer account simultaneously. One system said the subscription was ACTIVE. Another said the same account was SUSPENDED.

Security immediately blamed an external intrusion attempt after malformed telemetry appeared during log replay.

But Bob was already online. He ignored the intrusion alert completely.

A message appeared in the incident channel:

“Find which worker overwrote the state.”

evidence retained

reconcile consumed from q-pay.
The process consuming from q-sub produced SUSPENDED.
The process consuming from q-audit generated malformed telemetry.
reconcile produced ACTIVE.
The process that generated malformed telemetry produced CORRUPTED.
suspend did not consume from q-audit.

messages incident channel

Bob 02:14

Two writes. One stale snapshot. That's enough to corrupt state.

Ops 02:15

Logs from q-audit look damaged.

Security 02:16

We believe this may be external interference.

table queue analysis

incident

INC-0003

type

Shared-state overwrite

severity

SEV-1

	reconcile	suspend	replay	ACTIVE	SUSPENDED	CORRUPTED
q-pay
q-sub
q-audit
ACTIVE
SUSPENDED
CORRUPTED

answer input incident response

Question: Which process corrupted the account state? Enter: process, its queue, and the result it produced.

operator answer

03 / after the race

Andy reconstructed the incident.

Two workers read the same customer state simultaneously. One worker marked the subscription ACTIVE after payment confirmation. Another overwrote the account moments later using stale data.

The race condition corrupted the final state.

timeline — INC-0003

  t0  ┌──────────┐   read    ┌───────────┐
      │reconcile │ ────────► │ cust-9174 │  state: PENDING
      └──────────┘           └───────────┘
  t1  ┌──────────┐   read    ┌───────────┐
      │  replay  │ ────────► │ cust-9174 │  state: PENDING (stale)
      └──────────┘           └───────────┘
  t2  ┌──────────┐   write   ┌───────────┐
      │reconcile │ ────────► │ cust-9174 │  state: ACTIVE ✓
      └──────────┘           └───────────┘
  t3  ┌──────────┐   write   ┌───────────┐
      │  replay  │ ────────► │ cust-9174 │  state: CORRUPTED ✕
      └──────────┘           └───────────┘

  ► replay overwrote ACTIVE with stale snapshot
  ► final state: CORRUPTED

03 / loose thread

But Andy kept thinking about the malformed telemetry fragment:

nx::hrtz_see_all

It wasn’t a valid function call. It wasn’t in any codebase he could find. And Security erased the replay logs immediately after the alert appeared.

Bob said nothing.

NEXT INCIDENT →