INC-0006 Livelock

06 / polite forever

The graphs were beautiful. That was the first thing wrong with them.

[ALERT] INC-0006 — Reconciliation backlog static for 90m
[INFO ] recovery workers: 3/3 running
[INFO ] cpu: 94% sustained
[INFO ] state transitions against the static backlog: 2,114,880
[WARN ] commits against the static backlog: 3

Two million state transitions. Three commits.

Andy scrolled the transition log until the lines stopped meaning anything: claim, yield, reclaim, yield, claim. Two workers passing the same batch back and forth with the manners of people stuck in a doorway.

“Not a deadlock,” he said, before Bob could.

Bob sat down anyway. “No. A deadlock stops. You can see a deadlock from orbit — nothing moves, everyone panics, someone kills a process, we all go to lunch.”

He gestured at the CPU graph, flat and high.

“This one looks like the team working late. It’ll look like that until the quarter ends.”

Andy pulled the commit counters. One worker had done everything. One had done almost nothing. One had done nothing at all, and had been doing nothing at all with tremendous energy.

evidence retained

The 50ms backoff window was never entered by w-merge or w-reap.
The worker in the 50ms window committed something: it is not the one sitting at 0 ops.
w-drain is not the worker that committed 41k ops.
The 500ms window does not belong to w-reap.
w-merge did not commit 41k ops.
A recovery transition line the audit index says was never written: [recovery] transition rejected — nx::epoch=…. Bob reads it, shrugs, and calls it a torn write.

messages incident channel

Ops 03:31

Workers are healthy. CPU is healthy. Backlog hasn't moved since 02:01.

Bob 03:33

Nothing in that evidence list tells you a worker's backoff outright. You'll have to take the ones away that can't be.

Bob 03:35

Left columns are workers and backoff windows. Rows are commit counts and backoff windows. One backoff and one commit count each.

Bob 03:36

Find the worker that committed nothing at all. Format: worker, backoff, committed.

table recovery workers

fault

Progress without completion

sample

90m

severity

SEV-2

Skip the grid, go to the answer

	w-reap	w-drain	w-merge	50ms	500ms	5s
0 ops
3 ops
41k ops
50ms
500ms
5s

answer input incident response

Question: Which worker made no progress at all, on which backoff, with what committed?

operator answer

06 / after the livelock

w-merge and w-drain were each other’s trigger.

w-merge claimed a batch and began merging. w-drain saw a claimed batch aging and reclaimed it for the drain path. w-merge saw its claim revoked, backed off, and claimed again. Each one was correctly implementing a recovery policy. Each one was the reason the other had something to recover from.

w-drain got three transactions through in ninety minutes, in the gaps. w-merge got none. w-reap, on a five second backoff and talking to nobody, did all forty-one thousand.

transition-ring — INC-0006

        claim                     reclaim
   ┌──────────────┐          ┌──────────────┐
   │   w-merge    │ ───────► │   w-drain    │
   │  backoff 500 │ ◄─────── │  backoff 50  │
   └──────────────┘  revoke  └──────────────┘
          │                          │
          ▼                          ▼
     commits: 0                 commits: 3

  t+00:00  w-merge  claim   batch-88431
  t+00:00  w-drain  reclaim batch-88431
  t+00:00  w-merge  yield   backoff 500ms
  t+00:00  w-drain  yield   backoff 50ms
  t+00:01  w-merge  claim   batch-88431
           ... 2,114,880 transitions ...

  ► both workers changed state 700k+ times
  ► neither worker is blocked at any point
  ► no lock is ever held longer than 4ms

06 / loose thread

The fix was four lines: one of them had to stop being polite.

Andy filed it, then went back to the transition log, because two million lines of claim-yield had produced a handful that were not claim-yield. Torn writes, Bob had called them. Andy reassembled them anyway, in order, by offset.

recovered fragment retained

INC-0006 recovery transition log

nx::epoch=0416

Recovered from the malformed lines the flapping produced, at an offset the audit index has no record of.

06 / the index

He had seen that shape before. Five days on this floor and twice now a line had arrived in a log addressed to nobody, in a format no service emitted.

He checked the audit index, which is authoritative, which is the entire point of an audit index.

offset 0x41f0–0x4230   no record

The lines were in the log. The index said the log had never been written to.

Andy copied both to his laptop.

NEXT INCIDENT →