INC-0005 Starvation

05 / dead letter

The ticket had been open for nineteen hours before anyone opened it.

[ALERT] INC-0005 — Dead-letter depth above threshold (retry fabric)
[WARN ] q-dead: 41,207 messages, oldest 19h04m
[WARN ] All consumers report healthy
[WARN ] No consumer has been restarted in 6 days
[INFO ] SLA: acknowledgement budget 30m

Andy read it twice. Nothing had crashed. Nothing was blocked. Every consumer was answering its health probe on time and reporting itself ready for work.

One of them had not done any.

Bob appeared with a second coffee he did not offer.

“Three consumers, three lanes. One of them is starving.”

“Starving.”

“It’s eligible every round. It just never wins.” Bob tapped the depth graph, which had been climbing in a perfectly straight line since yesterday evening. “Nothing is broken. That’s the part people can’t hold in their heads. Something has to be more urgent, forever, and it is.”

Andy pulled the scheduler’s acknowledgement table. Two of the three numbers were inside the budget. The third was not close.

evidence retained

The consumer whose last acknowledgement was 14s is the one attached to q-dead.
notify did not record the 19h acknowledgement.
No consumer on q-hot has been silent for 19h.
ingest consumes from q-dead.

messages bob

Bob 14:06

Don't restart anything. A restart makes the graph look better and tells you nothing.

Bob 14:07

The budget is 30 minutes. Exactly one acknowledgement in that table is outside it. Start there and work outward.

Bob 14:09

Left columns are consumers and acknowledgements. Rows are lanes and acknowledgements. Each consumer holds one lane and one acknowledgement.

Bob 14:11

Tell me which consumer starved, which lane it is stuck behind, and when it last got a word in. Format: consumer, lane, last ack.

table retry fabric

fault

Consumer never scheduled

ack budget

30m

severity

SEV-3

Skip the grid, go to the answer

	ingest	settle	notify	14s	9m	19h
q-hot
q-warm
q-dead
14s
9m
19h

answer input incident response

Question: Which consumer starved, on which lane, and when did it last acknowledge?

operator answer

05 / after the starvation

settle had been ready for nineteen hours.

Every scheduling round it declared itself eligible. Every scheduling round the incident traffic on q-hot arrived first, took the worker, and released it a moment before settle was polled again. Its own backlog aged past the retry ceiling and fell through into q-dead, where ingest dutifully filed it.

Nothing failed. The system did exactly what it was told to do, continuously, for most of a day.

sched-trace — retry fabric

  round   eligible                     granted     reason
  ─────────────────────────────────────────────────────────────
  4471    settle, notify               notify      q-hot first
  4472    settle, notify               notify      q-hot first
  4473    settle, ingest, notify       notify      q-hot first
  4474    settle, notify               notify      q-hot first
   ...    (11,904 rounds omitted)
  16378   settle, notify               notify      q-hot first
  ─────────────────────────────────────────────────────────────
  ► settle eligible in 16,378 of 16,378 rounds
  ► settle granted in 0
  ► no error raised: starvation is not a failure state

05 / loose thread

Andy wrote up the incident, marked it resolved, and then did the thing he always did, which was keep reading after the answer.

He pulled the maintenance calendar for the last three weeks.

scheduled maintenance windows   21
windows completed                0
windows interrupted by incident 21

Twenty-one for twenty-one. He looked at it for a while and decided it was the least surprising number he had seen all week. Of course they were interrupted. Nobody on this team had finished anything in a month.

He closed the calendar.

Bob, already leaving: “Congratulations. You’ve found the part of the job nobody writes a runbook for.”

“Which part?”

“The part where nothing is wrong.”

NEXT INCIDENT →