INC-0009 Infinite Recursion

09 / all the way down

The repair service had opened four hundred thousand tickets against itself.

[ALERT] INC-0009 — Autorepair cascade, unbounded
[INFO ] unwind budget: d-8k
[WARN ] frames open: 412,880 and climbing
[WARN ] resident stack: 2.0 GB
[CRITC] defect queue is being produced faster than it is consumed
[CRITC] producer: autorepair
[CRITC] consumer: autorepair

It had started, as far as anyone could reconstruct, with a genuinely malformed config value at 04:12. resolve had corrected it. expand had looked at the correction, decided it was malformed, and corrected it. inline had looked at that, and so on, around a ring of four stages that were each doing their job perfectly well on an input that was now entirely made of previous repairs.

Andy watched the frame counter climb for a while.

“Why doesn’t it stop?”

“Something stops,” Bob said. “Three of those four stages have a base case and they’re hitting it — you can see it, their depth counters are sitting still.”

He drank some coffee.

“One of them has a base case it can never reach. That one has been going since quarter past four this morning and it’s the only reason the other three are still being handed work.”

evidence retained

No frame re-enters itself directly. Every call in the trace is to a different frame.
resolve bottomed out at d-128.
resolve calls expand.
emit peaked at 32 MB of stack.
The frame that reached d-4k peaked at 256 MB.
Whatever calls inline also peaked at 256 MB.
The frame that calls expand did not reach 2 GB.
inline did not stop at d-512.
Recovered from frame 4096 of the cascade, between two identical stack traces: nx::src=…

messages bob

Bob 09:14

Don't kill it yet. If you kill it now you get 400,000 half-applied repairs and a much worse morning.

Bob 09:16

Columns are frames, call targets, depths. Rows are stack, depths, call targets.

Bob 09:17

Nothing calls itself. That's the same trick as your first week — it tells you more than it looks like it does.

Bob 09:19

Find the frame that never terminates. Format: frame, calls, depth, stack.

table autorepair frames

fault

Unbounded self-repair

unwind budget

d-8k

severity

SEV-1

Skip the grid, go to the answer

	resolve	expand	inline	emit	→resolve	→expand	→inline	→emit	d-128	d-512	d-4k	d-∞
4 MB
32 MB
256 MB
2 GB
d-128
d-512
d-4k
d-∞
→resolve
→expand
→inline
→emit

answer input incident response

Question: Which frame never terminates — what does it call, how deep did it go, and how much stack did it take?

operator answer

09 / after the cascade

inline compared a normalised path against an unnormalised one.

That was all. The base case asked whether the value it was about to repair was already repaired, and it asked in a form the value could never take, so the answer was always no, so it repaired it again. Four hundred thousand times.

The other three stages were fine. They terminated correctly every time, on input that inline had guaranteed would keep arriving.

repair-ring — INC-0009

      ┌──────────┐      ┌──────────┐      ┌──────────┐      ┌──────────┐
   ┌─►│ resolve  │─────►│  expand  │─────►│  inline  │─────►│   emit   │──┐
   │  │  d-128   │      │   d-4k   │      │   d-∞    │      │  d-512   │  │
   │  │   4 MB   │      │  256 MB  │      │   2 GB   │      │  32 MB   │  │
   │  └──────────┘      └──────────┘      └──────────┘      └──────────┘  │
   └──────────────────────────────────────────────────────────────────────┘

  inline/base_case:
      if normalize(value) == value:        # value is never normalised here
          return                           # unreachable

  ► 04:12:07  first genuine defect
  ► 04:12:07  first repair of a repair
  ► 09:21:44  412,880 frames, still climbing

09 / loose thread

One line to fix. Normalise both sides.

Andy did not fix it immediately, because four hundred thousand frames of identical stack traces is the best hiding place in the building, and he had learned to look in the noise. He pulled frame 4096 — no reason, it was a round number — and read down until the trace stopped repeating.

recovered fragment retained

INC-0009 repair cascade, frame 4096

nx::src=harness-04

Pulled out of the cascade noise, between two identical stack traces.

09 / three of them

He opened the file he had been keeping and put all three on one page for the first time.

nx::hrs_see_all::hrs-07
nx::epoch=0416
nx::src=harness-04

A stream. An epoch. A source.

Separately, each one had looked like corruption. Together they looked like an address — the kind you would use to pull a single record out of somewhere very large, if you knew where to point it.

Andy did not know where to point it.

He sat with that for a while, then went and fixed the one line, and the cascade unwound over the next forty minutes like something being let go of.

NEXT INCIDENT →