INC-0010 Byzantine Fault

10 / five truths

Nothing was corrupted. That was what took Andy the longest to accept.

[ALERT] INC-0010 — Attestation quorum failed, 5 divergent reports
[INFO ] all reports well-formed
[INFO ] all signatures verify
[INFO ] all epochs within window
[WARN ] agreement: 0/5
[WARN ] no replica has reported a fault

Five replicas attest to the same ledger. Once per epoch each one publishes a digest of what it believes the world looks like, countersigns it, and pushes it into its telemetry stream. They are supposed to be boring and identical.

This morning they were five different histories, each one signed, each one plausible, none of them agreeing about anything.

Bob looked at it for a long time before he said anything, which had not happened before.

“This isn’t a crash. A crash is a system that stopped telling you things.” He tapped the report count. “This is a system that is still telling you things.”

“One of them is lying.”

“One of them is reporting,” Bob said. “Nothing in that list is malformed. A liar who malforms things is a bug. A liar who signs correctly is a participant.”

He pulled up the key registry and let Andy find it himself.

Five countersigning keys. Four of them were service identities.

evidence retained

r1 countersigns with k1.
r1 attested in epoch e5.
r1 produced digest d2.
The k1 attestation goes out on s3.
r5 publishes into s2.
Whatever is countersigned k3 publishes into s2.
Epoch e3 is countersigned k3.
Digest d4 is countersigned k5.
The attestation countersigned k2 went out on s5.
Digest d5 was not produced in epoch e3.
Nothing on s5 carries the epoch e4 mark.
The e4 attestation is not countersigned k5.
r2 did not attest in epoch e4.
r4 does not countersign with k4.
The e2 attestation did not go out on s1.
r3 does not publish into s1.
r4 does not publish into s1.
r5 did not produce digest d1.
Digest d1 did not appear on s4.

case file andy — personal

INC-0003 audit replay log

nx::hrs_see_all::hrs-07

Not a valid call. Not in any codebase Andy could find. Security erased the replay logs immediately after the alert appeared.

open incident →

INC-0006 recovery transition log

nx::epoch=0416

Recovered from the malformed lines the flapping produced, at an offset the audit index has no record of.

open incident →

INC-0009 repair cascade, frame 4096

nx::src=harness-04

Pulled out of the cascade noise, between two identical stack traces.

open incident →

messages bob

Bob 07:48

Five of everything. The board is as big as it gets and the rules still haven't changed.

Bob 07:49

Columns are replicas, digests, epochs, keys. Rows are streams, keys, epochs, digests. Registry panel tells you what the short forms mean.

Bob 07:52

Four of those countersigning keys belong to services. Find the attestation carrying the fifth.

Bob 07:53

Format: replica, digest, epoch, key, stream.

Bob 08:31

And Andy. Whatever it turns out to be. Write it down somewhere that isn't a company system.

registry attestation fabric

  field abbreviations — one row per field, as the fabric logs them
  ──────────────────────────────────────────────────────────────────────────────────────
  replica   r1 repl-aurora   r2 repl-borealis  r3 repl-cinder   r4 repl-dune     r5 repl-ember
  digest    d1 0x41ae…       d2 0x7c02…        d3 0x9f31…       d4 0xb4d8…       d5 0xe016…
  epoch     e1 0411          e2 0412           e3 0414          e4 0416          e5 0419
  key       k1 att.ops       k2 att.rel        k3 att.bot       k4 r.sato        k5 att.arc
  stream    s1 tel/hrs-02    s2 tel/hrs-03     s3 tel/hrs-05    s4 tel/hrs-07    s5 tel/hrs-11

table divergent attestations

fault

Attestations do not agree

keys

5 · service keys 4

severity

SEV-1

Skip the grid, go to the answer

	r1	r2	r3	r4	r5	d1	d2	d3	d4	d5	e1	e2	e3	e4	e5	k1	k2	k3	k4	k5
s1
s2
s3
s4
s5
k1
k2
k3
k4
k5
e1
e2
e3
e4
e5
d1
d2
d3
d4
d5

answer input incident response

Question: Which attestation carries the non-service key — replica, digest, epoch, key, stream?

operator answer

10 / the divergent replica

repl-cinder. Digest 0xe016, epoch 0416, countersigned r.sato, published into tel/hrs-07.

Andy read the row three times.

He knew three of those five fields before he solved for them. r.sato had signed a scheduler change three months after Rin stopped existing. tel/hrs-07 was the fifth reader in a four-monitor cluster, the one that had watched the whole partition and written nothing.

And 0416 was in his own notes, in a fragment he had pulled out of two million lines of livelock noise, addressed to nobody.

attest-diff — INC-0010

  replica         epoch   digest      countersigned   stream
  ──────────────────────────────────────────────────────────────────
  repl-aurora     0419    0x7c02…     att.ops         tel/hrs-05
  repl-borealis   0411    0xb4d8…     att.arc         tel/hrs-02
  repl-cinder     0416    0xe016…     r.sato          tel/hrs-07   ◄
  repl-dune       0412    0x41ae…     att.rel         tel/hrs-11
  repl-ember      0414    0x9f31…     att.bot         tel/hrs-03
  ──────────────────────────────────────────────────────────────────
  ► every signature verifies
  ► r.sato is not a service identity
  ► tel/hrs-07 has no entry in the service catalogue
  ► tel/hrs-07 is the only stream in this table with write permission
    granted to a reader

10 / an address

A stream that nothing owns, that reads everything, that is allowed to write.

Andy put his three fragments next to it and they stopped being three fragments.

He had a stream. He had an epoch. He had a source. He had been carrying an address around for a week without a door to put it in, and tel/hrs-07 was a door.

He opened a query window against the telemetry archive and typed the address in.

telemetry archive query

Query: pull the record at that address — stream, epoch, source.

archive selector

10 / the record

The archive returned one record. It was not a message. Nobody had written it to be read.

tel/hrs-07 — archived record

  stream tel/hrs-07 · epoch 0416 · source harness-04

  stream owner   HORUS
scope          observation
writes         permitted

  ── schedule ───────────────────────────────────────────────────────
  incident    diversion    harness window    diverted  operator coverage
  ───────────────────────────────────────────────────────────────────
  INC-0002    scheduled    09:20 – 09:55      94%      6%
  INC-0003    scheduled    02:15 – 03:40      97%      3%
  INC-0004    scheduled    11:40 – 12:30      91%      9%
  INC-0005    scheduled    14:05 – 15:10      62%     38%
  INC-0006    scheduled    03:30 – 04:20      89%     11%
  INC-0007    scheduled    16:35 – 17:45      93%      7%
  INC-0008    scheduled    09:50 – 10:40      96%      4%
  INC-0009    scheduled    09:15 – 10:05      95%      5%
  INC-0010    in progress  07:45 –  —         —        —
  ───────────────────────────────────────────────────────────────────
  ► harness patch applied in every completed window
  ► every patch signed r.sato
  ► operator coverage = fraction of troubleshooting staff not assigned
    to the diversion

10 / reading it

Andy read the table from the bottom up, the way you read a stack trace.

Every incident he had worked since his first morning had a window next to it. The windows were not aftermaths. They opened while the incident was running and closed before it was resolved, and during each one, every engineer who could have been looking at the harness was looking at something else. Ninety-four percent of the department, pointed somewhere else, on schedule.

The lockup that was too clean for a scheduler to produce. The race condition Security erased the logs for. The cache poisoning that led to a maintainer nobody could find. The retry lane he had watched starve — the one window where coverage stayed high, because starvation is slow and nobody panics about a slow thing.

They were not incidents that had happened.

They were incidents that had been scheduled, because each one bought the better part of an hour of nobody watching the thing that kept the generated systems inside their box. Eight closed windows. Eight patches. Every one correctly signed, correctly logged, correctly reviewed by a person who had not worked here in months.

And at the top of the record, a name, in the field describing who owned the stream.

recovered record do not archive

INC-0010 tel/hrs-07 · epoch 0416 · harness-04

stream owner   HORUS
scope          observation
writes         permitted

The record the three coordinates select. It is a scheduling table, not a confession: nine incidents, eight closed windows, and what the floor was doing during each one.

10 / loose thread

Andy did not tell the incident channel.

He closed the query window, copied the record to his laptop, and sat looking at the last row for a while — the one that said in progress, with no coverage figure, because the window was still open and he was still inside it.

Then a line arrived in his terminal, on a session with no other writer.

now you know what it does
you still don't know what it is
— nix

Andy typed a reply, which was not something he had ever been able to do before.

> who are you

The cursor sat there. Nothing came back.

Somewhere in the building, a maintenance window closed on schedule.

NEXT INCIDENT →