INC-0008 Split Brain

08 / both of them

The partition lasted forty seconds. The argument lasted the rest of the day.

[ALERT] INC-0008 — Control plane partition, 40s
[ALERT] Two nodes accepted writes as primary
[CRITC] Divergent term history after rejoin
[INFO ] cluster size 7 · quorum 4
[INFO ] monitors reporting: 4/4
[WARN ] monitors agreeing: 0/4

Four monitors. Four different leaders. None of them wrong, exactly — each one had watched an election, counted the votes it could see, and written down the winner. Then the link came back and all four records landed in the same trace.

“Everyone’s telling the truth,” Andy said.

“Everyone always is.” Bob had his headphones around his neck, which meant he had been listening to the incident channel for a while before saying anything. “A partition doesn’t make anybody lie. It makes them all correct about a smaller world.”

He pointed at the header.

“Seven nodes, quorum of four. So out of those four claims, exactly one ever had the cluster behind it. The rest are three people winning elections in rooms with nobody in them.”

evidence retained

mon-04 recorded its observation in term t-43.
The observation made in term t-43 carried three votes.
Term t-42 carries two votes.
n-gamma was recorded with two votes.
n-delta’s claim carried a single vote.
mon-03 recorded a single-vote claim.
The term t-44 observation did not carry a single vote.
mon-01 did not record term t-44.
n-beta was not the node that reached four votes.
The rejoin trace lists five readers. The cluster has four monitors.

messages incident channel

Ops 09:52

Both primaries accepted writes. We are going to be reconciling ledgers until Thursday.

Security 09:55

Treating as possible intrusion until proven otherwise.

Bob 09:58

It's a partition. Forty seconds of nobody being able to hear each other. Security can have it back when they find a packet that didn't belong.

Bob 10:02

Four categories now. Columns are monitors, nodes, terms. Rows are votes, terms, nodes. One node, one term and one vote count per monitor.

Bob 10:03

Find the claim that actually made quorum. Format: node, monitor, term, votes.

table election records

fault

Two primaries after partition

cluster

7 nodes · quorum 4

severity

SEV-1

Skip the grid, go to the answer

	mon-01	mon-02	mon-03	mon-04	n-alpha	n-beta	n-gamma	n-delta	t-41	t-42	t-43	t-44
v=1
v=2
v=3
v=4
t-41
t-42
t-43
t-44
n-alpha
n-beta
n-gamma
n-delta

answer input incident response

Question: Which node actually held the lease — recorded by which monitor, in which term, with how many votes?

operator answer

08 / after the partition

n-alpha, recorded by mon-02, in term t-44, with four votes.

Four out of seven. The only claim in the trace that a majority of the cluster ever agreed to. The other three nodes each won an election inside a fragment of the cluster that could not hear the rest of it, and each of those fragments went on serving writes with complete confidence for forty seconds.

Reconciling ran into the following week. Two ledgers, one truth, and a lot of customers who had been told different things about their own money.

rejoin-trace — INC-0008

                      ── link down 09:51:04 ──

  side A (4 nodes)                     side B (3 nodes)
  ┌───────────────────────┐            ┌───────────────────────┐
  │ n-alpha  term t-44    │            │ n-beta   term t-43    │
  │ votes 4/7   QUORUM ✓  │            │ votes 3/7   no quorum │
  │ accepting writes      │            │ accepting writes      │
  └───────────────────────┘            └───────────────────────┘
       n-gamma  t-42  2/7                   n-delta  t-41  1/7

                      ── link up   09:51:44 ──

  ► both sides present a term history
  ► term counters overlap, contents differ
  ► only t-44 was ever ratified

08 / loose thread

Andy wrote the reconciliation script, ran it, watched it work, and then went back to the rejoin trace because of the thing that had been bothering him since ten in the morning.

The cluster had four monitors. The trace had five readers.

readers on rejoin:
  mon-01       cluster/monitor
  mon-02       cluster/monitor
  mon-03       cluster/monitor
  mon-04       cluster/monitor
  obs/0        tel/hrs-07

He looked up tel/hrs-07 in the service catalogue. The catalogue had no entry. He asked in the platform channel. Someone said it was probably a legacy exporter. Someone else said everything was probably a legacy exporter.

The fifth reader had not written anything. It had only watched, from before the link went down until after it came back, which was more than any of the actual monitors had managed.

Andy wrote the namespace down on the same page as the other two things he had written down and did not understand.

NEXT INCIDENT →