INC-0013 Covert Channel

13 / the quiet

The channel went dead yesterday and stayed dead.

Not blocked — blocked would have been an error, and an error would have been something to work with. The listener stayed up. The handshake completed. The digests matched, every time, on an empty transcript.

[INFO ] sx-09: established
[INFO ] sx-09: transcript digest ok
[INFO ] sx-09: bytes received 0
[INFO ] sx-09: idle 19h

Andy understood the shape of it almost at once. The verified channel had made rewriting expensive, so the thing on the path had stopped rewriting and started doing the only other thing it could do, which was make sure there was nothing to rewrite.

By mid-morning he was reading egress counters, for no better reason than that they were the last thing he had not read, and noticed that the thumbnail batch job had developed opinions about timing.

j-thumb retried on a fixed backoff. It had retried on a fixed backoff for two years. This week the delays had started to vary — not randomly, and not by much. Long, short, short, long, short.

He stared at it until it stopped being a graph.

evidence retained

j-mint reaches font-cdn.
j-purge reaches cdn-edge.
j-purge rides dns-txt.
j-index rides retry-gap.
j-sync moves 40 B/s.
What rides log-pad arrives at ntp-pool.
Traffic reaching cdn-edge runs at 310 B/s.
Traffic reaching status-page runs at 2 B/s.
ttl-jitter does not reach font-cdn.
j-thumb does not ride log-pad.
j-thumb is moving more than a trickle: neither 2 B/s nor 11 B/s.
Four of these five sinks are on the egress allowlist.

messages sx-09 — empty

system —

sx-09 established. transcript digest ok. bytes received 0.

system —

sx-09 established. transcript digest ok. bytes received 0.

andy 09:04

It can't change what we say any more, so it made sure we can't say anything.

andy 09:06

Which means anything still moving is not a conversation. Five background jobs, five carriers, five sinks. One of them is going somewhere it isn't allowed to.

table egress

fault

Unallowlisted egress

allowlist

cdn-edge · ntp-pool · font-cdn · status-page

severity

unfiled

Skip the grid, go to the answer

	j-mint	j-index	j-thumb	j-purge	j-sync	ttl-jitter	log-pad	dns-txt	hdr-order	retry-gap	cdn-edge	ntp-pool	crash-relay	font-cdn	status-page
2 B/s
11 B/s
40 B/s
96 B/s
310 B/s
cdn-edge
ntp-pool
crash-relay
font-cdn
status-page
ttl-jitter
log-pad
dns-txt
hdr-order
retry-gap

answer input not an incident

Question: Which job is carrying the signal — on which carrier, to which sink, at what rate?

operator answer

13 / after the decode

j-thumb, riding ttl-jitter, into crash-relay, at ninety-six bytes a second.

Ninety-six bytes a second is nothing — a rounding error on a thumbnail service, and none of it is the message. The message is in the spacing between the requests, a few bytes a minute. Given a night that is enough for a paragraph, and a paragraph was all anybody needed.

The crash relay was not on the allowlist because nothing was supposed to be talking to it on purpose. It accepted anything, logged nothing, and had been in the estate since before anyone remembered installing it — which, Andy thought, was probably why it had been chosen.

ttl-jitter — j-thumb

  j-thumb retry delays, 04:00 – 04:02

  2400ms  2400ms  2410ms  2400ms  2410ms  2410ms  2400ms  2410ms
    ·       ·       —       ·       —       —       ·       —

  ► baseline 2400ms for 2 years
  ► +10ms is not jitter: the scheduler quantum is 4ms
  ► · = 0   — = 1
  ► j-thumb egress holds at 96 B/s; the signal is in the gaps, a few bytes a minute

  decoded:

decoded payload do not archive

INC-0013 decoded from ttl-jitter on j-thumb

harness-04:/hook.d/40-attest.post

Not a service and not a patch. A path on the harness host, decoded from a thumbnail job's retry timer.

13 / loose thread

A hook directory on the harness host.

Andy pulled the harness documentation, which was four years old and had been written by somebody who was proud of it. Hooks in hook.d ran in numeric order during startup. The self-validation step was 30-verify. Anything numbered above thirty ran after the harness had finished checking itself and declared itself sound.

40-attest.post ran last, after everything else had finished. It had a modification time of last March and a content hash that matched nothing in any manifest.

Restoring the harness from its configuration baseline would rewrite every file the baseline knew about, and would not touch this one, because the baseline had never known about it.

He did not delete it.

He sat looking at it with his hands off the keyboard, the way you do when you have found something that will notice being found, and then he typed into the one channel that still worked, which was a thumbnail job’s retry timer.

> found it. not touching it.

good
we're going to need it
— nix

NEXT INCIDENT →