← All stories

No Response Required


21:40. The payroll batch for Friday comes back half a country.

Not a failure. Failures I can work with. A failure is a complaint and a complaint has a ticket and a ticket has an SLA clock and somewhere down that chain a human being picks up. Half a batch is something else. Half a batch is two hundred thousand people whose rent money is sitting in a state that neither our bank nor their bank will describe, and every explanation I can offer a customer is arithmetic I don’t control.

I have the west Nairobi view up on the second monitor, the one I keep for nights like this, and I can watch the loss shape itself. My covering block is 197.159.0.0/16. I announce forty /24s out of it, plus a handful of /25s for customers who needed addressing that doesn’t fit in a polite subnet. The /24s are moving. The /25s are intermittent, and the intermittence has a geography: Mombasa drops them, Kampala delivers them, Johannesburg delivers one and drops two, and the bank’s failover range, 197.159.64.0/25, is dark at enough of the transit tier that the bank’s payment gateway is up and talking to nobody.

Invalid by maxLength. I can see it from space.

The ROA that does it was mine. I set it in 2019, the week we got the /16, covering block, maxLength 24, because in 2019 we did not deaggregate and the whole idea of publishing an authorization for addresses you were not yet announcing felt like the kind of superstition you do in case the network grows teeth. The network grew teeth. Last year the bank asked for a dedicated failover range, and a /25 inside a covering ROA that says maxLength 24 is Invalid on every relying party that holds the object. Not dropped with a notice. Not bounced back with a reason code. Dropped. Route origin validation has no return path to the prefix owner. The people who throw my packets away do not know they threw my packets away, and I do not know who they are, and there is no ticket.

I open a ticket anyway. Internal, against my own board. Title: ROV telemetry request, CA production subtree. Then I call Mutiso.

He picks up on the second ring, which means he was at his desk, which means he already knew I would call. His organization certifies the production subtree for East African route origin authorizations. He ran the migration. He is, by any metric his board will read in three weeks, the first on this continent to do it, and he has carried that around since the day the first object signed.

“Njeri,” he says. “I have your dashboard open as well.”

“Then you know about the bank.”

“I know about the bank. I also know about the university in Dar, and a content network in Lagos, and a mobile-money aggregator who filed with the regulator yesterday using the word sabotage.” He says it evenly. He is not lying. I can hear his room on the line, a fan and nobody typing.

I pull the ROA object out of the repository and verify the signature chain in front of him, live, my console mirrored. The object is well-formed, the publication point is clean, the signing certificate chains to the new composite suite without a gap. My own tooling, running against the repository, derives for 197.159.0.0/16 a validated payload set of forty-five entries: the covering authorization, the forty /24 authorizations, the four /25 authorizations, maxLength where maxLength belongs. Forty-five. I count them on air.

“Run it again through a relying party that shipped before the migration,” he says.

I have. That is the problem. I have a vantage point nobody should have to build: a shelf of old validator images, a Raspberry Pi farm I keep mostly out of habit, and three upstreams who let me pipe collector feeds off their route servers. Across that shelf, the same repository and the same bytes produce three answers.

Updated composite-aware validator: forty-five payload objects for my block. Correct.

A validator two major versions back, the build most of the tier-2 transit in Africa is still running, the build their procurement cycles blessed in 2023: one object. The 2019 authorization, still on the old chain, still legible, maxLength 24. Everything else I signed last month lives under the composite-signed intermediate and simply does not exist for it. My /24s fall to unknown on that fleet, and unknown is fail-open, so they pass, which is the only reason this week is not a total loss. My /25s fall to Invalid, because the one object that fleet can still read says maxLength 24, and Invalid is the one state every relying party in my shelf agrees on dropping.

And then the third answer. I have re-checked it four times now. My hands are still cold. A mid-tier build, partway through its own transition, derives twelve objects from the block. Twelve. Eleven of them I can reconcile if I squint: overlapping authorizations from the publication lag, the covering object double-counted across two manifest cycles. The twelfth is a covering authorization for the whole block with maxLength 28, attached to an origin ASN that is not mine, derived from repository bytes that contain no such string.

“You’re seeing it,” I say.

“I’m seeing it,” Mutiso says. “Three relying parties, one repository, three payload sets. Two of them defensible on the letter of the draft. One of them I cannot reconstruct from the data, and I have tried, Njeri. I have the source of that build open on my other screen and I am not going to pretend I understand where the field comes from.”

The transition documents, the ones his own working group published, call this systemic risk. They say, in a sentence I have read forty times, that relying parties SHOULD report payload-set divergence back to the certificate authority so the authority can distinguish a bad object from a bad parser. SHOULD. Not SHALL. And in the next paragraph the same documents congratulate the industry on composite ML-DSA migration and note, correctly, that mixed certification chains will persist through the transition because nobody retires a working validator over a signature algorithm.

“Publish the equivalence telemetry,” I say. “Your side already has it. You run reference relying parties against your own production point every six hours. I’ve seen the cron entries. Publish the outputs next to the objects. If a validator in the wild derives twelve payloads from a tree that yields forty-five, I want that number sitting in public where the tier-2s can see their own shadows.”

“The spec says SHOULD.”

“The spec says a lot of things, Mutiso.”

“Equivalence telemetry is advisory. Publication is advisory. My legal people have been clear with me since the pilot.” He pauses, and I hear him turn a page or click a calendar. “I have a board review on the fourteenth. The migration metrics are green. Every object we signed is published and every object we published verifies against our own reference validators. What the field does with them is, by the plain text, the field’s affair.”

On my board view, 197.159.64.0/25 goes dark in Nairobi, lights in Kigali, dark again. The bank’s clearing window closes in forty minutes. Somewhere in the bank’s operations floor a person is going to open a console and find no route and no error, and have nobody to call, and they will escalate upward as a payments problem, and by morning this will be an incident with a name and an invoice.

“There is a third thing,” I say. “And I want you to look at your collectors before you answer, not your calendar.”

I put it on the shared view. 197.159.64.0/24, origin AS32771, which is my ASN, announced forty minutes ago from Bahari Telecom’s peering edge. Not my router, not my upstreams. My origin, pasted into someone else’s path, covering the exact space the bank cannot reach, and graded Valid by every relying party that can read either generation of my authorizations, because the covering authorization allows a twenty-four and the origin matches and route origin validation was never built to ask whose router is speaking for an ASN, only whether the ASN is entitled to the addresses.

Mutiso is quiet for four seconds. I count them.

“That,” he says, “is outside the scope of the repository.”

“That is outside the scope of the repository.”

“I will note it in the migration log as an observation from the operator of the affected space.”

“Mutiso.”

“It is the only action available to me.”

I know it is. I have known it for forty minutes. I called instead of filing. A filing goes into a queue and comes back with a reference number, and the reference number is true and useless. His logs are true. My logs are true. The bank’s customers are out of rent money for the third night because two true logs and one false announcement meet inside a system that grades all three on a signature and a field and stops there.

23:50. I make tea I do not drink. I walk the shelf again, slower, writing down what each build actually holds for my block, not what it should hold. Updated fleet: forty-five payloads, my /25s Valid there, and those are the networks where the bank’s gateway answers. Old fleet: one payload, my /25s Invalid, my /24s unknown and therefore passing, Bahari’s /24 Valid because a /24 satisfies maxLength 24 and my origin satisfies my own authorization. Buggy fleet: twelve payloads, and among the twelve a covering authorization for the whole block with maxLength 28, attached to an origin ASN that is not mine, which would let that ASN carve my block down to /28 and grade every piece of it Valid. Nobody else has derived it yet. I do not know how long that holds.

The arithmetic of my own making: I published maxLength 24 in 2019. The bank needed a /25 in 2024. I did not supersede the covering authorization before the migration, because the migration re-signing queue reordered my publication order and the covering object went out under the composite chain while the object that would have replaced it, maxLength 25, sat in my outbox waiting on a publication window that the migration froze. I have the receipt for that. It is my error, sitting in my logs, doing what errors do.

What I did not do is announce anything Bahari is entitled to. My packets are signed right by my measure and dropped right by theirs, and the space is being spoken for by a party with the right signature and the wrong equipment behind it, and every validator on earth will tell you the route is clean.

01:20. The bank’s clearing window fails. I know it before the call comes because I watch the /25 go dark in four more cities at once and stay dark.

I do not sleep. I make a second cup and leave it beside the first. I pull the transition document up and read the section I have been avoiding, the one about mixed chains persisting, the one that says, in the register these documents use when they are describing a problem they have decided not to solve, that operators MAY reduce exposure by consolidating authorizations to a single certification chain where topology permits. Where topology permits. The topology is that half of my transit tier is reading one chain and a quarter is reading a chain that invents fields, and the consolidation they are describing would require me to pull my authorizations out of the chain that half of the continent can verify, which would take my /24s from unknown to unknown and buy me nothing, or to pull them out of the chain the other half can verify, which is the same trade from the other side. There is no consolidation. The armor I am wearing locks half the doors on my own traffic and leaves the other half unlatched for Bahari.

04:05. Bahari announces 197.159.64.0/24 again, and now 197.159.80.0/24, and 197.159.112.0/24, three of my forty, the three with the worst delivery on the old fleet. Same origin, same clean grade, from a peering edge I have never touched.

I call Mutiso at 05:30 and he answers, which means he has not left either, and I tell him what I am about to do, and he says my name once, the way a man checks a form field before he stops being able to change it.

“Board review is the fourteenth,” I say.

“It is.”

“Then you’ll want the migration log to say the operator of the affected space reduced her authorizations to zero at 06:00 and the payload set for the block derived by every relying party fell to zero and no route was dropped as a result.”

“Njeri, if you withdraw, your prefixes go to unknown everywhere.”

“I know what they go to.”

“Unknown is not a grade. It is the absence of one. Any network that wants to filter you later will have nothing to filter on.”

“There is nothing to filter on now. Half of them are filtering on a 2019 object I would revoke this minute if revocation propagated faster than Bahari can route. The other half are grading me Valid and my traffic is still late because Valid doesn’t survive a best-path loss to an incumbent with more peering. And the third half is grading me twelve objects I’ve never signed and I don’t know what they think I am.” I have the console up. My authorization objects are listed by serial, forty-five of them. All mine, all correct by my own keys, each one a spot where some stranger decides whether my packets exist. “I would rather be ungraded than graded wrong. The unknowns pass. Nobody drops what they can’t grade. I am staking the whole block on it.”

He does not tell me not to. He says, “For the record, the authority will observe the withdrawal and will not object to reissuance at any time.”

“For the record.”

“I am required to say it.”

“I know you are.”

06:00. I revoke the covering authorization. I revoke the forty /24s. I revoke the /25s, including the bank’s, including the ones nobody was reaching anyway. Serial by serial, the objects come out of the repository, and the manifests update, and the validators that can see my chain at all watch the payload set for 197.159.0.0/16 go from forty-five to zero, and from one to zero, and from twelve to zero. I was never the source of the twelfth object, so its disappearance is not mine to explain.

Then I watch the collectors.

I have four feeds open, four vantage points, and a box in Johannesburg that counts as its own weather system. For about ninety seconds there is nothing new, because that is how long it takes the withdrawal to reach the relying parties and the relying parties to reach their feeders. And then, one by one, in an order that has nothing to do with importance, the block comes back.

197.159.0.0/24. Origin AS32771. Unknown, because there is no authorization to grade it against, and unknown is the state no validator drops. Then the next /24. Then the bank’s /25, the one that was dark in fourteen cities at midnight, back in the table at every one of them, origin mine, no armor and no signature, only my word and a physical path that has always been there.

Bahari’s three announcements are still up. They are still origin AS32771. They are now also unknown, so they are not being dropped either, and the competition between their announcements and mine has gone back to being a routing question: path length, local preference, who peered with whom. My /25 is more specific than their covering /24, so in every table that now holds us both, longest prefix match sends the bank’s traffic to my routers.

06:11. The bank’s test transaction clears. I know because their monitoring posts a green in the shared channel we are not supposed to use for green things, and because 197.159.64.0/25 is lit from Lagos to Seattle for the first time in three nights.

Mutiso is still on the line. I have not hung up and neither has he.

“I’m logging the withdrawal,” he says. “Timestamp, serials, reason code.”

“There isn’t a reason code for this.”

“I’ll use operator judgment.”

“That’s a field they let you invent?”

“We invent several, in the small hours.”

I should say something to him. He is not my enemy and he knows it and I know it, and in three weeks he will sit in front of a board and show them green migration metrics and a continent’s worth of route operators who can now, if their procurement ever catches up to 2026, verify a signature their own validators cannot parse, and somewhere in that same deck, if anyone reads the operator channels, there will be a note from the small hours about an affected space that chose to stop being signed rather than keep being graded. He will not put that on a slide. I would not either.

The covering /24 is on my board, whole, origin mine, no authorization behind it. The repository lists my block as empty. Every collector I can reach reports the same empty set from four different directions.

06:14. The window goes gray and my tea is cold and I file the internal ticket I opened at 21:40 against my own board, against nobody, the one with no responder and no SLA, and I close it.

No response required.

← Previous
The Index
Next →
The Weekly Trust Retirement Call