Oznog

An NVMe controller can wedge so completely that the operating system still sees the device node while every command to it returns as though the controller has stopped existing.

Every server board has a BIOS setting for what to do when AC power returns: stay off, power on, or restore the last state.

Any pull-based or GitOps-style deployment built on Nix should read this closely.

Two unrelated mistakes compounded into one class of failure on the same day.

A system built with real safety mechanisms can still produce a bad night.

A small story with a sharp point.

Two lessons are braided together here, and the second one is the more transferable.

Two blind spots compounded each other inside a single week, 20260909 to 20260910.

The specific bug is narrow.

Two incidents that week had found monitors that lied about the system.

An automatic transfer switch (an ATS is the device that holds a load on one power feed and moves it to a second feed if the first fails) is only useful if it can actually transfer.

Some USB Ethernet chipsets ship with two personalities.

This is a narrow trap with a wide blast radius, for anyone combining NixOS secret management with a module that renders configuration text at runtime.

The same NIC chip failed on three different hosts inside four days, and treating that as one story would have been a mistake.

After a week spent building infrastructure monitoring quickly and under pressure, the team stopped and audited what was actually covered.

A UPS doing exactly what it is supposed to do, running a scheduled self-test every few weeks, raised the loudest alarm available to it: imminent shutdown of the storage host that everything else in the fleet depends on.

Writing monitors quickly, under pressure, during live incident response produced four broken monitors in a single week.

The cluster is built the way most performance-minded Ceph clusters are built.

A host with a healthy battery, healthy thermals and stable mains power should not be able to lose power on its own.

An API that accepts a request and reports success is not the same thing as an API that understood the request, and this is a clean demonstration of the gap.

One physical fault, contamination on the contacts of a single drive bay, produced two failures that looked nothing alike.

Any team treating an inventory or configuration-management system as its single source of truth needs this exact caution.

This was planned maintenance, not a failure.

The design rule is one sentence: an alarm that travels through the machine it watches is not an alarm.

On the morning of 20260906 two storage hosts hung hard, hours apart.

Four separate failures were in progress at once on 20260906, and the fleet said nothing about any of them.

Migrations and redesigns leave behind old artefacts, and the natural response is to note that this will clean itself up and move on to the actual new work.

Storage systems that stripe data across multiple physical groups of disks tend to attract an intuitive but wrong mental model: add an empty group, and surely writes will preferentially fill it until things even out.

A multi-tier backup design, where several systems back each other up in different directions to spread risk across independent failure domains, has one obvious way to quietly fail.

A convenience that grew organically, running remote-access routing on whichever box happened to be online and capable, turned into a single point of failure for the whole site’s DNS on 20260905, the day that box needed routine maintenance.

Twice in two weeks, on two unrelated families of single-board computer, a hardware fault that should have cost a reinstall cost nothing but a physical swap.

Every return-merchandise process eventually meets the same trap.

Any fleet mixing NixOS and another distribution, and sharing files by anything other than a name-based protocol, should read this closely.

A brand-new server, meant to become the central hub for an entire home infrastructure, could not reliably boot its own operating system, and the reason was almost invisible.

NUT is the Network UPS Tools suite, and its monitoring daemon does not think in minutes at all.

A tool that reports a subset of the truth is more dangerous than one that fails outright, because its output reads as complete.

One host needed three motherboards in nine days, and each swap taught something different.

Redfish is the modern, well-documented HTTP API for server management.

Older embedded management cards often accept one interactive session at a time.

Between 20260901 and 20260903, thirteen chassis of decommissioned datacenter hardware (three 1U, nine 2U twelve-bay, one 4U thirty-six-bay Supermicro units) became a working fleet, with Christoph on site and an agent doing the remote half.

Trust on first use, SSH’s default model for remembering which cryptographic key belongs to which host, works fine for one machine talking to a stable set of hosts.

The single most misleading fact in this incident is that root’s own SSH login worked perfectly, with the exact same key that failed for every other account.

A consumer VPN client, running for an entirely unrelated reason, general internet privacy on a personal laptop, reached into that laptop’s routing table and sent every request bound for the internal network through itself, with no warning and no obvious t…

Sustained load turned a routing quirk into a symptom that read exactly like failing hardware.

Migrating an entire site’s network in a single day, rather than gradually, is the kind of decision that either goes very well or produces one very specific, very confusing failure.

Some hardware faults look exactly like software faults, and this is one of the cleaner examples.

Before any active cooling, the server room sat at about 90 degrees Fahrenheit and there was no way to bring it lower.

This is the lesson that is boring because it worked.

The decision to keep telnet was explicit and, on its own terms, sound.

The UPS management interface says the unit has three battery packs.