Oznog

When we put Node0 in the open on 20260922, the Seed was a design: the smallest version of Node0 anyone can start from, as a ladder of rungs, with rung 0 in progress.

The Seed is the smallest self-hosted site we describe, and almost everything it does, you can already buy.

Node0 went online on 20260630.

Put plainly: nothing the site depends on runs on a machine that closes and leaves the building, and no secret stays on one.

Storage fails early, and it fails more than anything else.

Give the agents a place where a failure costs an afternoon.

What this layer is for # Every other layer assumes it has power and a room cool enough to run in.

Most of the lessons on this site ended in a fix to one machine.

Some infrastructure gaps are loud: something breaks, an alert fires, someone investigates.

Node0 measures its own electrical load twice, from the UPS units and from the PDUs, for the ordinary reason that both devices report it and both were already being polled.

None of these mismatches were dangerous on their own.

Partway through building the fleet’s alerting rules, a convention was adopted.

This one is worth including precisely because it is small.

This is a small lesson, but it is cheap to repeat and easy to avoid, which is why it was written down as a standing rule rather than handled as a one-off correction.

Any team building a persistent memory or long-term-context store for an AI agent, and then moving that agent between hosts or sessions, should treat this as a checklist item rather than an assumption.

Three different signals said this DNS cluster was healthy on 20260920, and all three were describing the wrong thing.

What this layer is for # Operations owns no rack position and runs no service of its own.

What this layer is for # Ceph’s replication and a firewall’s failover state can both behave correctly for months while nobody watches.

What this layer is for # Node0 is operated by agents, not only for them.

What this layer is for # Every layer below this one exists so somebody can type a name into a browser and get their paperwork back.

What this layer is for # Compute is four things a site of this size needs, and one it does not.

What this layer is for # Storage is where Node0 keeps everything that must survive a reboot, a drive failure, a whole-home event, or Christoph’s own mistakes.

What this layer is for # Bootstrap is what has to work before any other part of Node0 can come up on its own.

What this layer is for # The fabric is Node0’s data plane: the switches that move packets between racks, the VLANs (virtual LANs, separate broadcast domains carried over the same wires) that keep one purpose’s traffic away from another’s, and the bonding…

This page carries the detail the ladder on the seed page points at, in the design’s order, so a person and their agents can do each part without having done it before.

What this layer is for # The edge is the boundary between Node0 and the internet.

What this layer is for # Every other layer in this capture describes something that runs.

A remote console that still shows video is a console that will still be trusted, which is the whole problem here.

Security intuition and operational reality pulled in opposite directions here, and operational reality won for a specific, well-reasoned cause.

Scoping a maintenance silence to the name of the host you are about to power down is the obvious thing to do.

This is a small, precise example of a mistake that is easy to make with any service running inside a monitored host.

Some safety features fail by being too helpful, and this is a clean pair of examples on the same mechanism, four days apart.

The mechanism here is counterintuitive enough to be worth sitting with.

In this fleet, the inventory system is not documentation.

Two parity disks exist so a single drive’s death can pass with no drama.

A component can fail completely and keep announcing it once a day for five days without producing a single alert, if the only thing watching is a rule built to match one specific log line.

A whole class of network adapters in the fleet ran at a fraction of their rated speed, and nothing about it was visible from either end.

The whole power-return design rests on one command reaching each UPS during an outage: cut your output, and bring it back when mains returns.

The rule was not buggy.

Getting a fleet to come back on its own requires the UPS to cut its own output and restart it when mains returns, so that the return is a real AC loss and return event the BIOS has something to react to.

A number typed once into a monitoring rule, correct on the day it was written, became a landmine the moment the system it described changed shape.

Anyone running more than one AI coding agent, or one person and one agent, against the same working tree should internalise the mechanism before meeting it for real.

Cooling for the server room is driven by a smart thermostat, and the whole point of the setup is that it should decide from a remote sensor placed by the racks rather than from its own built-in sensor somewhere else.

The plan was reasonable when it was written.

Both quirks are worth knowing on their own.

An automated tuning system that stops reporting recommendations looks exactly like a system that has decided everything is already correct, and those two states require opposite responses.

The proximate cause was a mismatched or defective controller board.

A deploy script wrote a shared secrets folder with a restrictive mode and intended to relax it thirty lines later in the same run.

The original alert said one thing and only one thing: the cluster has been in a warning state for 15 minutes.

A monitoring rule can read its target correctly every single time and still misrepresent the system.