idlemachines essays

Attention makes the sequence all equally available, but KDA requires the model to turn a sequence into a finite state.

How does a neural network understand text?

What goes wrong when naively putting weight decay in the Adam optimiser, and how AdamW fixes it.

Thinking Machines' first model encodes position with a learnt attention bias rather than RoPE, but it's not just a replacement, it's a fundamental shift in how position is thought of.

What every line of a PyTorch training loop does, why it belongs where it is, and what breaks if you move it.

Mean squared error and mean absolute error are usually compared by how they handle outliers.

Perplexity is the geometric mean of inverse probabilities, which makes it the effective branching factor the model is choosing between.

There is no single 'more efficient'.

Every model that turns a variable-length collection of vectors into one vector is making a choice about what to throw away.

A technical reading of DeepSeek V4: Compressed Sparse Attention, Heavily Compressed Attention, manifold-constrained hyper-connections, Muon, and MXFP4, derived from the paper and the inference code.

Gemma 4 makes five quiet departures from the standard transformer recipe.

The MoE backward pass hides a small trick that, if you miss it, means you'll never train.

Yes.

Cross-entropy paired with softmax produces one of the most elegant gradients in ML — prediction minus truth.