snimu’s blog

In response to my post announcing my second modded-nanogpt medium world record, several people made suggestions for variations of or alternatives to the technique introduced in the article.

Adding the output of layer 11 to the final output latents in a learned, weighted sum leads to a separation of concerns for early model layers: it enables them to focus only on providing context to the next layer, without directly impacting the final predi…

I have achieved a (not yet official) modded-nanogpt medium track world record.

Adding more value-embeddings to modded-nanogpt led me to an (as of yet unofficial) modded-nanogpt medium record as seen in PR#119.

The modded-nanogpt medium-track makes use of multiple tricks to improve performance, which rely on learned scalar values for mixing two vectors: U-Net: at three points in the model, the residual is modified by an earlier value of the residual via a weight…

I’ve seen multiple papers where LLM training data included tags providing metadata, which improved the models’ downstream performance.

In my article about Model Stacking, I proposed a method for decentralized pre-training of models.

An LLM should never output anything but tool calls and their arguments.

I’ve read some complaints about Llama using too many special tokens.

Edited on: 2025-04-11; 2025-05-04.