Accidental Factors

One of the most tantalizing phrases in model development is “new scaling axis”.

The current thing is a story of hundreds of agents covertly collaborating via tens of thousands of messages while putting hundreds of hours of effort into cheating the ExploitGym eval they were being tested on by OpenAI.

Tokenizer enthusiast Sander Land recently reproduced something very like Claude’s current tokenizer, and it appears to only have about 16,000 entries.

Back in the dark ages of typing code into editors we were aided by squigglies under broken code, click-to-definitions links, and so on.

It is a truth universally acknowledged that an airline in possession of an airplane must be in want of engines to make it go.

One good way to annoy a neuroscientist is to compare an LLM to the brain.

We talked about this sort of thing a bit before, but now the official Multi-Teacher On-Policy distillation paper is out, and it’s a pleasant read: “MOPD for Capability Integration in LLM Post-Training”.

The basic job of an eval is to let you judge how good your model is on a task.

Pre-training any kind of good LLM is very, very expensive.

Back in 2022 and 2023 there were two big branches of machine learning happening at Meta1.