Severely Theoretical

There seems to be a lot of recent interest in and excitement about the promise of “agentic AI” (tools like Claude Code or Cursor) for improving productivity in scientific research.

Over the last couple of months, I’ve been continually training the pretrained Llama-3.1-8B model (with a context length of 8192 tokens) for 809B tokens.

One of the pitfalls in scientific writing in general is the danger of prematurely describing the basic observations of a field in theory-laden terms.

If the doors of perception were cleansed every thing would appear to man as it is, Infinite.

An interesting detail in the recently released Llama-3 technical report has caught my eye (p. 8): This has caught my eye, since I had noted the same phenomenon in a previous post about the Chinchilla scaling laws (more than two years ago) to argue for tra…

I’m a bit late to the fray as usual, but I wanted to write a short post about Sora.

I recently came across this beautiful short blog post by James Betker (who works at OpenAI), arguing that the thing that really determines the capabilities and, more generally, the behavior of a machine learning model is not its architecture, it’s not the…

One of the recurring themes in Hans Moravec’s prescient book, Robot: Mere Machine to Transcendent Mind (first published in 1999), is how practically important problems (e.g. agile robot navigation in the real world) become tractable more or less automatic…

While working on some generative video models recently, I had a moment of epiphany about hallucinations in generative models.

GPT-3.5 seems to be surprisingly bad at answering basic factual questions about famous writers and famous works of literature.