Jonah's

Introduction The goal of this blog is to better understand the progression that led to KDA and, by building some intuition for each step along the way, think about where this family of recurrent attention might go next.

Introduction The goal of this blogpost is to walk through the whole lifetime of a deepseek-ai/DeepSeek-V4.1-Flash request as it relates to the KV cache.

Introduction The goal of this blog post is to explain four implementations of context parallelism.

Intro In RL, every policy update requires moving newly updated weights from trainer to inference ranks.

Introduction I recently built a small (mainly pedagogical) distributed training engine to understand how different forms of model parallelism work in practice.

Introduction What is being asked in these experiments is if some future information, in the form of a rollout being successful or unsuccessful, is "leaked" (in the form of entropy over time) somewhere early enough in a completion that by intervening mid-g…

Intro This post is about the bubbles in you see in your Nsys traces and a couple interesting methods (tvm-ffi and Nvidia's Programmatic Dependent Launch) that aim to close those gaps.

Introduction The core function of an epilogue is to write data to gmem.

Intro In a recent blog, in part we spoke about how a given tile scheduler might affect the speed of your kernel.

Motivating the issue A matrix multiply is embarrassingly parallel.

Motivating the issue A lot of the blogs that I've read on swizzling don't motivate the problem in a way that I can understand.