Justin T Chiu

Branching rollouts give a tree replay buffer; the policy gradient is four expectations, and each one must be Rao-Blackwellized correctly

Token-level policy gradient is correct, average over batch tokens, and a packed batch is a replay buffer

Differentiating through optimization

Unf***ing Your RL Gradient Estimator

The math behind replay buffers

Brute-forcing VAE math

RL is better than SFT

Information-theoretic definitions of memorization

Sampling given character prefixes

Everything is a latent variable model

Some calculus busywork

Variational inference for a simple Ising model

First post