Henry Freed

This’ll probably be my last post on variable MoEs for a little while.

This guide is based on this excellent post by Shelby Jenkins, extended with clock locking and V/F curve offsets for undervolting.

Work in progress.

I’ve been working on variable-sized experts in MoEs (previous post) using a modified version of Megablocks.

Whole wheat bread that tastes like whole wheat bread, but isn’t dense Whole wheat bread gets a really bad rap.

A quick Thursday set of visualizations I’ve been working on pruning models a bit lately, mainly based off of this Nvidia paper.

(Work In Progress) I want to write up my work as I’m working on it, and not all at once.

Acknowledgements Thanks to: Sam Lehman for the conversations that got this project off the ground as well as reading drafts and feedback Craig Robinson for feedback on drafts Andrej Karpathy for NanoGPT, which this project is based on Trevor Gale, Deepak…

(Work In Progress) Adding Mixture of Experts support to Karpathy’s NanoGPT, MoE interpretability, general MoE Notes In progress: July 14, 2025-?

Acknowledgements Huge thanks to: Anthropic for Scaling Monosemanticity, Towards Monosemanticity, and Update on how we train SAEs.