ezyang's blog

In Code review as human alignment, in the era of LLMs, I talked about how we should approach code review with other team members, where you have a pre-existing relationship and care about building human-to-human alignment over time.

How does PyTorch autograd deal with mutation?

I was coaching some juniors on effective use of AI coding agents (like, looking over their shoulder as they were prompting the LLMs) and one reoccurring theme was that the AI agents were demanding a lot more of their reading code skills, compared to how t…

Previously in AI-assisted programming for spmd_types, I mentioned that I have been enjoying using Sapling (the version control system Meta uses internally) to manage parallel agents on worktrees.

spmd_types is a type checker for distributed PyTorch programs.

A silent BC breaking change is a change of semantics in an API that is not immediately obvious; e.g., the old usages of the broken behavior won’t raise a compile or runtime error on upgrade.

I recently received the following question about vibe-coding for tlparse, a structured log parser for torch.compile (slightly edited for ease of reading): Hi Ed, I have some thoughts on vibe-coding tlparse and I’m curious what your opinions are.

A central thesis of sharding in types is that the backward sharding can be directly computed from the forward sharding.

DTensor has famously terrible eager mode performance; for example, this paper measured a 35-60% slowdown in end-to-end training performance with and without DTensor (with DTensor operations taking at least 7x longer than actually running the computation f…

Conventionally, a type system is something that classifies values into data types like float32 or int64.

Global SPMD (also known as the “global view”, exposed by code using DTensor or jax.Array) refers to writing multi-device code as if it was on a single device, with an orthogonal mechanism for expressing how these full tensors are distributed over multiple…

In Computing sharding with einsum, we worked an example of Megatron style tensor parallelism where we discover that the ordinary backwards formula for linear results in a pending reduction on grad_input, even though the input was replicated and no communi…

Mental arithmetic in grade school (e.g., memorizing your times tables) is typically justified on the grounds that facility in basic calculations makes it easier to focus on higher-level problems that require being able to do these manipulations.

This blog has lived on WordPress since it was initially created during a social challenge at MIT to write a blog post a week or pay up with beer.

Let’s suppose you asked an AI coding agent to “implement a CLI calculator”.

I’ve recently been doing a lot of both submitting and reviewing pull requests to PyTorch that were authored with substantial LLM assistance.

Famously, PyTorch and JAX don’t agree on how shardings should be represented: PyTorch takes a mesh-dim oriented view, where for each dimension in your device mesh, you specify what sharding should be applied; JAX takes a tensor-dim oriented view, where for

I have recently needed to draw the contents of high-dimensional (e.g., 4D and up) tensors where it is important to ensure that is clear how to identify each of the dimensions in the representation.

With contributions from Richard Zou.

When training large scale LLMs, there is a large assortment of parallelization strategies which you can employ to scale your training runs to work on more GPUs.

CuTe is a C++ library that aims to make dealing with complicated indexing easier.

The purpose of this post is to sum up, in one place, the state of torch.compile for training as of August 2025.

A lot of strong engineers that I know haven’t really taken a serious look at AI coding; they’ve used LLMs to ask questions or write simple scripts and appreciate that it is a useful tool, but haven’t actually tried building a nontrivial application entire…

Do you use an LLM for coding?

In my previous two posts “`Ways to use torch.compile <http://blog.ezyang.com/2024/11/ways-to-use-torch-compile/>`_” and “`Ways to use torch.export <http://blog.ezyang.com/2024/12/ways-to-use-torch-export/>`_”, I often said that PyTorch would be good for a…

Previously, I discussed the value proposition of torch.compile.

On the surface, the value proposition of torch.compile is simple: compile your PyTorch model and it runs X% faster.

Tensor libraries like PyTorch and JAX have developed compact and accelerated APIs for manipulating n-dimensional arrays.

One of the things that I learned in grad school is that even if you’ve picked an important and unsolved problem, you need some reason to believe it is solvable–especially if people have tried to solve it before!

One of the annoying things about scraping websites is bouncing back and forth between the browser where you are using Dev Tools to work out what selectors you should be using to scrape out data, and your actual scraping script, which is usually some batch…

I’m launching a new podcast, the PyTorch Developer Podcast.

At Facebook, we have an internal convention for tooling called “rage”.

PyTorch is a fairly large and active open source project, and sometimes we have people come to us and ask if there are any lessons from how we run PyTorch that they could apply to their own projects.

Years ago, Nadav Rotem related to me this story about why basic block procedures in Swift are not as good as they seem.

Greetings from 2024!

If this is your first time reading about PyTorch internals, you might want to check out my PyTorch internals post first.

For the longest time, I thought of implicit parameters and dynamic scoping were basically the same thing, since they both can be used to solve similar problems (e.g., the so called “configuration problem” where you need to plumb down some configuration de…

I’ve recently been working on a revamp of how we specify tensor shape formulas in PyTorch.

vmap is an interface popularized by JAX which offers you a vectorizing map.

This post is a long form essay version of a talk about PyTorch internals, that I gave at the PyTorch NYC meetup on May 14, 2019.

Some notes collected from a close read of Conal Elliot’s Compiling to Categories and The Simple Essence of Automatic Differentiation.

Long time readers of mine may be aware that I used a ThinkPad X61T for the past decade.

My name is Rahul Muttineni, CTO of TypeLead, working on building services around a language named Eta.

It’s been a year since I got my hood and gown and joined Facebook (where I’ve been working on PyTorch), but while I’ve been at Facebook Backpack hasn’t been sleeping; in fact, there’s been plenty of activity, more than I could have ever hoped for.

A run-time debugger allows you to see concrete values in a program, make changes to them, and continue running your program.

Raise your hand if you’ve ever put one of these commands in your continuous integration scripts: apt install somepackage pip install -r requirements.txt or pip install somepkg conda install blah cabal update or cabal install blah git clone https://github.…

The best and worst thing about semantic import versioning is that it makes BC-breaking changes hard.

JG: Joseph Gonzalez GG: Garth Gibson (CMU) DS: Dawn Song (UC Berkeley) JL: John Langford (Microsoft NY)j YQ: Yangqing Jia (Facebook) SB: Sarah Bird M: Moderator A: Audience M: This workshop is bringing together ML and systems.

The below is a transcript of a talk by Daniel Lo on BrainWave, at the ML Systems Workshop at NIPS'17.

The below is a transcript of a talk by Virginia Smith on MOCHA, at the ML Systems Workshop at NIPS'17.