Rachit Singh

Mixture of Experts (MoEs) are very popular.

$$ \newcommand{\on}[1]{\operatorname{#1}} \newcommand{\dmodel}{d_{\on{model}}} \newcommand{\dexpert}{d_{\on{expert}}} $$ MoEs are a new major paradigm for scaling models upwards without necessarily increasing compute costs, and they seem to work quite wel…

Hi, I'm Rachit.

Applied research teams slow down not because they lack good ideas but because of simpler issues: unclear metrics, confusing experiment design, poor iteration loops, and vague takeaways.

$\newcommand{\on}[1]{\operatorname{#1}}$ I think I should be writing down some of the things I learn a little more frequently, so I'll try with a small note here.

Recently at work I had a side quest on the following task: choose a binning scheme for some 1-dimensional data that has an annoying distribution.

One of the best things about LLMs is that the cost of writing bash or zsh has gone to 0.

Gardner was so serious about this learning imperative, so determined that the message would get through, that he wrote the speech out in advance because he wanted “every sentence to hit its target.”

macOS has a few UX differences from Windows that can make it difficult to switch from Windows to macOS.

When disagreeing with someone about a decision or a fact, it's useful to ask someone why rather than just presenting your own view.

This post is a work in progress.

It turns out it's really easy to add a click-to-expand element (also a dropdown or accordion element) in Markdown + HTML without any JavaScript, since it's part of the spec.

Setting up CUDA on WSL2 probably costs you a little bit of performance but gives you a lot of flexibility - you're essentially running a VM with (some) Microsoft support.

Lately I've been tinkering with optimizing Sasha's llama2.rs, a fast Rust port of Karpathy's llama2.c.

I recently heard that one of my high school mentors, Dr.

In the few areas where I have a clue about what people are thinking about, there's words that make me wonder whether this person is either clueless or assuming I'm clueless.

Pandas added support for nullable float32 and float64 datatypes in the past few years (Float32 and Float64 respectively)1, but there's a lot of footguns, so it feels kind of bolted on.

When teaching someone how to do something, don't say that "it's easy".

This one is for the poor soul who is attempting to push their code to Github right now via Jetblue's excellent ViaSat wifi, but confused about why it's hanging: It's because JetBlue blocks connections over port 22 (i.e.

Last updated: November 1st, 2023 Windows Subsystem for Linux (WSL) is my main programming environment, after giving up Windows in 2015.

This is probably an idea familiar to most researchers (and really anyone who produces written output for their job), but writing it up as you go has huge advantages over writing it up at the end.

Checked in code and commits are highly static artifacts intended for reliability.

I completely failed to achieve my goal of writing something non-technical at least once a month, so here we're trying not to fail with abandon.

I spent an unfortunate amount of time writing and re-writing my recent blog post about Jupyter notebooks, mostly double checking that I liked the overall style and tone.

I spend roughly half of my programming time working in Jupyter notebooks1.

These are things I always forget but need to be really comfortable working in a remote server.

If you're like me and have spun up a t4g.small during AWS's free trial period, you might want to run docker and docker compose.

This post covers the Langevin equation, a stochastic differential equation that models the dynamics of particles in Brownian motion1.

In class we recently discussed the simplified elastic rod model for polymers, which assumes that polymers can be modeled as an inextensible rod, i.e. that the length of the rod doesn't change, and that the twist of the polymer is ignorable (possibly becau…

This isn't an exhaustive list, and I will inevitably forget some papers.

Random sampling I've been working lately to implement random samplers from a number of distributions in PyTorch, both on CPU and CUDA.

tldr: The ubiquitous isotropic Gaussian prior for generative models doesn't make sense / doesn't work, which motivates work on priors.

I'm starting this blog to share research ideas that I have, and some solutions to problems I find along the way.