blank

Softmax backward is a Jacobian-vector product.

The source code for this post is available on GitHub: shreyansh26/MLSys-Experiments/decompose-k.

A useful map for KV cache work: are we selecting existing token states, or synthesizing new compact states?

Paper: Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models Official implementation: DeepSeek-AI/Engram Attention Context mixing Self-attention links tokens inside the current sequence and carries global context forward.

Paper: Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers Official implementation: facebookresearch/PhysicsLM4 K = 4the short causal window used in the main Canon implementation DKdepthwise parameter cost, instead of t…

Papers: Hyper-Connections and mHC: Manifold-Constrained Hyper-Connections n = 4common residual-stream expansion in the mHC experiments 3000 -> 1.6reported composite gain reduction from HC to mHC 6.7%training overhead after fused kernels and recomputation…

The source code for this post is available on GitHub.

Paper: Arxiv Link Video: YouTube Link Annotated Paper: Github repo Key Questions Large Language Models can store knowledge which can be extracted with question-answering.

Simplified illustration of Multi-Head Attention (MHA), Grouped-Query Attention (GQA), Multi-Query Attention (MQA), and Multi-head Latent Attention (MLA).

Source: GPT-4o image generation Forward Pass Recap First, let’s write down the forward pass for a single input vector \(x\) (a row from \(X\)) of dimension \(N\): \[y = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} * \gamma + \beta\] We can break down the fo…