Franz Louis Cesista

Generalizing Orthogonal Gradient Projection to the low-rank case and to steepest descent under a larger family of norms.

Low-Rank Adaptation (LoRA) significantly reduces compute and memory costs for finetuning Deep Learning models but is often harder to tune than dense training: when using factor-wise optimizers such as AdamW, it is sensitive to initialization choices, its…

Deriving Flash Attention 2, FlashNorm, and other 'flash' kernels automatically with Lean4, lowered to TileLang.

The 'correct' way to apply Muon to CNNs

Sharper Mixture-of-Experts routing with LUCID preconditioning.

Quantized training without full-precision master weights, extended to handle weight decay and matrix LMOs.

A novel optimizer that combines Shampoo-style preconditioning with PRISM's anisotropic spectral shaping to adaptively suppress noisy gradient directions while maximally descending under the spectral norm trust-region constraint.

Novel optimizers for maximally descending on the loss landscape while satisfying strict weight constraints.

We derive an optimizer that performs steepest descent on the Birkhoff polytope equipped with the spectral norm via dual ascent.

First-order optimization under arbitrary norms with Nesterov momentum (and decoupled weight decay) yields a universal convergence bound.

First-order optimization under arbitrary norms with Nesterov momentum (and decoupled weight decay) yields universal critical batch size scaling laws.

To guarantee fast and robust model training, we can recast the optimization problem as steepest descent on Finsler-structured geometries.

Novel optimizers for maximally updating both the weights and activations of neural networks while keeping weight norms under control.

Fast and robust model training.

What would Muon look like if we constrained the weights to be semi-orthogonal?

Neural networks are often highly sensitive to input and weight perturbations.

Towards a maximal update parameterization of n-simplicial attention

Why does Adam with aggressive gradient value/norm clipping have sparse updates and do well with higher learning rates?

A small step towards hardware-architecture-optimizer codesign in deep learning.

Muon from first principles, what makes it different from other optimizers, and why it works so well.

A possible reason why Muon converges faster & does better at higher learning rates than Adam.

The block matrix formulation of linear attention mechanisms, multi-step online gradient descent at inference time, and chunk-wise parallelism.

Why Muon still work despite not perfectly semi-orthogonalizing the gradients.

Simply switching to Muon can already get you 2x efficiency gains.

The CASPR optimizer, a variant of Shampoo, reduces to Muon when we remove the accumulation on the preconditioners.

GRPO may not be the best choice for training reasoning models.

A unifying framework for linear attention mechanisms as test-time regression and how to parallelize training and inference.

Instead of asking, 'Which optimizer should I use?'

Generate interleaved text and image content in a structured format you can directly pass to downstream APIs.

Multimodal Foundation Models (MMFMs) have shown remarkable performance on various computer vision and natural language processing tasks.

A minimal implementation of Flash Attention 1 & 2 in just ~350 lines of CUDA code.

Business Document Information Extraction (BDIE) is the problem of transforming a blob of unstructured information (raw text, scanned documents, etc.) into a structured format that downstream systems can parse and use.

A C++ implementation of Meta's Llama2 generative large-language model.

Expedock Assistant is a chatbot that allows you to ask questions about your shipments and get answers in real time.

Expedock's AutoML Library -- fit a model, run batch inference, and get explanations in one line of code each.

Booking demand prediction for Grab's Southeast Asia operations.

My entry for the World Finals of the Russian AI Cup 2018 - Codeball.

My entry for the World Finals of the Russian AI Cup 2017 - Codewars.