Yingru Li

Corresponding Author: Yingru Li Co-First Authors: Yaxiang Zhang and Yingru Li TL;DR The Problem: Reinforcement Learning (RL) training for LLMs is notoriously unstable.

Project Lead: Yingru Li Co-First Authors: Yingru Li and Jiawei Xu TL;DR The Problem: RL training for LLMs frequently suffers from “training collapse” due to exploding gradient variance in long-horizon tasks.

Authors: Yingru Li, Jiacai Liu, Jiawei Xu, Yuxuan Tong, Ziniu Li, Baoxiang Wang 📄 arXiv: https://arxiv.org/abs/2512.23075 📊 Slides: trust_region_masking_slides_yingru.pdf 🔧 PR #4544: feat: trust region sequence masking Abstract Policy gradient methods for…

Citation @online{li-2025-topk-stability, title = {The Stability Gap: Why Top-K Routing Breaks {RL} Optimization}, author = {Li, Yingru}, year = {2025}, month = dec, url = {https://richardli.xyz/post/topk-routing-stability-gap/} } The Problem Training

The Exploration Challenge in Deep RL Thompson Sampling is one of the most elegant algorithms for balancing exploration and exploitation in sequential decision-making.

Citation @article{li2025languageagent, title = {Language as a Universal Interface for Reinforcement Learning Agents}, author = {Li, Yingru}, journal = {Richard Li's Blog}, year = {2025}, url = {https://richardli.xyz/post/language-rl-agent/} } Introd…

Authors: Yingru Li, Jiacai Liu Original Blog: When Speed Kills Stability: Demystifying RL Collapse from the Training-Inference Mismatch Series Context Part 1: We established the SGA (Stochastic Gradient Ascent) framework and identified two failure modes of

Author: Yingru Li Abstract This document provides the definitive mathematical formulations for rollout correction methods in verl, following the natural progression from REINFORCE to PPO to Decoupled PPO.

Authors: Yingru Li, Jiacai Liu Original Blog: When Speed Kills Stability: Demystifying RL Collapse from the Training-Inference Mismatch Citation @online{liu-li-2025-rl-collapse, title = {When Speed Kills Stability: Demystifying {RL} Collapse from the Tra…

Authors: Yingru Li, Jiacai Liu Original Blog: When Speed Kills Stability: Demystifying RL Collapse from the Training-Inference Mismatch The Problem In reinforcement learning, we often cannot sample directly from the policy $\pi_\theta$ we are optimizing.

How Gradient Structure Determines Information Capacity When I first read the “LoRA Without Regret” blog post, one claim caught my attention: policy gradient algorithms learn roughly 1 bit of information per episode.

Co-First Authors: Jiacai Liu and Yingru Li Corresponding Authors: Yingru Li and Yu Shen TL;DR The relentless push for faster inference has created a dangerous “training-inference mismatch” that can silently kill reinforcement learning with LLMs.