Zhiting’s space

Overview / Takeaway Piper is a PyTorch distributed training system that separates strategy specification from runtime execution, letting users express composed strategies such as PP + DP/EP + ZeRO without hard-coding each combination into the framework.

Attention math Why $frac{1}{\sqrt(dk)}$ X, Y are two independent random variable.

Sharpen the Spec, Cut the Code: A Case for Generative File System with SYSSPEC link Design (1) Functionality specifications, which use concepts like Hoare logic (pre/post-conditions) and invariants to describe the behavior of individual modules.

PerfTracker: Online Performance Troubleshooting for Large-scale Model Training in Production link Observation LMT function (Python functions, GPU/CPU kernel functions, memory operations, etc.) executions exhibit two significant characteristics.

Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models link Motivation Contexts should function not as concise summaries, but as comprehensive, evolving playbooks—detailed, inclusive, and rich with domain insights.

GEPA: REFLECTIVE PROMPT EVOLUTION CAN OUTPERFORM REINFORCEMENT LEARNING link Motivation How can we extract maximal learning signal from every expensive rollout to enable effective adaptation of complex, modular AI systems in low-data or budgetconstrained…

Note to concepts in Mathematical Foundations of Reinforcement Learning link State: describes the agent’s status with respect to the environment.

Fault-tolerant training A driver process identifies the failed pod by heartbeat.

Mycroft: Tracing Dependencies in Collective Communication Towards Reliable LLM Training link Summary: Mycroft, a novel, lightweight distributed tracing and root cause analysis system designed to enhance reliability in Large Language Model (LLM) training by

Robust LLM Training Infrastructure at ByteDance link Control plane Robust controller Robust controller react to events from proactive realtime check: High confidence event for a specific machine: force drain node, evict the machine, skip stop time diagnos…