Lequn Chen || abcdabcd987

In the previous post I complained at length about pipeline parallelism.

Recently, the center of my work shifted from building inference systems to building post-training systems.

Last week, our team summarized some recent progress we made on point-to-point communication for LLM systems and posted a paper on arXiv.

In the previous blog post, I walked through how we achieved cross-machine RL weight updates in just 2 seconds.

I just spent the past two weeks getting cross-machine parameter updates for Qwen3-235B (BF16 training, FP8 inference) to run in just 2 seconds (128 GPUs for training, 32 GPUs for inference).

In the previous chapter, we solved the pre-test warmup issue, yet the transmission speed was still only 293.461 Gbps, reaching just 9.2% of the total 3200 Gbps bandwidth.

In the previous chapter, we enabled multi-threading and created a thread for each GPU.

In the previous chapter, we bound CPU cores for the multi-threaded version, which boosted the transmission speed to 1237.738 Gbps.

In the previous chapter, we sharded the states across different threads to avoid synchronization between threads, which increased the transmission speed to 1522.738 Gbps.

In the previous chapter, by submitting new WRITE operations in batches, we achieved a transmission speed of 2589.488 Gbps, reaching 80.9% of the total 3200 Gbps bandwidth.