Creează un cont pentru a te abona la acest blog și a-i urmări articolele pe pagina Lectură.
Articole: 3
-
In Post 1, I doubled the request arrival rate on an Azure-derived replay and diagnosed what limit vLLM was hitting first.1 Before pushing harder, I needed a definition of what counted as a request being served well.
-
I took production traffic from Azure’s LLM inference service and replayed it at a scale that fit a single Nvidia A100, preserving the original request sizes and arrival times.1 When I doubled the arrival speed of the requests, median time to first token (…
-
Making the data smaller is a different problem from making the machine read it well.