I do things like get in a taxi and say, “The library, and step on it.”
David Foster Wallace, Infinite Jest
Work with the Google Kubernetes Engine team. Three changes: direct streaming with a control-plane-only endpoint picker, a new vLLM Ray executor backend, and HAProxy ingress. Together they give up to 4.4x more throughput on prefill-heavy workloads and 24x on decode-heavy ones.
The GKE-side write-up of the same work: up to 5x more throughput and 8x lower latency for distributed inference.
Serving Qwen on GKE with Ray Serve and vLLM, with an ADK agent in front of it for chat and tool use. Runs on TPU nodes, with Ray handling autoscaling and fault tolerance.
Ray 2.55 adds fault tolerance for data-parallel groups in large MoE deployments. When one rank fails, the whole group is marked unhealthy and recovers together, instead of the failure cascading across every GPU in it.
Guest lecture on distributed LLM serving with Ray Serve and vLLM: wide expert parallelism and disaggregated prefill/decode.
HAProxy ingress and direct gRPC on the data plane, measured on recommendation and LLM inference workloads.
An invite-only coffee hour for people building LLM serving infrastructure, with folks from vLLM, SGLang, Triton, Ollama, and TensorRT.
How DeepSeek reaches 2.2k tokens per second per H200: Wide Expert Parallelism, Dual-Batch Overlap, and disaggregated prefill/decode.
New Ray Serve LLM APIs for wide expert parallelism and prefill/decode disaggregation on sparse Mixture-of-Experts models.
How Ray Serve and vLLM serve Mixture-of-Experts models across nodes, including KV-cache optimizations and prefill/decode disaggregation.
Lightning talks on making LLMs efficient and designing agents that scale: CLI agent benchmarking, sparse MoE model serving, disaggregated inference.
PrefixCacheAffinityRouter sends each request to a replica that already has its prefix cached, which cuts time-to-first-token by 60%.
Deploying DeepSeek-R1 on Kubernetes with vLLM and Ray Serve, both the managed and the self-managed path.