llm-d

Today at PyTorch Conference 2026, we announced llm-d 1.0! 🎉 Distributed inference is too complex to solve behind closed doors. 17 months in, llm-d is now a @cncf.io project powering fleets worldwide. Here’s what’s inside 1.0 🧵👇

🔒 Stable v1 API: A versioned configuration surface and hardened Endpoint Picker designed to give platform teams a dependable foundation for years to come. No breaking surprises—just reliable, production-ready infrastructure.

Today at PyTorch Conference 2026, we announced llm-d 1.0! 🎉 Distributed inference is too complex to solve behind closed doors. 17 months in, llm-d is now a @cncf.io project powering fleets worldwide. Here’s what’s inside 1.0 🧵👇

What’s landing in 1.0: ⚡ Battle-Tested Models: Production recipes for DeepSeek, GLM, Qwen & Nemotron. ⚖️ Multi-Tenant QoS: Priority & fairness-aware flow control to protect latency-sensitive traffic during bursts.

Today at PyTorch Conference 2026, we announced llm-d 1.0! 🎉 Distributed inference is too complex to solve behind closed doors. 17 months in, llm-d is now a @cncf.io project powering fleets worldwide. Here’s what’s inside 1.0 🧵👇

Since launching in May 2025, llm-d has become a massive community effort: 🟣 900+ contributors hardening code 🟣 ~5,000 GitHub stars 🟣 Fleets across the AI ecosystem Every bug caught upstream made this possible. This milestone belongs to you.

The era of the uniform KV cache is officially over. ❌ Modern hybrid models mix attention types—interleaving full attention with sliding-window or Mamba layers. This means the entire serving stack has to adapt to handle this heterogeneity. 👇

3/ 🌐 Global Multi-Tier Routing: Capacity is an instance property, but throughput is about placement. By scoring GPU cache & CPU offload tiers globally, the llm-d scheduler drives a 115% throughput increase under load while keeping TTFT completely flat.

The era of the uniform KV cache is officially over. ❌ Modern hybrid models mix attention types—interleaving full attention with sliding-window or Mamba layers. This means the entire serving stack has to adapt to handle this heterogeneity. 👇

By extending vLLM’s Hybrid Memory Allocator (HMA) into llm-d’s tiered architecture, they tackled the hybrid cache challenge across 3 major fronts: 1/ 🔥 Nearly Double Load Speed: HMA-aware reads are 1.8x to 1.9x faster, loading only the exact KV data slices layers need.

The era of the uniform KV cache is officially over. ❌ Modern hybrid models mix attention types—interleaving full attention with sliding-window or Mamba layers. This means the entire serving stack has to adapt to handle this heterogeneity. 👇

Can a cache-aware, saturation-aware router make a mixed, 3-vendor GPU cluster perform like one unified, high-performance inference service? 🚀 Yes, it can. We benchmarked llm-d v0.7.0 across a real-world heterogeneous cluster with IBM Research, Red Hat, and NxtGen Cloud. Here is what we found 👇

How? llm-d concentrates cache hits on warm pods using precise (tokenizer-backed) or approximate (hash-based) prefix routing. It dynamically adapts to load and saturation signals while fully preserving the platform-specific driver and runtime settings each vendor needs. 🛠️

Can a cache-aware, saturation-aware router make a mixed, 3-vendor GPU cluster perform like one unified, high-performance inference service? 🚀 Yes, it can. We benchmarked llm-d v0.7.0 across a real-world heterogeneous cluster with IBM Research, Red Hat, and NxtGen Cloud. Here is what we found 👇

The Test: We deployed llm-d’s prefix-cache-aware routing across a 20-GPU pool spanning THREE different vendors on the NxtGen sovereign cloud, serving granite-4.1-8b and sarvam-30b on prefill-heavy RAG/chat workloads. The results speak for themselves:

Can a cache-aware, saturation-aware router make a mixed, 3-vendor GPU cluster perform like one unified, high-performance inference service? 🚀 Yes, it can. We benchmarked llm-d v0.7.0 across a real-world heterogeneous cluster with IBM Research, Red Hat, and NxtGen Cloud. Here is what we found 👇

🎉 llm-d v0.7 is officially live! 🚀 If our earlier releases proved what llm-d could do, v0.7 is about making sure you can easily deploy it in production. Backed by a massive 3.5x surge in community PR volume, this release hardens the stack for serious scale. 👇 llm-d.ai/blog/llm-d-v...

🧠 Workload-Aware Routing & Caching • Flow Control: Centralized queuing at the Router level to prevent noisy neighbors. • Batch Gateway: OpenAI-compatible API for heavy offline workloads. • Real-time prefix cache tracking + tiered offloading to AWS EFS/NVMe.

🎉 llm-d v0.7 is officially live! 🚀 If our earlier releases proved what llm-d could do, v0.7 is about making sure you can easily deploy it in production. Backed by a massive 3.5x surge in community PR volume, this release hardens the stack for serious scale. 👇 llm-d.ai/blog/llm-d-v...

⚙️ Streamlined Day-1 Ops We’ve fundamentally simplified the day-one operator experience: • Standalone Mode: Go from clone to serving in minutes using a lightweight Envoy proxy. • Kustomize-First: Shifted from Helm to Kustomize for cleaner GitOps (ArgoCD/Flux) pipelines.

Moving an LLM from demo to production is where the real work begins. It’s not just about accuracy, it’s about latency, GPU efficiency, and cost-to-serve. We’re thrilled to see @OracleCloud’s deep dive on scaling llm-d on OCI for enterprise workloads! 🧵👇

How do you scale LLM inference without the "Day 2" operational headaches? We just shared a new deep dive on the llm-d blog: building a production-grade stack with KServe + vLLM + llm-d. Collaborative work with contributors from Red Hat & Tesla. Read the full breakdown: 🔗 llm-d.ai/blog/product...

Standard load balancers are "AI-blind," but by using KServe’s Inference Gateway + llm-d’s prefix-cache aware routing, we’re seeing massive wins: ✅ 3x boost in output tokens/sec ✅ 2x reduction in TTFT ✅ Disaggregated scaling for prefill & decode