Post

llm-d
@llm-d.ai

How do you scale LLM inference without the "Day 2" operational headaches? We just shared a new deep dive on the llm-d blog: building a production-grade stack with KServe + vLLM + llm-d. Collaborative work with contributors from Red Hat & Tesla. Read the full breakdown: 🔗 llm-d.ai/blog/product...

llm-d.aiProduction-Grade LLM Inference at Scale with KServe, llm-d, and vLLM | llm-dHow migrating from a simple vLLM deployment to a robust MLOps platform utilizing KServe, llm-d's intelligent routing, and vLLM solved significant scaling and operational challenges in LLM deployment…
llm-d
@llm-d.ai

Standard load balancers are "AI-blind," but by using KServe’s Inference Gateway + llm-d’s prefix-cache aware routing, we’re seeing massive wins: ✅ 3x boost in output tokens/sec ✅ 2x reduction in TTFT ✅ Disaggregated scaling for prefill & decode

llm-d
@llm-d.ai

This is the "well-lit path" for LLM infra: high performance, operational simplicity, and full alignment with the standard Kubernetes API. Huge thanks to the community for collaborating on these patterns. Check out the benchmarks and join us: 🔗 llm-d.ai/blog/product...

llm-d.aiProduction-Grade LLM Inference at Scale with KServe, llm-d, and vLLM | llm-dHow migrating from a simple vLLM deployment to a robust MLOps platform utilizing KServe, llm-d's intelligent routing, and vLLM solved significant scaling and operational challenges in LLM deployment…

0 reposts1 like

No replies yet.