Enterprise gen AI inference demo with vLLM on Red Hat AI stack

How can you scale enterprise LLM inference to production? In this demo from Red Hat Summit, Grace Ableidinger, AI Developer Advocate at Red Hat, breaks down how to scale AI workloads from a single RAG chatbot to a multi-model enterprise architecture. Learn how AI Gateway manages control plane routing, how vLLM handles paged attention and continuous batching, and how llm-d optimizes routing across replicas. Discover optimization techniques including model quantization, speculative decoding, and prefill-decode disaggregation to reduce latency and maximize GPU efficiency. 00:00 Introduction to enterprise gen AI inference 00:15 AI Gateway, vLLM, and llm-d architecture overview 00:49 Scaling a single GPU RAG chatbot with vLLM 01:16 Model quantization with vLLM and Hugging Face 01:50 Speculative decoding for faster token generation 02:20 Horizontal scaling and smart routing with llm-d 03:34 Prefill-decode disaggregation for agentic workflows 04:06 Exploring the Red Hat AI model catalog Find more gen AI resources from Red Hat: ✨ Explore Red Hat AI → https://www.redhat.com/en/products/ai 📖 Read the Red Hat AI blog → https://www.redhat.com/en/blog/channel/artificial-intelligence 🤝 Browse models on Red Hat AI Hugging Face → https://huggingface.co/RedHat #RedHatAI #OpenShiftAI #vLLM #LLMD #GenAI #AIInference #LLM #MachineLearning