How to Self-Host an LLM Gateway with Docker! Bifrost AI Gateway Setup

Build a self-hosted LLM gateway in Docker with Bifrost and two local Ollama models, no API key needed, and watch it cut off a runaway loop with a 402. Every model call goes through one process that speaks the OpenAI request shape, so moving a client over is a single base URL change. From there I set up an internal rate card, a team budget and per-service virtual keys, split traffic 80/20 across two providers, kill one container to get a 502 and then a clean fallback to Gemma 3, and run a spend loop until its budget refuses it. Then I add exact and semantic caching with Redis and all-MiniLM, show the semantic cache confidently answering a MySQL question with a Postgres answer, and read the per-call logs for tokens, cost and gateway overhead. Along the way: the two Ollama settings that made the models usable inside Docker, and why a model with no price reports as free forever. Built on Bifrost v2.2.1. ⏱️ Chapters 0:00 - Call 31 refused: a runaway loop hits its budget 0:51 - What an LLM gateway actually does 2:12 - The Docker Compose stack: Ollama, Redis, Bifrost 3:34 - Two fixes that made Ollama usable in Docker 4:41 - The usage block every cost is built from 5:24 - Configuring Bifrost providers and the cache 7:30 - Migrating a client: one base URL change 8:30 - Rate cards, teams and virtual keys 11:03 - Weighted routing across two providers 11:45 - Failover when a provider goes down 13:07 - Budget caps: the spend loop hits a 402 14:48 - Exact and semantic caching with Redis 16:48 - When the semantic cache answers wrong 17:33 - Per-call logs, costs and gateway overhead 19:01 - Recap and production trade-offs 🔗 Links Bifrost: https://github.com/maximhq/bifrost Ollama: https://ollama.com Redis: https://redis.io Docker: https://www.docker.com #LLMGateway #AIGateway #Bifrost #MaximAI #Ollama #Docker #DockerCompose #Redis #selfhostedLLM #localLLM #semanticcache #LLMOps #Qwen #Gemma #OpenAI