How to Run LLMs Locally in Production (Ollama Explained)

Ollama lets you run large language models on hardware you already own — no per-token API bill. Here's what's really happening under the hood. We start with the token math that turns a "basic data job" into a $17,500 bill, then pull Ollama apart into what it actually is: a Docker-style registry, a scheduler, an HTTP API, and model files you can pin like a Dockerfile. From there we cover quantization (what Q4KM and Q5KM really mean and which to pick), how to size hardware for the two constraints that matter — capacity vs memory bandwidth — and how to wire Ollama into an Airflow pipeline without melting your VRAM. We finish with a clear decision rule for when to stay local, when to reach for a frontier model, and the hybrid routing trick that slashes your API spend. ⏱️ Chapters 0:00 - Run LLMs on hardware you already own 0:48 - The token bill that wrecks AI pipelines 2:22 - Why local costs cents, not thousands 3:17 - What Ollama actually is, demystified 5:09 - Model files: pin configs like Docker 6:04 - Quantization: shrink models to fit 7:44 - Picking Q4KM vs Q5KM for your task 10:49 - Sizing hardware: capacity vs bandwidth 13:44 - Wiring Ollama into your pipeline 15:03 - Tuning concurrency & KV cache 16:19 - Local vs frontier: when to use each 17:45 - The decision rule + hybrid routing 19:06 - Recap & key takeaways 🔗 Links Ollama: https://ollama.com llama.cpp: https://github.com/ggml-org/llama.cpp MLX: https://github.com/ml-explore/mlx Docker: https://www.docker.com Apache Airflow: https://airflow.apache.org #Ollama #LocalLLM #LLM #Quantization #Q4KM #AIpipelines #Airflow #GPUinference #SelfHostedAI #DataEngineering #MachineLearning #llamacpp #KVcache #ModelServing #DataGuy