This Tiny Qwen 27B Model Runs on a 12GB GPU

Underdog Saluki 27B is a new 7.9GB local AI model that compresses the original 55.6GB Qwen 27B model using aggressive 2-bit quantization, yet reportedly outperforms the full-size model at tool calling. But there's a catch. In this video, I test Saluki 27B locally using llama.cpp, parallel tool calls and JSON output, memory usage and inference performance, and show how such a heavily compressed LLM can achieve higher tool-calling benchmark scores while losing accuracy in math, coding, and reasoning. 🔗 Relevant Links Hugging Face - https://huggingface.co/ConwayResearch/Underdog-Saluki-27B-1.0 ❤️ More about us Radically better observability stack: https://betterstack.com/ Written tutorials: https://betterstack.com/community/ Example projects: https://github.com/BetterStackHQ 📱 Socials Twitter: https://twitter.com/betterstackhq Instagram: https://www.instagram.com/betterstackhq/ TikTok: https://www.tiktok.com/@betterstack LinkedIn: https://www.linkedin.com/company/betterstack 📌 Chapters: 00:00 Underdog Saluki 27B: A 7.9GB AI Model That Beats the Original? 00:35 Why Running Qwen 27B Locally Is So Difficult 01:10 Testing Saluki 27B with llama.cpp and Tool Calling 01:57 How 2-Bit Quantization Shrinks a 55GB AI Model 02:50 How Saluki 27B Outscored the Original Qwen Model 04:15 Saluki vs Unsloth, Ternary Bonsai 2, and Other Quantizations 05:10 Real-World AI Agent and Tool-Calling Benchmarks 06:18 The Catch: Math, Coding, Reasoning, and Performance Losses 07:20 Can We Trust Saluki 27B's Benchmark Results? 08:00 Should You Run Saluki 27B or a 4-Bit Qwen Model?