Inference
Running models in production: serving stacks, latency, throughput and the engines behind them
Latest stories See all →
- I Mapped the Courses I’d Take to Become an ML Infrastructure Engineer at Google
- Build a Validated Kubernetes Runtime for NVIDIA Dynamo
- Busy GPUs can hide slow AI inference on Kubernetes
- Show HN: Spinf-serve – inference server for bulk scoring and decisions
- My GPU fix was 13% faster on one card and 81% slower on another
- How a Bank Serving 100 Million Users Tests Agentic AI in Production
- Batching Strategies: Throughput vs Latency in Practice
- Why Self-Hosted GLM-5.3 Needs 8 GPUs, Not 4, for Long Coding Sessions
- DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
- Your GPU Is Fast. So Why Is Your LLM Still Slow?
- How DoorDash Reduced LLM Search Costs by 98% Without Losing AccuracyEngineeringGen AILLMML Infrastructure
- LLM in Production: A Complete Guide for Enterprise Engineering Teams
- How to Run Nemotron 3.5 Transcribe Locally with vLLM
- Bringing the MONAI Model Zoo to AMD Instinct™ GPUs: Validated Inference on ROCm
- Serverless GPU inference vs. GPU Pods: cost benchmarks
- Postman and Baseten: Build Performant Agents, Run Them At Lower Cost
- Baseten Releases joey-1, a 2B Decision Model at $0.01 per 1M Tokens
- High-Speed Decision Pipelines Jev and Prefill-Only Inference from Scratch
- Prime Intellect Releases Prime Mega MoE: Trainable Fused MoE Kernel, 3.11x Faster Forward Pass
- Surface vulnerabilities in your AI services
- Baseten and Postman: Building reliable, cost-efficient agents for the enterprise
- Speeding up language models for travel applications
- Benchmarking Vector search in six database engines
- Inference cost explained: why your AI bill rises as prices fall
Videos
- What Is Quantization? How AI Models Get Smaller & Faster
- Nemotron 3.5 Super VL with Native Speculative Decoding
- Build A Reasoning Model From Scratch 8: Distillation
- EfficientML.ai Lecture 9 - Knowledge-Distillation (MIT 6.5940 Fall 2026)
- How Mixture of Experts Could Make Local LLMs Practical
- EfficientML.ai Lecture 6 - Quantization (Part II) (MIT 6.5940 Fall 2026)
- Complete Mac mini M6 Local LLM Stress Test — 5 Models, 3 Inference Engines
- Xiaomi and DeepSeek Just Solved the Biggest Problem in AI
- LM Studio Full Tutorial - The BEST Way to Run Local Models
- Can A Free 4GB AI Replace Your Cloud AI Coder?
- Your 128GB Of RAM Won't Save A 24GB Card For Local AI
- Optimize KV cache with llm-d and vLLM
Podcasts
Recent episodes
- Inside the Performance Constraints of Modern AI Workflows - with Cameron Bahar of Everpure
- AI Inference 101
- Tom Milton on how we could verify a US–China deal on AI
- Amazon CEO Andy Jassy on the $220 Billion AI Buildout
- Amazon CEO Andy Jassy on the $220 Billion AI Buildout
- Amazon CEO Andy Jassy on the $220 Billion AI Buildout
- D2DO316: Local AI Foundations
- 1049: WTF is a Software Factory?
- 1049: WTF is a Software Factory?
- Eric Schmidt's $80 Billion Anthropic Bet
- Ep. 038 - GLM 5.3 on AMD, NVIDIA, and Chinese Chips From Day One (InferenceX) | Kimbo Chen, Wenyao Gao, Jordan Nanos
- SED News: Jev, Meta’s Muse, and the AI Slowdown Paradox