Latest AI and tech news
A practical learning path covering model internals, fast inference, and the hardware that runs it all...
Teams building AI applications need a repeatable path from a model configuration to a working inference endpoint. NVIDIA Dynamo is a distributed inference framework that coordinates model serving across GPU workers. Deploying it on Kubernetes is only...
A Kubernetes cluster can be healthy while the AI service running on it disappoints users. A model might return its first token quickly but take too long to finish an answer. It might respond promptly and still give the wrong answer. And a busy GPU te...
On October 17 I opened a small vLLM pull request that made int8 matrix multiplies about 13% faster on an RTX 4090. The next day a reviewer ran it on a smaller Ada card and one layer got up to 81% slower....
TL;DR: A KDD 2026 paper follows five customer support AI agents through testing and deployment. The team checks automated graders against expert judgments, starts new versions at 1% to 5% exposure, and tracks whether satisfaction gains come with more...
Production inference is fundamentally a scheduling problem. Once you've addressed serving architecture and memory, the next question isn’t whether the GPU can do the work; it's how to schedule it....
Revolut’s David Abitbol says GLM-5.3 fits on 4 GPUs but runs short of memory. The reason is in vLLM’s source: every GPU keeps a full copy of each chat’s memory, the KV cache....
Authors: ~500 authorsAffiliations: DeepSeek-AIPaper: https://arxiv.org/abs/2609.19969Code: N/AModel: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash...
The surprising reason memory bandwidth can matter more than GPU computing power during LLM inference....
How DoorDash Reduced LLM Search Costs by 98% Without Losing Accuracy
The Medical Open Network for AI (MONAI) Model Zoo is often the first stop when a researcher is
building a medical imaging application: a shared shelf of ready-to-use, peer-reviewed models, where
you can find a whole-body computed tomography (CT) segm...
Everyone is building AI agents, but far fewer enterprises run them at scale without costs getting out of hand. We’re excited to partner with Baseten to fix this problem for enterprises by combining the fastest inference runtimes, cross-cloud high ava...
Baseten has released joey-1, a 2B model from its in-house research lab, Base Labs. The joey-1 decision model returns a label and a calibrated probability instead of generated text. It targets high-volume routing, classification and guardrail checks. ...
Autoregressive token generation is notoriously slow and memory-intensive, especially when you only need a model to make a structured classification or routing decision rather than write out paragraphs of text....
Prime Intellect has released Prime Mega MoE, a training extension of DeepSeek’s fused Mega MoE kernel. DeepSeek’s kernel only ran the forward pass. Prime Intellect added a fast, overlapping backward pass. That makes the fused kernel usable for traini...
AI services are becoming a fast-growing attack surface in production environments. The processes that run your models, agents, and inference pipelines tend to rely on open-source dependencies, ranging from AI-specific frameworks like LangChain and Hu...
We loaded 990,000 OpenAI embeddings of 1,536 dimensions into six databases that index vectors themselves, built an HNSW index in each, and measured five things: time to a queryable index, storage and memory, recall against throughput, throughput unde...
SAN JOSE, Calif., October 29, 2026 — SambaNova, the AI chip company delivering highest performance inference, today announced a strategic partnership with Emergys, a digital transformation services company, to accelerate the deployment of agentic AI ...
Lessons from bringing up RDMA-backed Prefill/Decode serving across H200, B200, GB200, and GB300 on GKE...
Your LLM feature is only as fast as its slowest 1 in 100 requests, because those are the ones users remember, screenshot, and complain about....
A slow walk from first principles. We’ll do the math by hand and run a little code....