Mixture of Depths

STANDARD DEFINITION · CACHED

(MoD) is a architectural technique developed by that optimizes in by dynamically allocating across a sequence. Unlike traditional that apply the same number of and layers to every , MoD employs a to decide which tokens in a batch require full processing through specific and which can bypass them via a . This approach implements a form of , similar to (MoE), but operates across the depth dimension rather than the width. By enforcing a fixed per , can maintain high performance while significantly reducing the latency and requirements during , particularly on hardware optimized for .

[PREVIEW MODE] Definitions streamed at five depths, references, and related terms are available to signed-in readers — [SIGN IN]

Contextual terminology map

Active Learning AI Alignment AI Audit AI Bubble AI Capex AI Watermarking Algorithmic Bias Anthropic Attention Mechanism Backpropagation Bayesian Inference Big Data Content Credentials Convolutional Neural Network CUDA Data Broker Data Lake Data Normalization Data Poisoning Decision Trees Deepfake Differential Privacy Diffusion Models Digital Divide Disparate Impact End-to-End Encryption EU AI Act Explainable AI Export Controls Feature Engineering Fine-tuning Frontier Model GDPR (General Data Protection Regulation)Generative AI Google DeepMind GPU Computing Gradient Boosting Gradient Descent HBM (High-Bandwidth Memory)Hugging Face Hyperscaler Inference Interpretability Job Displacement Large Language Model Mistral AI Model Card Model Distillation Neocloud Neural Network NIST AI RMF (AI Risk Management Framework)NVIDIA OpenAI Overfitting Power Purchase Agreement Principal Component Analysis Prompt Engineering Prompt Injection PUE (Power Usage Effectiveness)PyTorch Quantization RAG (Retrieval-Augmented Generation)Random Forest Recurrent Neural Network Red Teaming Reinforcement Learning Small Modular Reactor Sovereign AI Stochastic Gradient Descent Supervised Learning TensorFlow TPU (Tensor Processing Unit)Transformer Architecture Underfitting Unsupervised Learning Vector Database WCAG (Web Content Accessibility Guidelines)Zero-Day

KEEP IN VIEW

Infrastructure, markets & the economy

The compute, power and capital behind the boom

Concept inspired by Dev Valladares’s Infinite Wiki. Independently built; not affiliated with or endorsed by the original.

Mixture of Depths

Contextual terminology map

KEEP IN VIEW

Ethics, safety & society

Infrastructure, markets & the economy

Keyboard shortcuts

Go to

On this page