Transformer Architecture

STANDARD DEFINITION · CACHED

The is a model structure introduced by in the 2017 paper "Attention Is All You Need," which revolutionized by replacing and units. It is built upon the mechanism, specifically , which allows the model to process input data sequences in parallel rather than serially, effectively capturing and global context. The architecture typically consists of an and a , utilizing to maintain word order and to stabilize training. This framework serves as the foundation for modern , including the series by , by , and by , and has been adapted for other domains such as via the .

[PREVIEW MODE] Definitions streamed at five depths, references, and related terms are available to signed-in readers — [SIGN IN]

Contextual terminology map

Active Learning AI Alignment AI Audit AI Bubble AI Capex AI Watermarking Algorithmic Bias Anthropic Attention Mechanism Backpropagation Bayesian Inference Big Data Content Credentials Convolutional Neural Network CUDA Data Broker Data Lake Data Normalization Data Poisoning Decision Trees Deepfake Differential Privacy Diffusion Models Digital Divide Disparate Impact End-to-End Encryption EU AI Act Explainable AI Export Controls Feature Engineering Fine-tuning Frontier Model GDPR (General Data Protection Regulation)Generative AI Google DeepMind GPU Computing Gradient Boosting Gradient Descent HBM (High-Bandwidth Memory)Hugging Face Hyperscaler Inference Interpretability Job Displacement Large Language Model Mistral AI Model Card Model Distillation Neocloud Neural Network NIST AI RMF (AI Risk Management Framework)NVIDIA OpenAI Overfitting Power Purchase Agreement Principal Component Analysis Prompt Engineering Prompt Injection PUE (Power Usage Effectiveness)PyTorch Quantization RAG (Retrieval-Augmented Generation)Random Forest Recurrent Neural Network Red Teaming Reinforcement Learning Small Modular Reactor Sovereign AI Stochastic Gradient Descent Supervised Learning TensorFlow TPU (Tensor Processing Unit)Transformer Architecture Underfitting Unsupervised Learning Vector Database WCAG (Web Content Accessibility Guidelines)Zero-Day

KEEP IN VIEW

Infrastructure, markets & the economy

The compute, power and capital behind the boom

Concept inspired by Dev Valladares’s Infinite Wiki. Independently built; not affiliated with or endorsed by the original.

Transformer Architecture

Contextual terminology map

KEEP IN VIEW

Ethics, safety & society

Infrastructure, markets & the economy

Keyboard shortcuts

Go to

On this page