Self-Attention & Multi-Head Attention

Linear Algebra for Machine Learning · Axiom Academy

Self-Attention: Complete Numerical Example Full walkthrough of attention computation For clarity, we'll use a 2-token sequence with small dimensions. Step 2: Compute Attention Scores Token 1: Attends to 50% Token 1, 50% Token 2 (balanced) Token 2: Attends to 50.1% Token 1, 49.9% Token 2 (nearly balanced, slightly prefers itself) Result: Each token's representation is updated by blending with other tokens based on computed similarity In practice, with learned weights, attention patterns become more specialized and semantic. All matrix operations: Query, Key, Value projections, multiplication, softmax, weighted sum Fully differentiable: Every step can be backpropagated Quadratic complexity: Q × K^T is O(seq_len²) in memory and computation Dynamic: Attention weights change based on inputs, allowing context-dependent processing Scaled dot-product: Division by √d_k prevents softmax saturation Unit 8: Linear Algebra in Neural Networks | Module 12: Self-Attention Example

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.