Real World: TF-IDF Vectorization

Linear Algebra for Machine Learning · Axiom Academy

Representing text for search and classification Search engines and document classifiers need to represent documents as vectors for comparison. The classic approach: TF-IDF (Term Frequency - Inverse Document Frequency) Core Idea: Each unique word in the corpus is a dimension. A document is represented by how much of each word it contains. But some words are common (the, is, a) while others are meaningful (algorithm, vector, learning). TF-IDF weights terms by importance. TF(term, doc) = (# times term appears in doc) / (total terms in doc) Inverse Document Frequency (IDF) IDF(term) = log(total documents / documents containing term) TF-IDF(term, doc) = TF(term, doc) × IDF(term) • Word "learning" appears 20 times in doc1, doc1 has 500 words • "learning" appears in 100 documents • TF("learning", doc1) = 20/500 = 0.04 • IDF("learning") = log(1000/100) = log(10) ≈ 1 • TF-IDF ≈ 0.04, meaning moderate importance With TF-IDF, each document becomes a high-dimensional vector (one dimension per word in vocabulary). To find similar documents, we compute cosine similarity (the dot product we learned about!): similarity = (doc1_vec · doc2_vec) / (||doc1_vec|| ||doc2_vec||) • Web search: TF-IDF + cosine similarity finds relevant pages • Email spam detection: TF-IDF vectors of emails compared to spam corpus • Document recommendation: Vectors enable finding similar documents • Now: Deep learning replaced TF-IDF for many tasks, but the concept remains foundational

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.