Search Engines & Information Retrieval
How distributed web crawlers traverse hyperlinks, compile multi-terabyte inverted indexes, and retrieve relevant results across 50 billion pages in 150 milliseconds.
Pieces in this series
How Search Engines Actually Work
How search engines crawl the web, build inverted indexes, and rank billions of pages in 150 milliseconds using BM25, PageRank, and neural vector search
Search engines do not search the live web when you click search. They search a pre-built, compressed Inverted Index—a reverse dictionary mapping words to document postings—scored across 200+ signals including link graph authority…
Why Search Results Differ Between People
Geographic IP Anycast routing, session history vectors, freshness ranking, and A/B experiment flight allocation
Search engines do not maintain a single static leaderboard of the web. Search results diverge based on geographic IP resolution, implicit session context, device capabilities, language preferences, real-time freshness signals, and…
How PageRank and Link Analysis Rank the Web
From academic citation networks to the random surfer model, Markov transition matrices, the damping factor, and power iteration numerical convergence
Early search engines ranked pages by counting keyword frequency, making them trivial to spam. PageRank transformed search into linear algebra: treating hyperlinks as democratic peer citations, modeling web navigation as a Markov random…
How Vector Search and Dense Retrieval Work
From word embeddings and bi-encoder transformers to cosine similarity, HNSW small-world graphs, and product quantization
Traditional search relies on lexical inverted indexes and BM25 term matching. Dense retrieval translates text into geometric vector embeddings via transformer bi-encoders, measuring semantic similarity as spatial proximity. To search…
How Multi-Stage Search Ranking Pipelines and Learning to Rank Work
From candidate generation and BM25/vector retrieval to LambdaMART gradient boosting, cross-encoder neural re-ranking, and SERP blending
Searching billions of web pages in milliseconds is physically impossible with a single deep model. Modern search engines solve this using a multi-stage funnel: fast inverted indexes and vector indexes retrieve the top 1,000 candidates in…