Search Engines (Elasticsearch)

The failure mode is treating a search box as a simple SQL filter. A query like WHERE title LIKE '%phone%' scans lots of text, misses synonyms and typos, cannot explain relevance, and struggles to return facet counts such as brand, size, and price range. Users expect the opposite: instant results, spelling tolerance, ranked matches, and filters that update as they type.

• Text is messy: users type plurals, punctuation, casing, misspellings, abbreviations, and phrases.
• Ranking matters: a document that matches the title should usually rank above one that only matches a footnote.
• Facets matter: commerce and document search often need counts by category, tag, price range, author, or geography.
• Operational isolation matters: expensive exploratory queries should not slow down checkout, payments, or writes.

A normal index maps a document ID to fields. An inverted indexflips that around: it maps each term to the documents that contain it, often with positions, frequencies, and field information. A query becomes a fast lookup of posting lists, then a merge and rank operation.

Documents
D1: "fast red bike"
D2: "red running shoes"
D3: "bike repair manual"

Inverted index
bike    -> D1(pos=3), D3(pos=1)
fast    -> D1(pos=1)
manual  -> D3(pos=3)
red     -> D1(pos=2), D2(pos=1)
running -> D2(pos=2)
shoes   -> D2(pos=3)

Query: "red bike"
lookup red + bike -> intersect/merge -> D1 ranks high, D3 may rank lower

Real indexes store much more than document IDs. Positions enable phrase queries like "red bike". Term frequencies help scoring. Field names let a title match count more than a body match. Compression keeps posting lists compact enough to search quickly in memory and on disk.

Before text enters the inverted index, Elasticsearch runs ananalyzer. Analysis turns raw text into normalized tokens. The same analysis, or a compatible query-time analysis, runs when the user searches.

Why one match ranks above another

Relevance scoring asks, "How well does this document satisfy the query?" Classic TF-IDF rewards terms that appear often in a document but are rare across the corpus. Elasticsearch commonly usesBM25, a modern descendant that also accounts for document length so very long documents do not win just because they contain more words.

score(document, query) roughly increases when:
  - query terms appear in important fields (title > description > comments)
  - a term appears multiple times, up to a saturation point
  - a term is rare across the whole corpus ("kubernetes" beats "the")
  - matched words are close together for phrase/proximity queries

score decreases or normalizes when:
  - the document is very long and matches only weakly
  - filters exclude it (tenant, visibility, stock status)

Your primary database is built for correctness: transactions, constraints, joins, point lookups, and writes. A search engine is built for retrieval: tokenized text, relevance scoring, aggregations, and horizontal query fan-out. Combining those jobs in one system usually makes both worse.

Because the index is derived, correctness-critical actions must still re-check the database. If search says a hotel room, seat, or product is available, the booking or checkout transaction confirms it against the source of truth before committing.

Applications usually feed search asynchronously. On every database change, a pipeline builds a search document: a denormalized JSON projection containing exactly the fields needed for search, filters, display snippets, and ranking. Change Data Capture (CDC) is a common way to do this without making every write synchronously update Elasticsearch.

Postgres transaction commits
  -> WAL/binlog records the change
  -> Debezium captures it
  -> Kafka topic: product.updated
  -> indexer service builds search document
  -> Elasticsearch _index /products/_doc/product_123

Search document example:
{
  "id": "product_123",
  "title": "Red running shoes",
  "brand": "Contoso",
  "category": "shoes",
  "price": 49.99,
  "in_stock": true,
  "popularity": 0.82
}

• At-least-once events: indexers must be idempotent because CDC events can be retried.
• Deletes matter: tombstone or remove documents when the source row is deleted or becomes invisible.
• Backfills matter: you need a safe way to rebuild an index from the database when mappings or analyzers change.

Elasticsearch distributes an index into primary shards. Each shard is a Lucene index that owns a slice of the documents. Replica shards copy primaries for high availability and query throughput. A query fans out to relevant shards, each shard returns top candidates, and the coordinating node merges the results.

client query: "red shoes" + filters brand=Contoso
        │
        ▼
coordinating node
  ├─ shard 0 searches local inverted index -> top 10 + facet counts
  ├─ shard 1 searches local inverted index -> top 10 + facet counts
  └─ shard 2 searches local inverted index -> top 10 + facet counts
        │
        ▼
merge scores, sort globally, combine aggregations, return page 1

Elasticsearch is near-real-time. A write is indexed, then a refresh makes new segments visible to search, commonly about once per second. Lower refresh intervals improve freshness but increase indexing and merge overhead; higher intervals improve throughput but increase visible lag.

Facets and aggregations

Faceted search computes counts over the result set: brand, color, price range, rating, author, file type, or region. Aggregations also power log dashboards and analytics, such as errors per service over time.

query: "running shoes", filter: price < 100

hits:
  1. Red running shoes, $49.99
  2. Trail running shoes, $79.99

facets:
  brand:
    Contoso: 120
    Fabrikam: 88
  color:
    red: 41
    black: 132
  size:
    9: 57
    10: 64