Elasticsearch and full-text search: How search engines work

In short: A full-text search finds words in large amounts of text quickly by creating an index beforehand that knows the occurrences of every word.

In more detail: With LIKE '%word%' in SQL the database would have to search every text, which is slow with lots of data and knows no relevance. Search systems such as Elasticsearch, OpenSearch, Meilisearch or Apache Solr (all based on Lucene or similar methods) build an inverted index and deliver hits sorted by relevance.

In Depth

The inverted index

Like a book’s keyword index: for each word it lists in which documents and at which positions it occurs.

"cat"  -> document 1, 7, 12
"dog"  -> document 2, 7

A search for “cat dog” intersects or unites these lists very quickly.

Analysis of the text

Before indexing, text is analysed: split into words (tokenisation), converted to lower case, punctuation removed, stop words (the, and) left out and words reduced to their stem (“houses” → “house”). Language-specific analysers for German take umlauts and compound words into account. The same analysis runs on the search query.

Relevance and further functions

Hits are scored, classically with TF-IDF and BM25: a word that occurs often in the document but rarely overall counts strongly. Further functions: typo tolerance (fuzzy search with Levenshtein distance), autocomplete, highlighting, facets (filter by category), geo search and increasingly vector search (semantic similarity via embeddings).

Use and operation

Typically it is an additional system next to the main database: data is copied there (synchronisation), searches run on the index. Elasticsearch is distributed (shards and replicas) and also suits logs (ELK stack: Elasticsearch, Logstash, Kibana). Note: the index is not the source of truth and can lag slightly. For small applications the built-in full-text search of PostgreSQL (tsvector) or SQLite (FTS5) is often enough.

See also: Vector databases and similarity search, SQL index, RAG