Keyword search is excellent when the user knows the exact words that appear in the document. Semantic search becomes useful when the user knows what they mean, but not necessarily how the document phrases it.
A search for:
how do I stop duplicate payments after a timeout?
may need to find a document titled:
Idempotency and Ambiguous Outcomes in Payment APIs
even though the important words barely overlap.
That is the promise of semantic search—but production search systems are rarely “replace BM25 with embeddings.” The strongest systems combine lexical matching, semantic retrieval, metadata filters, reranking, and domain-specific relevance signals.
The goal is not to use the most AI-heavy retrieval stack. The goal is to return the right results at acceptable latency and cost.
Semantic search vs keyword search
Traditional lexical search scores documents largely from terms that appear in both the query and document.
This is extremely strong for things such as:
exact product codes
API method names
error messages
people/company names
legal references
rare technical terms
Semantic search represents text in a numerical space where content with similar meaning can be close even when the exact words differ.
For example:
query: "database is slow after analytics jobs start"
could retrieve:
"OLAP workloads are saturating PostgreSQL I/O and buffer cache"
That makes semantic retrieval powerful for natural-language queries, paraphrases, and conceptual search.
But keyword search did not suddenly become obsolete.
How embedding-based semantic search works
The common dense-vector pipeline is:
Documents
↓
Embedding model
↓
Vectors
↓
Vector index
User query
↓
Same compatible embedding model
↓
Query vector
↓
Nearest-neighbor search
↓
Top candidate documents
An embedding compresses some semantic properties of text into a vector of numbers.
The search engine then measures similarity using a metric such as cosine similarity, dot product, or another model/index-compatible distance function.
The most important rule is simple:
> Query and document vectors must be produced in a compatible embedding space.
If you change embedding models, you normally need to re-embed the corpus or deliberately operate multiple indexes during migration.
Dense retrieval is not the only semantic approach
Modern search stacks may use several retrieval families.
Dense vectors
Good at broad semantic similarity and paraphrases.
Learned sparse retrieval
Produces sparse learned representations that can preserve term-level behavior while adding semantic signals.
Lexical retrieval
BM25-style search remains excellent for exact strings, rare identifiers, and precise terminology.
Rerankers
A reranker looks at a smaller candidate set and applies a more expensive relevance model to reorder it.
These techniques can be combined instead of forcing one to solve every query.
Why pure vector search often disappoints
A demo with 100 documents may look magical.
Production data exposes harder cases.
Exact identifiers
A query such as:
ERR_PAYMENT_4217
may be handled more reliably by lexical matching than semantic similarity.
Numbers and versions
Queries involving:
PostgreSQL 19
API v2.4
invoice 817263
can lose precision if the retrieval system relies only on meaning.
Very similar documents
Semantic search may retrieve five nearly identical chunks from the same long document.
Domain-specific language
A generic embedding model may not perfectly capture the distinctions that matter in medicine, law, finance, source code, or a niche internal platform.
Filters and permissions
Similarity alone cannot enforce:
tenant_id = 42
status = published
language = en
user has permission to read document
Real search needs metadata and authorization too.
Hybrid search is often the strongest default
Hybrid search combines lexical and semantic retrieval.
Conceptually:
┌→ keyword/BM25 search ──┐
User query ──────┤ ├→ fuse rankings → results
└→ vector/semantic search ┘
The lexical side is good at precision.
The semantic side is good at meaning.
Elastic's current search guidance uses this same idea: modern retrieval can combine keyword, dense vector, and other semantic signals, then fuse and rerank the candidates.
Why raw scores cannot simply be added
A BM25 score and a vector-similarity score do not necessarily live on comparable scales.
That is why ranking-fusion techniques such as Reciprocal Rank Fusion (RRF) are useful.
RRF combines the rank positions from multiple result lists instead of assuming their raw scores are directly comparable.
This makes it a practical way to merge lexical and semantic retrieval.
Retrieval should usually be multi-stage
Search quality improves when expensive methods operate only on a small candidate set.
A common design is:
Stage 1: retrieve broadly and cheaply
↓
Stage 2: fuse/filter
↓
Stage 3: semantic rerank top candidates
↓
Stage 4: business/relevance rules
↓
Final results
For example:
BM25 top 100
+
vector top 100
↓
RRF
↓
top 50
↓
cross-encoder reranker
↓
top 10
Elastic's semantic-reranking guidance makes the same production trade-off: rerankers are more computationally expensive, so they are best used on a bounded top-k candidate set rather than the whole corpus.
Reranking can matter more than increasing initial top-k
First-stage retrieval is optimized for recall: do not miss potentially relevant material.
A reranker is optimized for precision: put the actually useful candidates first.
This is especially valuable for RAG, where only a few chunks will enter the LLM context.
If your retriever returns the correct passage at rank 27 but you only send the top five to the model, the system behaves as if the passage did not exist.
A good reranker can move that passage into the final context.
Metadata filtering belongs inside retrieval
Suppose you run a multi-tenant SaaS knowledge system.
The query is:
How do refunds work?
The embedding may find relevant refund documents from ten customers.
That is not acceptable.
The search query must also enforce scope:
tenant_id = current_tenant
AND access_policy permits user
AND document_status = active
Do this before results reach the model.
Never retrieve unauthorized content and then ask the LLM not to reveal it.
The model is not an access-control boundary.
Chunking is part of search quality
For document search and RAG, you often do not index entire documents as one vector.
Instead you split them into retrievable units.
But “500 tokens with 50-token overlap” is not a universal law.
Chunking should match document structure and question shape.
Good boundaries may be:
Markdown heading sections
API endpoint sections
individual support articles
code symbols
legal clauses
product specifications
Bad chunking can create:
- missing context
- duplicate results
- irrelevant fragments
- chunks that are too broad to rank precisely
Preserve useful parent metadata so the system can return the exact passage while still knowing its document, section, URL, title, and permissions.
Parent-child retrieval can improve context
Sometimes the best searchable unit is smaller than the best answerable unit.
For example:
small chunk → excellent search precision
larger section → better context for LLM/user
A useful pattern is:
index small child chunks
↓
retrieve relevant child
↓
expand to parent section
↓
rerank / assemble context
This prevents you from choosing between tiny precise fragments and huge noisy sections.
Query rewriting can help—but do not overdo it
Users often submit poor search queries:
it broke again after deploy
A model can sometimes rewrite this into a retrieval query based on conversation context:
checkout latency regression after latest production deployment
Other useful transformations include:
- extracting exact entities
- expanding abbreviations
- adding known product context
- generating one lexical query and one semantic query
But every query-rewrite model call adds latency and cost.
Use it when evals show that it actually improves retrieval.
Semantic search for code is different from prose
Source code has strong exact-match structure:
function names
class names
API paths
error strings
imports
types
Pure embeddings can be useful for conceptual questions such as:
where is retry logic implemented?
But code search often benefits from combining semantic retrieval with:
- exact text search
- symbol indexes
- AST-aware search
- language-server navigation
- file/path filters
Again, semantic search is one signal—not a replacement for every search primitive.
Search quality needs an evaluation dataset
You cannot tune relevance reliably by trying a handful of queries manually.
Build a set of representative queries with relevance judgments.
For each query, record which documents/chunks should be considered relevant.
Then measure metrics such as:
Recall@k
Did at least one relevant result appear in the first k candidates?
Precision@k
How many of the first k results were actually relevant?
MRR
How early did the first relevant result appear?
nDCG
Useful when several results have different relevance levels and ordering matters.
For RAG, also measure whether the retrieved context is sufficient for the downstream model to answer correctly.
A search system can have good vector-similarity metrics and still produce poor product outcomes.
Offline relevance and online behavior are both useful
Offline evals tell you whether a search change improves a known dataset.
Online signals tell you how users behave.
Depending on the product, signals may include:
click-through
query reformulation
zero-result rate
result abandonment
time to successful result
support-case resolution
Be careful: clicks are not pure relevance labels. Position bias and UI design affect them.
Use online signals as evidence, not absolute truth.
Latency is an architecture constraint
A sophisticated retrieval pipeline may produce excellent relevance but be too slow for the product.
Consider the cost of:
query embedding
keyword search
vector ANN search
ranking fusion
reranking model
permission filters
LLM generation
For an interactive RAG application, retrieval is only one part of total response latency.
You may need to:
- precompute document embeddings
- cache query embeddings where appropriate
- bound top-k
- rerank only top candidates
- use faster rerankers
- run independent retrieval paths in parallel
Optimize end-to-end latency, not one component in isolation.
Approximate nearest-neighbor indexes trade precision for speed
Large vector collections rarely compare the query against every vector exactly.
They typically use approximate-nearest-neighbor (ANN) indexes such as HNSW or IVF-family approaches.
The practical trade-off is:
more exhaustive search
→ potentially higher recall
→ more compute/latency
more approximate search
→ lower latency
→ possible recall loss
Tune index/search parameters using your relevance dataset and real latency targets.
Do not copy benchmark defaults blindly.
Embedding-model migrations need a plan
Embedding models change.
A production system should avoid assuming one vector schema will live forever.
A safer migration can look like:
current embedding index = v1
↓
create v2 index
↓
backfill embeddings
↓
shadow queries / compare relevance
↓
shift traffic
↓
retire v1
Store model/version metadata with the index so you know how each vector was generated.
Semantic search and RAG are related, not identical
Semantic search returns relevant information.
RAG uses retrieved information as context for a generative model.
You can have semantic search without an LLM:
query → search → ranked results shown to user
You can also build RAG using lexical or hybrid retrieval without relying only on vector search.
A good RAG architecture treats retrieval quality as its own engineering discipline.
A practical production pipeline
A strong default architecture for many knowledge products is:
Ingestion
├─ parse documents
├─ normalize text
├─ chunk by structure
├─ attach metadata + ACLs
├─ build lexical index
└─ generate semantic representation
Query
├─ authenticate user
├─ derive filters
├─ lexical retrieval ─────┐
├─ semantic retrieval ────┤
└─ optional other signals ┘
↓
rank fusion
↓
permission-safe candidates
↓
rerank top-k
↓
diversify / deduplicate
↓
final results
↓
UI or RAG context
This is much closer to production search than “store embeddings and call nearest neighbors.”
Common mistakes
Replacing keyword search entirely
Exact terms still matter.
Using one fixed top-k everywhere
Different queries and downstream tasks may need different candidate depths.
Ignoring filters
Semantic relevance cannot replace tenant, language, date, product, or permission constraints.
Sending raw nearest-neighbor scores directly to users
Similarity is not necessarily calibrated relevance.
No reranking
High-recall first-stage retrieval may still rank the best result too low.
No deduplication
The top results can become five overlapping chunks from one document.
Evaluating only answer quality
If the RAG answer is wrong, you need to know whether retrieval failed or generation failed.
Re-embedding with no version strategy
Model upgrades can become operationally painful if vectors are not versioned.
Production checklist
Before shipping semantic search, verify:
- Exact keyword queries still work well
- Embedding model is appropriate for your language/domain
- Query and document embeddings are version-compatible
- Metadata/ACL filters are enforced before exposure
- Chunking matches document structure
- Hybrid retrieval has been evaluated against semantic-only search
- Rank-fusion method is explicit
- Reranking depth is bounded and measured
- Duplicate/near-duplicate results are handled
- ANN parameters are tuned against recall and latency
- Relevance has an offline eval dataset
- RAG retrieval metrics are separated from generation metrics
- Embedding migrations can run safely
- Search latency fits the product's end-to-end budget
Final takeaway
Semantic search is not “AI search instead of keyword search.”
It is another powerful relevance signal.
The most reliable architecture usually looks like:
lexical precision
+
semantic meaning
+
metadata and permissions
+
ranking fusion
+
bounded reranking
+
real relevance evaluation
Start with the simplest retrieval stack that solves your queries. Add embeddings where exact terms are insufficient. Add hybrid retrieval when both lexical and conceptual intent matter. Add reranking when the right candidate is being found but ranked too low.
> Search quality comes from combining the right signals and measuring relevance—not from choosing the fanciest vector database.
References and further reading
Discussion (0)