Hybrid search and reranking

In lesson 02 we saw that embeddings capture meaning. That's great for questions like "how do I cancel my subscription?", where language variations exist. But consider these cases:

• A developer searches for NullPointerException in error logs.
• A security analyst queries the code CVE-2024-1234.
• A user types an exact product name: XR-7000 Pro.

The embedding model will vectorize these terms and find neighbors in semantic space. The problem is that CVE-2024-1234 and CVE-2024-5678 end up very close vectorially — they're structurally similar. The system retrieves the wrong chunk with high confidence.

Lexical search (BM25, the classic term-frequency ranking algorithm) doesn't have this problem. It treats CVE-2024-1234 as a literal string and only returns documents containing that exact token. It's deterministic, fast, and needs no GPU.

BM25's weakness is the inverse: no synonyms, no tolerance for variation. "Cancel subscription" won't find "terminate contract". For natural language, it loses badly to embeddings.

Neither approach is superior in every scenario. The right answer is to run both in parallel and merge the results — that's hybrid search.

The mechanics are simple: you fire two searches in parallel — one vector, one lexical — and each returns a ranked list of chunks. The challenge is merging those lists into one, since the scores are incomparable (cosine similarity vs. BM25 score).

The most widely used technique in practice is Reciprocal Rank Fusion (RRF). The idea is elegant: instead of normalizing scores (which is brittle), you use only each document's position in the lists.

The formula is:

RRF(doc) = Σ 1 / (k + rank_i(doc))

Where k is a constant (typically 60) and rank_i is the document's position in list i. A document that appears 3rd in vector search and 5th in BM25 will have a high RRF score. A document that only appears in one list will score lower.

OpenSearch Serverless and OpenSearch Service support hybrid search with RRF natively — you configure the hybrid query type and set weights for each sub-query. Amazon Bedrock Knowledge Bases also exposes this option in the console and via API.

An important parameter is the relative weight between vector and lexical search. There's no universal value: it depends on your domain. For technical documentation with lots of codes and acronyms, I give more weight to BM25. For natural-language FAQ, more weight to vector. Start 50/50 and adjust based on evaluation results (lesson 08).

Hybrid search improves recall — you retrieve more relevant chunks. But the ranking is still imperfect. RRF doesn't know why the question was asked; it only knows a chunk ranked well in two lists.

The reranker solves this. It's a cross-encoder: it receives the query and each chunk together as input and produces a true relevance score. Unlike embeddings (which vectorize query and document separately), the cross-encoder reads both at the same time and can capture subtle dependencies — for example, that the answer to the question is in the third sentence of the chunk, not the first.

The usage pattern is: retrieve many candidates (20, 50, even 100 chunks) with hybrid search, then pass everything through the reranker and keep the top-3 or top-5. This works because rerankers are expensive per call but you use them once per query, over a small set.

On AWS, the most direct path is Cohere Rerank available via Amazon Bedrock. You pass the query and the list of chunks, and get back the reordered indices with scores. Integration with Knowledge Bases is still manual in this flow — you call the reranker after retrieve and before building the prompt.

An important detail: rerankers have a token limit per document. Very long chunks (from lesson 03) will be truncated. If you use 1024-token chunking, check the reranking model's limit — typically 512 tokens per passage is the safe maximum.