Your RAG Pipeline Is Missing a 1994 Algorithm

Your RAG Pipeline Is Missing a 1994 Algorithm | Towards Data Science

Large Language Models

Your RAG Pipeline Is Missing a 1994 Algorithm

Embeddings know that "broken" means "defective". They may not notice that ZX-4821 is not ZX-4812.

Spyros Georgopoulos

October 28, 2026

17 min read

Prefer us on Google

Share

Different keys, different locks. Photo by Silas Köhler on Unsplash

Each key has a differently shaped bit, suggesting that each opens a different lock. In the context of the article, the keys stand for retrieval methods such as BM25 and dense retrieval: none of them opens everything, so a good system keeps more than one on the ring.

Suppose a customer- support assistant receives the following question:

What does error code ZX-4821 mean?

The answer exists in the knowledge base. One document contains the exact code, followed by a short explanation of the hardware failure that produced it.

Yet the retriever does not return that document.

Instead, it retrieves three passages about common device errors, troubleshooting procedures, and hardware diagnostics. Semantically, the results make sense. They discuss the correct product, the correct type of problem, and the correct general topic. But none of them contains ZX-4821 .

The embedding model understood everything except the one string that mattered.

Now consider a second question:

Can I get my money back if the product arrived broken?

The relevant policy never uses the words "money back" or "broken". It says that "customers are eligible for a refund when an item is delivered in a defective condition". A lexical search may miss it entirely. Dense retrieval, however, can recognize that the two sentences express nearly the same idea.

Neither system is malfunctioning. They are preserving different kinds of information.

Lexical retrieval treats the exact words in the query as evidence. Dense retrieval tries to look beyond those words and recover their meaning. The first is strong when a rare name, number, acronym, or product code is decisive. The second is strong when the query and the answer describe the same concept using different language.

The history of information retrieval is often presented as a sequence of replacements. Boolean search gave way to ranked retrieval. TF–IDF was improved by probabilistic methods such as BM25. Sparse vectors were followed by latent representations, and latent representations eventually gave way to neural embeddings.

Read this way, the story appears to be a straight line: every generation understood language a little better than the previous one. But retrieval did not evolve by replacing bad methods with uniformly better ones. Instead, at every step, we seemed to purchase a new capability by weakening another.

Exact matching preserved rare terms but could not cross the vocabulary gap. Latent methods crossed that gap by compressing the original word space. Dense embeddings became remarkably good at paraphrases, but their semantic smoothness could make an exact identifier less distinctive.

This is why a ranking function introduced in the 1990s still appears inside modern AI search systems. BM25 did not survive because the field failed to invent something newer. It survived because dense retrieval solved a different problem.

This article follows that trade-off through the history of search, from exact keyword matching to dense vector retrieval and hybrid search. At each step, we will ask the same two questions: what did the new method learn to preserve, and what did it quietly agree to lose?

The question is not whether old search or new search wins. It is whether progress in retrieval has ever allowed us to stop choosing what kind of information we are willing to lose.

1. When Search Required the Exact Word

#### Boolean retrieval was precise, fast and completely literal.

Early computerized retrieval systems treated search as a logical operation. A query such as damaged AND product NOT warranty returned documents containing the required terms and excluded those containing the forbidden one. Behind this process sat the inverted index: instead of scanning every document, the system stored each term alongside a list of the documents in which it appeared.

Documents

Inverted index

D1: The product arrived damaged .

product → D1, D2

D2: The customer returned the product .

damaged → D1

D3: The refund was approved.

returned → D2

-

refund → D3

Instead of scanning every document, the system can jump directly from a term to the documents containing it. This reversal from document → words to word → documents is what makes the index “inverted.”

This design was efficient and wonderfully precise when the exact word mattered. A product code such as FSM-133i either appeared in a document or it did not. But the system had no concept of semantic proximity: a document containing defective item could not be considered partially relevant to a query about a broken product unless someone explicitly connected those expressions.

The problem was not that Boolean retrieval ranked the wrong document. It produced no ranking at all: a document matched the logical conditions or remained outside the result set. To move beyond exact words, retrieval first needed a way to say that one document was closer to a query than another. For that, words and eventually entire documents would have to become numbers!

2. When Relevance Became a Score

#### Documents stopped matching the query and started competing with each other.

Boolean retrieval could tell us which documents satisfied a query, but not which one deserved to appear first. The vector space model changed that by representing every document along with the query itself as a vector whose dimensions corresponded to words. With the vocabulary [product, damaged, refund] , for example, the query "damaged product" could become [1,1,0], while a document about a "product refund" could become [1,0,1].

Once queries and documents occupied the same space, relevance could be measured geometrically. Cosine similarity compared the direction of their vectors:

cos

⁡

(

q

,

d

)

=

⊤

∣

\cos(q,d)=\frac{q^\top d}{|q||d|}

∣∣

​

Where, ( q q q ) is the query and ( d d d ) is a candidate document.

A document containing both "damaged" and "product" pointed more closely toward the query than one sharing only "product". Search was no longer a gate producing matches and non-matches. It became a ranking problem.

Then, raw word counts treated every term as equally informative; TF–IDF corrected this by increasing the weight of terms that were frequent inside one document but rare across the collection. The word "product", appearing almost everywhere, contributed little. A term such as FSM-133i , appearing in only two documents, could dominate the ranking.

This was a major advance: retrieval gained degrees of relevance while preserving the precision of exact terms. Next problem in line was its geometry that was still built from shared vocabulary. Thus, "broken product" and "defective item" occupied different dimensions, even if they meant nearly the same thing. Search had learned how to rank words but it had not yet learned how to look beyond them.

3. The 1994 Algorithm That Refused to Die

#### BM25 learned that the tenth repetition of a word should not count like the first.

TF–IDF introduced an essential idea: a rare term should contribute more to relevance than one appearing in almost every document. But term frequency created another problem. If a document repeated "damaged" twenty times, it did not necessarily deserve twenty times the relevance of a document using it once. Longer documents also had an unfair advantage: they contained more words and therefore more opportunities to match the query.

BM25 kept the rare-term logic of TF–IDF but added two corrections. Repeating a term should eventually stop helping, and a long document should not win merely because it has more opportunities to contain the query words. The equation looks heavier than the idea:

∑

t

∈

IDF

f

k

1

+

−

b

avgdl

\sum_{t\in q} \operatorname{IDF}(t) \frac{f(t,d)(k_1+1)} {f(t,d)+k_1\left(1-b+b\frac{|d|}{\operatorname{avgdl}}\right)}

Here:

• q q q is the query and d d d is a candidate document.

• t t t is one of the terms appearing in the query.

• f ( t , d ) f(t,d) f ( t , d ) counts how many times term t t t appears in document d d d .

• The term IDF ⁡ ( t ) \operatorname{IDF}(t) IDF ( t ) measures how rare that term is across the collection.

• ∣ d ∣ \left| d \right| ∣ d ∣ is the length of the document, while avgdl ⁡ \operatorname{avgdl} avgdl is the average document length.

• k 1 k_1 k 1 ​ controls how quickly repeated occurrences stop adding much value.

• b b b controls how strongly the score corrects for document length.

In plain language, BM25 combines three ideas:

BM25 score

≈

term rarity

×

saturated frequency

length correction

\text{BM25 score} \approx \text{term rarity} \times \text{saturated frequency} \times \text{length correction}

Consider one more time our error code ZX-4821 . Its IDF will be high because it appears in very few documents, making it much more informative than a common term such as "device". If one document repeats ZX-4821 ten times, its term frequency increases, but the fraction gradually saturates like having a diminishing return: the tenth occurrence contributes far less than the first. If that document is also unusually long, the ratio ∣ d ∣ avgdl ⁡ \frac{|d|}{\operatorname{avgdl}} avgdl ∣ d ∣ ​ reduces the advantage it gains simply from containing more text.

This makes BM25 a remarkably disciplined lexical retriever. It preserves the precision of exact identifiers, names, numbers and rare terms, while preventing repetition and document length from dominating the ranking. A short document that directly explains ZX-4821 can outrank a fifty-page manual that happens to mention it several times.

But BM25 still rewards only words it can observe. If the query says "broken power supply" and the relevant document says "defective energy unit", their conceptual similarity does not appear anywhere in the equation. No setting of ( k 1 k_1 k 1 ​ ) or ( b b b ) can create a relationship between terms that never match.

BM25 became extremely good at deciding which exact words mattered but the next generation of retrieval would attempt something more ambitious: finding shared meaning when the words themselves were different.

4. When Search Looked Beneath the Words

#### Latent Semantic Indexing compressed the vocabulary to recover relationships that exact matching could not see.

On continue and as said before BM25 had become remarkably good at evaluating which shared words mattered the most, but the phrase "shared words" remained its limitation. A query about a "broken power supply" could still miss a document describing a "defective energy unit", even though both referred to the same problem.

Latent Semantic Indexing (LSI) approached this vocabulary gap through the structure of the entire document collection. It began with the same term–document matrix used by lexical methods:

Term

D1

D2

D3

broken

0

defective

power

refund

Every row represented a term, every column a document, and every value recorded the term’s presence or weight. In a realistic corpus, this matrix could contain hundreds of thousands of terms while remaining mostly empty.

LSI applied truncated Singular Value Decomposition:

X

U

Σ

V

X\approx U_k\Sigma_kV_k^\top

The full matrix might contain one dimension for every word in the vocabulary. Truncated SVD kept only the k k k strongest directions. U k U_k U k ​ represented terms in this reduced space, V k V_k V k ​ represented documents, and Σ k \Sigma_k Σ k ​ recorded the importance of each latent direction.

The key change was that the new dimensions no longer corresponded to individual words. One latent direction might capture a pattern shared by broken , defective , failure and repair because those terms repeatedly appeared in similar documents. A query containing "broken power supply" could therefore move closer to a document about a "defective energy unit", even without an exact match on the most important words.

This was an early form of semantic retrieval even though the model did not understand language in the modern sense. It was just able to discover statistical patterns of co-occurrence as words were becaming related due to their appearance in similar textual environments.

Compression, however, came with a price. The latent dimensions were harder to interpret than the original vocabulary, and truncated SVD deliberately discarded directions considered less important to the overall matrix. A rare but decisive identifier such as ZX-4821 might explain very little of the corpus-wide structure. The same compression that connected "broken" with "defective" could weaken the exact signal that made the error code valuable.

LSI therefore introduced a trade-off that still defines retrieval today:

By moving beyond exact words, search gained semantic flexibility—but began giving up lexical precision.

The next generation would push this idea much further. Instead of extracting latent directions from a term–document matrix, neural models would learn dense representations directly from language.

5. When Meaning Became Learned

#### Dense retrieval learned which texts should be close instead of discovering proximity through matrix compression.

Latent Semantic Indexing had shown that retrieval could move beyond exact word overlap. But its semantic structure came from a linear decomposition of one particular term-the document matrix. Change the corpus, and the latent space changes with it. The model could only discover relationships present in the co-occurrence patterns of the documents it had been given.

Neural representations changed where that knowledge came from. Early word embeddings such as Word2Vec learned that words appearing in similar contexts should receive similar vectors. Instead of giving "broken" and "defective" separate, unrelated dimensions, the model could place them near each other in a learned space. This was the distributional hypothesis turned into geometry: words used in similar environments tend to carry related meaning.

But one vector per word was still restrictive. The word "bank" received the same representation in "river bank" and "investment bank". Transformer-based models later introduced contextual representations, allowing the vector assigned to a token to depend on the surrounding sentence. Meaning was no longer attached only to the word itself, but to the word in context.

Modern dense retrieval extends this idea from words to entire queries and documents. An encoder maps a piece of text into a fixed-size vector:

θ

text

→

R

f_\theta(\text{text})\rightarrow\mathbb{R}^d

A 30-word query and a 300-word passage may both become vectors with, for example, 768 dimensions. The dimensions no longer correspond to recognizable terms or topics. They are learned coordinates that help the model place related texts close together.

The Bi-Encoder

Most dense retrieval systems use a bi-encoder architecture. The query and each document are encoded separately:

query

i

g

ϕ

document

q=f_\theta(\text{query}), \qquad d_i=g_\phi(\text{document}_i)

This separation is operationally important. Document embeddings can be computed once, stored in an index and reused for every query. At search time, the system only needs to encode the new query and compare its vector with the stored document vectors.

Similarity is usually measured with a dot product or cosine similarity: q ⊤ d i ∣ q ∣ ∣ d i ∣ \frac{q^\top d_i}{|q||d_i|} ∣ q ∣∣ d i ​ ∣ q ⊤ d i ​ ​

The documents with the highest scores become the retrieved neighbourhood.

This looks similar to the vector space model we saw earlier, but the source of the geometry is different. TF–IDF creates vectors directly from observable words. LSI compresses those word dimensions using SVD. A dense retriever, on the other hand, learns the space through examples of what should and should not be considered relevant.

Teaching the Space What “Close” Means

During training, the model receives a query, a relevant document d + d^+ d + , and one or more irrelevant documents d − d^- d − . Its objective is simple:

L

log

exp

sim

/

τ

j

L = -\log \frac{\exp(\operatorname{sim}(q,d^+)/\tau)} {\exp(\operatorname{sim}(q,d^+)/\tau)+ \sum_j\exp(\operatorname{sim}(q,d_j^-)/\tau)}

lo

The temperature τ \tau τ controls how sharply the model distinguishes between candidates. The numerator rewards a high score for the relevant document, while the denominator forces it to compete against the negatives. Training repeatedly adjusts the encoders until relevant query–document pairs occupy nearby regions of the space.

This means that a query such as:

The device stopped working because its power supply is broken.

can retrieve a passage saying:

A defective energy unit may prevent the device from starting.

The two texts share little exact vocabulary, but the model has learned that they describe a similar situation. This is the ability lexical retrieval could never obtain from term matching alone.

What the Vector Must Leave Behind

Dense retrieval solves much of the vocabulary problem, but it introduces a different form of compression. The entire query or document must pass through one fixed-size vector:

many tokens, relations and details

⟶

one dense representation

\text{many tokens, relations and details} \longrightarrow \text{one dense representation}

The model preserves the information that its training objective has taught it to treat as useful for similarity. Broad meaning, intent and paraphrase may survive extremely well. But exact lexical identity is no longer protected explicitly.

Consider our original question:

A dense retriever may correctly place it near documents about ZX devices, diagnostic errors and power failures. Yet nothing in cosine similarity guarantees that a passage containing the exact string ZX-4821 will outrank a semantically similar troubleshooting guide that never mentions it. The issue is not that the model cannot process the code. Modern encoders can assign it a representation. The problem is that the code must compete with every other aspect of the text for influence over the final vector. Unless the model’s training has made exact identifiers important, the surrounding semantic topic may dominate.

BM25 handles the situation differently. If ZX-4821 is rare and appears in both query and document, its high IDF contributes directly to the score. The lexical match cannot quietly dissolve into the rest of the representation.

Dense retrieval therefore completed the transition that LSI had begun:

Search gained powerful semantic flexibility, but exact lexical evidence became implicit rather than guaranteed.

This is not a story in which dense retrieval replaced BM25 because it understood language better. The two methods preserve different evidence. Dense retrieval is designed to recognize the same meaning behind different words. BM25 is designed to recognize when the same rare word appears on both sides.

For a paraphrase, semantics may be exactly what we need. For a serial number, product code, legal clause or dosage, the exact string may be the evidence.

The modern retrieval problem is not choosing which of these abilities belongs to the past. It is deciding how to keep both.

6. Why Hybrid Search Exists

#### Semantic similarity and lexical identity preserve different kinds of evidence.

At this point, the history of retrieval stops looking like a sequence of replacements. BM25 and dense retrieval are not older and newer solutions to exactly the same problem. They answer two different questions.

BM25 asks:

Which documents contain the same informative terms as the query?

Dense retrieval asks:

Which documents express a similar meaning to the query?

For "broken power supply" and "defective energy unit", dense retrieval has the natural advantage. For ZX-4821 , BM25 can treat the exact match as decisive evidence. A production query may need both:

Can error ZX-4821 cause the device to lose power?

Hybrid retrieval runs both searches and combines their rankings into the following schema:

Query
├── BM25              → lexical ranking
└── Dense retriever   → semantic ranking
↓
fused ranking

The raw scores cannot simply be added. A BM25 score of 14.7 and a cosine similarity of 0.82 live on unrelated scales and thus in different worlds. One practical solution is Reciprocal Rank Fusion (RRF), which ignores the original scores and uses only the position of each document:

c

rank

\sum_{r\in R} \frac{1}{c+\operatorname{rank}_r(d)}

Here, R R R is the set of rankings, rank ⁡ r ( d ) \operatorname{rank}_r(d) rank r ​ ( d ) is the position of document d d d in ranking r r r , and c c c is a constant that prevents the first position from dominating too aggressively.

Consider three candidate documents:

BM25 rank

Dense rank

RRF score

Changelog mentioning ZX-4821 without explaining it

8

0.0311

Generic guide to power failures

—

0.0164

Document explaining what ZX-4821 means

2

3

0.0320

Two details matter here: The generic troubleshooting guide falls outside the BM25 candidate list because it does not contain the exact error code. It therefore receives no contribution from the lexical retriever. The changelog has the opposite problem: it contains ZX-4821 , so BM25 places it first, but it merely mentions the code without explaining it.

The third document ranks first in neither list. BM25 places it second because it contains the exact code, while the dense retriever places it third because its meaning closely matches the query. It is the only document supported strongly by both retrieval signals.

In Reciprocal Rank Fusion, a missing result contributes zero from that retriever. Using c = 60 c=60 c = 60 , the score of the third document is:

60

≃

0.032

RRF(d) = \frac{1}{60+2}+\frac{1}{60+3}\simeq0.032

That is enough to move it above both specialists: the changelog preferred by BM25 and the generic guide preferred by dense retrieval.

This is the logic behind hybrid search. The winning document does not have to rank first in either system. It can win because two imperfect retrievers independently agree that it deserves to be near the top. After decades of progress, modern retrieval did not finally choose between words and meaning. It learned to search with both.

Conclusion: Retrieval Did Not Replace Its Past

The history of retrieval is often told as a sequence of replacements. Boolean search gave way to ranked retrieval. TF–IDF was improved by BM25. Sparse representations were followed by latent spaces, and latent spaces eventually became learned embeddings.

But that is not what actually happened.

Each generation solved a weakness of the previous one by accepting a new weakness of its own. Boolean retrieval preserved exactness but could not rank results. TF–IDF weighted terms by informativeness. BM25 improved lexical relevance but still could not understand paraphrases. LSI looked beneath individual words but compressed away some lexical detail. Dense retrieval learned semantic similarity but became less reliable when exact terms mattered.

That is why BM25 survived.

It is still used not because information retrieval failed to progress after 1994, but because neural retrieval solved a different problem. A dense model can recognize that "the device suddenly switched off" resembles "unexpected power failure". BM25 can recognize that ZX-4821 is not ZX-4812 . A production search system may need both facts at the same time.

Hybrid search is therefore not an awkward compromise between an old algorithm and a modern one. It is an acknowledgement that relevance has more than one dimension. Meaning matters. Exact wording matters. Rare identifiers matter. And no single representation preserves all of them equally well.

The Practical Takeaway

If you are building a RAG system, do not treat dense retrieval as the automatic starting point and BM25 as a historical baseline. Treat them as two different sensors.

Run them separately first. Inspect the queries where they disagree. Dense retrieval will often win on paraphrases, natural-language descriptions, and conceptual similarity. BM25 will often win on product codes, names, numbers, error messages, technical terminology, and rare expressions.

Then combine their ranked lists with a simple method such as Reciprocal Rank Fusion. You do not need to redesign the entire pipeline or train a new model. The first useful hybrid baseline may require only two retrievers and a few lines of ranking logic.

Most importantly, evaluate the retrieval stage independently from the final LLM answer. If the necessary document never enters the retrieved context, prompt engineering cannot recover it. Before changing the model, inspect the neighbourhood it was given.

Your RAG pipeline may contain the newest language model available. That does not mean every component around it should be new. Sometimes the missing piece is an algorithm from 1994.

References

• Salton, G., Wong, A., & Yang, C. S. (1975). A vector space model for automatic indexing. Communications of the ACM, 18(11), 613–620.

• Spärck Jones, K. (1972). A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1), 11–21.

• Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press.

• Robertson, S. E., Walker, S., Jones, S., Hancock-Beaulieu, M., & Gatford, M. (1995). Okapi at TREC-3. Proceedings of TREC-3.

• Robertson, S., & Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 3(4), 333–389.

• Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., & Harshman, R. (1990). Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6), 391–407.

• Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781.

• Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL.

• Cormack, G. V., Clarke, C. L. A., & Büttcher, S. (2009). Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods. SIGIR.

• Thakur, N., et al. (2021). BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS Datasets & Benchmarks.

• Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS.