Course map 14 lessons

LESSON 05 / 14 · Learn to retrieve

BM25: why a rare word can beat repeated words

Work through term frequency, rarity and length normalization. Tune tokenization and field weights before chasing a larger model.

5 min readAuthor: MESA contributorsPublished: Download Markdown ↓
BY THE END, YOU CAN
  • Read the BM25 formula
  • Calculate a document-length effect
  • Design exact-identifier and Turkish tokenization tests

Before this lesson: Embeddings and semantic search, with numbers

Three questions behind lexical ranking#

For “Atlas Friday hours”, a lexical retriever finds passages containing query terms. It needs to decide which match matters most. BM25 combines term rarity, within-document frequency and a correction for document length. Repeating “hours” fifty times should not make a long irrelevant document unbeatable; an uncommon identifier may be a more discriminating clue.

An inverted index maps a term to the documents containing it. Tokenization determines those terms before ranking begins. If “A-17” is split into “a” and “17”, a scoring adjustment cannot restore an identifier boundary that was never indexed. Inspect tokens and query parsing first. Exact identifiers may need a separate normalized field or equality filter.

Read the formula one factor at a time#

Worked example
score(d,q) = sum over query terms t:
 IDF(t) * f(t,d)*(k1+1) /
 [f(t,d) + k1*(1-b + b*len(d)/avgdl)]

One common non-negative IDF variant:
IDF(t) = ln(1 + (N - df(t) + 0.5)/(df(t) + 0.5))
N: corpus documents   df: documents containing t
f: term frequency    avgdl: average document length

IDF gives rarer terms greater influence. The frequency fraction saturates: going from zero to one occurrence matters more than going from ten to eleven. k1 controls that saturation; b controls length normalization. At b=0, length has no effect. At b=1, the full length ratio enters. BM25 variants differ in IDF details, so record the implementation as well as the parameter names.

Calculate the length effect#

QuantityShort passage ALong passage B
Term frequency22
Length / average length0.52
With k1=1.2, b=0.75: denominator2.754.10
Numerator: 2 × 2.24.44.4
Frequency factor1.601.073

With the same IDF and term frequency, the shorter passage receives the larger contribution. This is arithmetic, not evidence that short chunks always win. Splitting documents changes term statistics, context completeness and the unit judged relevant. Keep chunking fixed when testing a ranking change, then evaluate chunking as its own experiment.

Tune against failure cases#

  1. Inspect · A-17 / I / İ
  2. Baseline · fixed chunks
  3. Judge · exact and paraphrase
  4. Change · one field weight
  5. Compare · held-out queries
Do not tune on the same examples used for your final score.

For Turkish, inspect dotted/dotless I, inflections and diacritics using real query/document pairs. Lowercasing and accent removal can help some matches while collapsing meaningful distinctions. Preserve original text and test both identifier precision and ordinary-language recall. Search syntax and tokenizer behavior are separate from your UI’s text normalization.

SQLite FTS5 uses fixed k1=1.2 and b=0.75; its bm25(table, ...) arguments are column weights, not these parameters. It also returns smaller values for better matches, so ascending order is intentional. Do not copy a descending-score rule from another engine. Our worked formula uses an explicit IDF variant; FTS5’s exact scoring definition is in its own reference. FTS5.

Choose the first debugging step#

Check your understanding

Every search for policy A-17 returns policy A-71 first. Should you immediately increase a ranking weight?

Reveal the explanation

Inspect indexed tokens and query syntax, then test an exact identifier field. If the index no longer distinguishes the identifiers, weighting cannot reliably repair it. Only tune scores after the representation is correct.

Sources and further reading#

  1. Manning, Raghavan & Schütze · Okapi BM25
  2. SQLite · FTS5 reference
Ready for the next idea?

Try the question before continuing. Your progress is saved in this browser when storage is available.

← Back to the learning map
In this lesson