BENCHMARK · MESA 0.9.3

Turkish legal retrieval benchmark

Same retrieval quality, faster ingest: with the built-in reranker, 0.9.2 and 0.9.3 both reach Hit@5 74.4% and MRR 0.656 at the displayed precision.

74.4%

Hit@5

Correct passage in the first five results

57.9%

Hit@1

Correct passage in the first result

0.656

MRR

Ranking of the first correct passage

100.0%

Citation Hit@5

Explicit citation questions

MESA 0.9.3 · BGE-M3 + built-in BGE reranker (T2)

Compare results

Hit@5 by configurationMESA 0.9.2MESA 0.9.3

Models off

39.7%38.0%-1.7 pp

BGE-M3 embeddings

67.8%67.8%0.0 pp

Built-in reranker

74.4%74.4%0.0 pp

Passage retrieval comparison
Configuration0.9.20.9.3Hit@5 change
Hit@5MRRHit@5MRR
Models off39.7%0.30038.0%0.297-1.7 pp
BGE-M3 embeddings67.8%0.51467.8%0.5220.0 pp
Built-in reranker74.4%0.65674.4%0.6560.0 pp

Changes are in percentage points, calculated as selected version minus reference. Profiles are matched by model setup. Search settings and hardware can differ. See the individual reports for timing and offline experiments.

Hit@5 by configuration

Models off

MESA 0.9.033.1%
MESA 0.9.239.7%
MESA 0.9.338.0%

BGE-M3 embeddings

MESA 0.9.051.2%
MESA 0.9.267.8%
MESA 0.9.367.8%

Built-in reranker

MESA 0.9.0Unavailable
MESA 0.9.274.4%
MESA 0.9.374.4%

Search results

November 18, 2026 · 141 documents · 121 / 145 answerable test questions

Built-in configurations
ConfigurationHit@5Hit@1MRRHit@10Doc@1Doc@5p50p95Hardware
0.9.0 · Keyword search (Mode A)20261029T190820Z33.1%22.3%0.29338.8%36.4%60.3%0.96 s1.93 sColab T4
0.9.0 · BGE-M3, equal weights (S0)20261029T234645Z_exp51.2%34.7%0.43160.3%58.7%70.3%1.91 s3.49 sColab A100
0.9.0 · BGE-M3, tuned vector weight (S4)20261029T234645Z_exp62.0%39.7%0.50070.3%60.3%75.2%1.87 s3.29 sColab A100
0.9.2 · Models off (T0)20261107T141454Z39.7%22.3%0.30046.3%44.6%57.9%0.45 s0.94 sColab A100
0.9.2 · Magibu embeddings (T1m)20261107T141454Z50.4%32.2%0.41059.5%52.9%68.6%1.39 s2.33 sColab A100
0.9.2 · BGE-M3 embeddings (T1b)20261107T141454Z67.8%38.0%0.51476.9%56.2%76.9%1.35 s2.27 sColab A100
0.9.2 · BGE-M3 + built-in BGE reranker (T2)20261107T141454Z74.4%57.9%0.65681.0%69.4%79.3%1.98 s2.98 sColab A100
0.9.3 · Models off (T0)20261117T230132Z_v09338.0%22.3%0.29747.9%43.8%57.9%0.45 s0.94 sColab A100-SXM4-80GB
0.9.3 · BGE-M3 embeddings (T1b)20261117T230132Z_v09367.8%38.8%0.52277.7%57.9%77.7%1.37 s2.36 sColab A100-SXM4-80GB
Offline experiments
ConfigurationHit@5Hit@1MRRHit@10Doc@1Doc@5p50p95Hardware
0.9.0 · Article chunks + offline Jina20261029T234645Z_exp76.0%57.0%0.65480.2%69.4%82.6%——Colab A100

Results by question type

Hit@5 per question type. Darker cells mean the right passage was found more often.

Correct passage in the first five results
Question type0.9.0 · Keyword search (Mode A)0.9.0 · BGE-M3, equal weights (S0)0.9.0 · BGE-M3, tuned vector weight (S4)0.9.0 · Article chunks + offline Jina0.9.2 · Models off (T0)0.9.2 · Magibu embeddings (T1m)0.9.2 · BGE-M3 embeddings (T1b)0.9.2 · BGE-M3 + built-in BGE reranker (T2)0.9.3 · Models off (T0)0.9.3 · BGE-M3 embeddings (T1b)0.9.3 · BGE-M3 + built-in BGE reranker (T2)
Exact wording (36) 66.7%86.1%86.1%94.4%75.0%77.8%86.1%97.2%69.4%88.9%97.2%
Paraphrase (39) 10.3%28.2%56.4%79.5%15.4%48.7%69.2%71.8%18.0%69.2%71.8%
Cross-document (17) 11.8%47.1%47.1%70.6%35.3%23.5%58.8%76.5%35.3%52.9%76.5%
Number, date or docket (29) 34.5%41.4%48.3%51.7%31.0%34.5%48.3%48.3%27.6%48.3%48.3%

Citation questions

0.9.3 keeps the 0.9.2 result: explicit citation Hit@5 is 100.0% (50/50) for T0, T1b and T2. Natural wording is 97.5% (39/40) for T0 and T1b, and 100.0% (40/40) for T2.

Citation questions, Hit@5
ConfigurationExplicit citation (50)Natural wording (40)
0.9.2 · Models off (T0)100.0%97.5%
0.9.2 · Magibu embeddings (T1m)100.0%95.0%
0.9.2 · BGE-M3 embeddings (T1b)100.0%97.5%
0.9.2 · BGE-M3 + built-in BGE reranker (T2)100.0%100.0%
0.9.3 · Models off (T0)100.0%97.5%
0.9.3 · BGE-M3 embeddings (T1b)100.0%97.5%
0.9.3 · BGE-M3 + built-in BGE reranker (T2)100.0%100.0%

Time until searchable

Full corpus ingestion
ConfigurationTotal timeChunksSeconds per chunkHardware
0.9.0 · BGE-M3, equal weights (S0)29.7 min3,4800.51 sColab A100
0.9.2 · Models off (T0)15.8 min5,0410.19 sColab A100
0.9.2 · Magibu embeddings (T1m)23.7 min5,0410.28 sColab A100
0.9.2 · BGE-M3 embeddings (T1b)20.7 min5,0410.25 sColab A100
0.9.3 · Models off (T0)7.4 min5,0410.09 sColab A100-SXM4-80GB
0.9.3 · BGE-M3 embeddings (T1b)12.0 min5,0410.14 sColab A100-SXM4-80GB

0.9.3 batches ingest work; retrieval quality is unchanged.

Download results

MESA 0.9.0 · Report and aggregate metrics (JSON) ↓

MESA 0.9.2 · Report and aggregate metrics (JSON) ↓

MESA 0.9.3 · Report and aggregate metrics (JSON) ↓

How we measured

Tier M contains 141 Turkish laws, regulations and court decisions. A language model generated questions from the corpus. A separate model and rule checks reviewed them. Configurations were chosen on 101 development questions before evaluation on 145 held-out questions. Retrieval scores use the 121 answerable questions.

What do the metrics mean?
Hit@1 · Hit@5 · Hit@10
Share of questions with the correct passage in the first 1, 5 or 10 results.
MRR
Average reciprocal rank of the first correct passage. A score of 1 means it always came first.
Doc@1 · Doc@5
Share of questions with the correct document among the first 1 or 5 results.
p50 · p95
Half of queries finished within p50 and 95% within p95. Both are measured in seconds.

Scope and limitations

The test measures whether MESA retrieves the correct passage or document. The 24 questions with no answer in the corpus are outside the retrieval scores. The corpus is small and its decisions come from the Court of Cassation. The questions are synthetic.

For number, date and docket questions, the T2 result is unchanged in 0.9.3: 14/29 = 48.3% Hit@5. The benchmark requests 50 results. With a 10-result request the reranker sees 20 candidates; on the dev split Hit@5 was 75.3% instead of 77.7%. This comparison changes both the candidate pool and the number of returned results.

0.9.3 run: 18 November 2026 · mesa-memory 0.9.3 · 20261117T230132Z_v093.

NEXT STEP

Where to go from here