BENCHMARK · MESA 0.9.2
MESA 0.9.2 benchmark report
With the built-in BGE reranker, the correct passage was first in 57.9% of answerable cases and in the top five in 74.4%. Number, date and docket questions were the weakest group at 48.3% Hit@5.
74.4%
Hit@5
Correct passage in the first five results
57.9%
Hit@1
Correct passage in the first result
0.656
MRR
Ranking of the first correct passage
MESA 0.9.2 · BGE-M3 + built-in BGE reranker (T2)
Search results
November 7, 2026 · 141 documents · 121 / 145 answerable test questions
| Configuration | Hit@5 | Hit@1 | MRR | Hit@10 | Doc@1 | Doc@5 | p50 | p95 | Hardware |
|---|---|---|---|---|---|---|---|---|---|
| Models off (T0)20261107T141454Z | 39.7% | 22.3% | 0.300 | 46.3% | 44.6% | 57.9% | 0.45 s | 0.94 s | Colab A100 |
| Magibu embeddings (T1m)20261107T141454Z | 50.4% | 32.2% | 0.410 | 59.5% | 52.9% | 68.6% | 1.39 s | 2.33 s | Colab A100 |
| BGE-M3 embeddings (T1b)20261107T141454Z | 67.8% | 38.0% | 0.514 | 76.9% | 56.2% | 76.9% | 1.35 s | 2.27 s | Colab A100 |
| BGE-M3 + built-in BGE reranker (T2)20261107T141454Z | 74.4% | 57.9% | 0.656 | 81.0% | 69.4% | 79.3% | 1.98 s | 2.98 s | Colab A100 |
- All rows ran on a Colab A100. T2 reused the T1b store, so it has no separate ingestion measurement.
Results by question type
Hit@5 per question type. Darker cells mean the right passage was found more often.
| Question type | Models off (T0) | Magibu embeddings (T1m) | BGE-M3 embeddings (T1b) | BGE-M3 + built-in BGE reranker (T2) |
|---|---|---|---|---|
| Exact wording (36) | 75.0% | 77.8% | 86.1% | 97.2% |
| Paraphrase (39) | 15.4% | 48.7% | 69.2% | 71.8% |
| Cross-document (17) | 35.3% | 23.5% | 58.8% | 76.5% |
| Number, date or docket (29) | 31.0% | 34.5% | 48.3% | 48.3% |
Citation questions
| Configuration | Explicit citation (50) | Natural wording (40) |
|---|---|---|
| Models off (T0) | 100.0% | 97.5% |
| Magibu embeddings (T1m) | 100.0% | 95.0% |
| BGE-M3 embeddings (T1b) | 100.0% | 97.5% |
| BGE-M3 + built-in BGE reranker (T2) | 100.0% | 100.0% |
Time until searchable
| Configuration | Total time | Chunks | Hardware |
|---|---|---|---|
| Models off (T0) | 15.8 min | 5,041 | Colab A100 |
| Magibu embeddings (T1m) | 23.7 min | 5,041 | Colab A100 |
| BGE-M3 embeddings (T1b) | 20.7 min | 5,041 | Colab A100 |
Download results
Report and aggregate metrics (JSON) ↓How we measured
Tier M contains 141 Turkish laws, regulations and court decisions. A language model generated questions from the corpus. A separate model and rule checks reviewed them. Configurations were chosen on 101 development questions before evaluation on 145 held-out questions. Retrieval scores use the 121 answerable questions.
What do the metrics mean?
- Hit@1 · Hit@5 · Hit@10
- Share of questions with the correct passage in the first 1, 5 or 10 results.
- MRR
- Average reciprocal rank of the first correct passage. A score of 1 means it always came first.
- Doc@1 · Doc@5
- Share of questions with the correct document among the first 1 or 5 results.
- p50 · p95
- Half of queries finished within p50 and 95% within p95. Both are measured in seconds.
Scope and limitations
The test measures whether MESA retrieves the correct passage or document. The 24 questions with no answer in the corpus are outside the retrieval scores. The corpus is small and its decisions come from the Court of Cassation. The questions are synthetic.
