BENCHMARK · MESA 0.9.0

MESA 0.9.0 benchmark report

Tuned BGE-M3 retrieved the correct passage in the first five results on 62.0% of answerable questions. The offline Jina experiment reached 76.0% with a separate article-aware pipeline.

62.0%

Hit@5

Correct passage in the first five results

39.7%

Hit@1

Correct passage in the first result

0.500

MRR

Ranking of the first correct passage

MESA 0.9.0 · BGE-M3, tuned vector weight (S4)

Search results

October 29, 2026 · 141 documents · 121 / 145 answerable test questions

Built-in configurations
ConfigurationHit@5Hit@1MRRHit@10Doc@1Doc@5p50p95Hardware
Keyword search (Mode A)20261029T190820Z33.1%22.3%0.29338.8%36.4%60.3%0.96 s1.93 sColab T4
BGE-M3, equal weights (S0)20261029T234645Z_exp51.2%34.7%0.43160.3%58.7%70.3%1.91 s3.49 sColab A100
Offline experiments
ConfigurationHit@5Hit@1MRRHit@10Doc@1Doc@5p50p95Hardware
Article chunks + offline Jina20261029T234645Z_exp76.0%57.0%0.65480.2%69.4%82.6%——Colab A100

Results by question type

Hit@5 per question type. Darker cells mean the right passage was found more often.

Correct passage in the first five results
Question typeKeyword search (Mode A)BGE-M3, equal weights (S0)BGE-M3, tuned vector weight (S4)Article chunks + offline Jina
Exact wording (36) 66.7%86.1%86.1%94.4%
Paraphrase (39) 10.3%28.2%56.4%79.5%
Cross-document (17) 11.8%47.1%47.1%70.6%
Number, date or docket (29) 34.5%41.4%48.3%51.7%

Time until searchable

Full corpus ingestion
ConfigurationTotal timeChunksHardware
BGE-M3, equal weights (S0)29.7 min3,480Colab A100

Download results

Report and aggregate metrics (JSON) ↓

How we measured

Tier M contains 141 Turkish laws, regulations and court decisions. A language model generated questions from the corpus. A separate model and rule checks reviewed them. Configurations were chosen on 101 development questions before evaluation on 145 held-out questions. Retrieval scores use the 121 answerable questions.

What do the metrics mean?
Hit@1 · Hit@5 · Hit@10
Share of questions with the correct passage in the first 1, 5 or 10 results.
MRR
Average reciprocal rank of the first correct passage. A score of 1 means it always came first.
Doc@1 · Doc@5
Share of questions with the correct document among the first 1 or 5 results.
p50 · p95
Half of queries finished within p50 and 95% within p95. Both are measured in seconds.

Scope and limitations

The test measures whether MESA retrieves the correct passage or document. The 24 questions with no answer in the corpus are outside the retrieval scores. The corpus is small and its decisions come from the Court of Cassation. The questions are synthetic.