Short answer

Freeze inputs, separate query creation from judgment, report multiple metrics and failure cases, and bind every result to versions and raw evidence.

Scope#

This method applies to comparative retrieval evaluations. It does not turn one corpus or query set into a universal performance claim.

Dataset and query selection#

  • Define source eligibility and temporal cut-off.
  • Record licensing and preservation constraints.
  • Version corpus, queries, and exclusions.
  • Include difficult and zero-result cases deliberately.

Relevance judgment#

Judgment guidance should define graded relevance, evidence requirements, disagreement handling, and reviewer independence. System output must not silently define its own ground truth.

Metrics#

MeasureQuestion
Recall@kWas relevant material retrieved within the cut-off?
nDCG@kWere more relevant items ranked earlier?
MRRHow early did the first relevant item appear?
Latency distributionWhat operational cost accompanied retrieval?

Reporting and corrections#

  • Configuration and code revision
  • Dataset, query, and judgment versions
  • Aggregate and per-query outputs
  • Failure cases and exclusions
  • Known limitations
  • Correction date and change record when findings are amended