Choose the unit of evaluation
Ranking metrics describe ordered results for a query or user. Define relevance judgments, eligibility, and the candidate universe before comparing systems. Average across queries carefully: pooling every item can let a few large query groups dominate. Include difficult and low-frequency queries, not only common ones.
Define relevance for a documentation search where an outdated answer is topically similar but unusable.
Calculate reciprocal rank
Reciprocal rank is one divided by the position of the first relevant result, or zero when no relevant result is found within the evaluation cutoff. First relevant positions of 1, 2, and no result give reciprocal ranks 1, 0.5, and 0. Their mean is 0.5. This measure emphasizes finding one useful answer early.
Recalculate the mean if the second query improves from rank 2 to rank 1. The result is 2/3.
Choose complementary measures
A single first-hit metric ignores additional relevant results. Precision at a cutoff describes how much of the returned list is relevant; recall describes how much of the labeled relevant set was recovered. Graded relevance measures can distinguish partially useful from fully useful results, but require consistent judgments.
A list has one excellent first result and nine irrelevant results. Explain why reciprocal rank alone looks strong.
Investigate disagreements
Compare systems on the same query set and inspect queries with the largest differences. Check labels, duplicate results, missing permissions, and stale documents before attributing an improvement to the ranking model. Preserve a final holdout because repeated tuning against the same judgments can overfit the benchmark.
Build a review table with query, first relevant position, duplicate count, and whether the answer is current.
What does reciprocal rank emphasize?
Keep exploring
Use the official certification page and its linked resources to validate your coverage. For guided labs, explore the Google Skills learning path. Revisit the practical exercise and explain your decision in your own words before moving on.