This document is a version of our Search Quality Rater Guidelines, which gives evaluators examples and guidelines for appropriate ratings. The document focuses on a type of rating task called “URL rating.” In this kind of task, the evaluator looks at a search query and a result that could be returned. They rate the relevance of the result for that query on a scale described within the document. Sounds simple, right? As you can see, there are many tricky cases to think through.
Download Now (English only)