ThesaHelp: needle-in-a-haystack test of directed search

Summary

Use the needle-in-a-haystack test to estimate recall and precision for large collections. In a large collection, an exhaustive search is impractical, making recall and precision difficult to measure.

1. Have someone select random quotations in Thesa, and write questions which the quotations help answer. The question should not use keywords from the quotation.
   
2. Ask a subject to locate all material relevant to the question.
   
3. If the subject does not select the quotation, ask whether it is relevant and if not, why not.
   
4. Repeat this procedure to estimate the recall and precision ratios. If subject looks at 200 items, identifies 70 relevant items, and locates 90% of the selected quotations, the recall is 0.90 and the precision is 0.35.
   
5. You may use the same procedure, to test keyword searching.
   

Related Topics

ThesaHelp: a preliminary test of directed search
ThesaHelp: read-everything test of directed search
Topic: information retrieval by searching
(35 items)