Exercising these ideas You have a description of each item in a small collection.

Exercising these ideas

• You have a description of each item in a small collection. (30 web sites)

• Assume we are looking for information about boxers, a type of dog.

• List the items that are relevant to this information need. (If it is impossible to tell, as with #9, mark it not relevant.)

• Rq = { }

• Assume that an unspecified search system has returned a ranked result as follows:

• (1, 4, 6, 8, 10, 12, 15, 16, 17, 18, 20, 23,26, 28, 30)
• Show the ranking for the query as done on page 76.

• Plot the precision vs. recall curve

Exercise, continued

• We do a second query against the same collection. This time we are interested in the Boxer rescue organizations.

• Rq = { }
• Ranked result = (4, 6, 7, 10, 11, 15, 16, 17, 18, 19, 22, 23, 25, 28, 29)

• Calculate the average precision at recall level 3 for these two queries.

More exercise

• We redo the first query, using a new search algorithm. The ranked result is now this:

• (4, 5, 6, 7, 8, 10, 11, 15, 16, 17, 18, 20, 23, 28, 29)

• Produce the average recall vs. precision figures for these two algorithms. How would you describe the performance?

• We redo the second query using the new search algorithm. The ranked result is this:

• (7, 10, 11, 15, 17, 18, 19, 21, 23, 24, 26, 28, 29, 30)

Precision Histogram

• For each of the four searches, what is the R-precision?

• Use a precision histogram to compare these two algorithms for the two queries shown.

• Calculate the harmonic mean at document 5 in the rankings, using the first query and repeating for each algorithm.

• Calculate the E measure for algorithm 1 using query 1 and a moderate preference for recall rather than precision.

Collection

• Assume that the even number elements in the collection are known to the user.

• Calculate the coverage ratio for algorithm 1 on search 1

• Calculate the novelty ratio for the same algorithm and search