Methodology and Campaign Design for the Evaluation of Semantic Search Tools

author: Stuart N. Wrigley, Department of Computer Science, University of Sheffield
published: May 17, 2010,   recorded: April 2010,   views: 3942


Related content

Report a problem or upload files

If you have found a problem with this lecture or would like to send us extra material, articles, exercises, etc., please use our ticket system to describe your request and upload the data.
Enter your e-mail into the 'Cc' field, and we will keep you updated with your request's status.
Lecture popularity: You need to login to cast your vote.
  Delicious Bibliography


The main problem with the state of the art in the semantic search domain is the lack of comprehensive evaluations. There exist only a few e fforts to evaluate semantic search tools and to compare the results with other evaluations of their kind. In this paper, we present a systematic approach for testing and benchmarking semantic search tools that was developed within the SEALS project. Unlike other semantic web evaluations our methodology tests search tools both automatically and interactively with a human user in the loop. This allows us to test not only functional performance measures, such as precision and recall, but also usability issues, such as ease of use and comprehensibility of the query language. The paper describes the evaluation goals and assumptions; the criteria and metrics; the type of experiments we will conduct as well as the datasets required to conduct the evaluation in the context of the SEALS initiative. To our knowledge it is the first eff ort to present a comprehensive evaluation methodology for Semantic Web search tools.

See Also:

Download slides icon Download slides: www2010_wrigley_mcd_01.pdf (844.6 KB)

Download slides icon Download slides: www2010_wrigley_mcd_01.ppt (1.1 MB)

Help icon Streaming Video Help

Link this page

Would you like to put a link to this lecture on your homepage?
Go ahead! Copy the HTML snippet !

Write your own review or comment:

make sure you have javascript enabled or clear this field: