Ulster University Logo

Ulster Institutional Repository

Several methods of ranking retrieval systems with partial relevance judgment

Biomedical Sciences Research Institute Computer Science Research Institute Environmental Sciences Research Institute Nanotechnology & Advanced Materials Research Institute

Wu, Shengli and McClean, Sally (2007) Several methods of ranking retrieval systems with partial relevance judgment. In: 2nd International Conference on Digital Information Management, 2007 (ICDIM '07), Lyon, France. IEEE. 6 pp. [Conference contribution]

Full text not available from this repository.

URL: http://dx.doi.org/10.1109/ICDIM.2007.4444193

DOI: doi:10.1109/ICDIM.2007.4444193

Abstract

Some measures such as mean average precision and recall level precision are considered as good system-oriented measures, because they concern both precision and recall that are two important aspects for effectiveness evaluation of information retrieval systems. However, such good system-oriented measures suffer from some shortcomings when partial relevance judgment is used. In this paper, we discuss how to rank retrieval systems in the condition of partial relevance judgment, which is common in major retrieval evaluation events such as TREC conferences and NTCIR workshops. Four system-oriented measures, which are mean average precision, recall level precision, normalized discount cumulative gain, and normalized average precision over all documents, are discussed. Our investigation shows that averaging values over a set of queries may not be the most reliable approach to rank a group of retrieval systems. Some alternatives such as Bar da count. Condorcet voting, and the zero-one normalization method, are investigated. Experimental results are also presented for the evaluation of these methods.

Item Type:Conference contribution (Paper)
Faculties and Schools:Faculty of Computing & Engineering
Faculty of Computing & Engineering > School of Computing and Information Engineering
Faculty of Computing & Engineering > School of Computing and Mathematics
Research Institutes and Groups:Computer Science Research Institute
Computer Science Research Institute > Artificial Intelligence and Applications
Computer Science Research Institute > Information and Communication Engineering
ID Code:14635
Deposited By:Professor Sally McClean
Deposited On:10 Aug 2010 11:08
Last Modified:10 Aug 2010 11:08

Repository Staff Only: item control page