{"id":224,"date":"2014-01-03T01:47:02","date_gmt":"2014-01-03T01:47:02","guid":{"rendered":"https:\/\/staff.fnwi.uva.nl\/m.derijke\/?p=224"},"modified":"2015-05-03T10:28:29","modified_gmt":"2015-05-03T10:28:29","slug":"wsdm-2014-paper-on-efficient-on-line-ranker-evaluation-online","status":"publish","type":"post","link":"https:\/\/staff.fnwi.uva.nl\/m.derijke\/wsdm-2014-paper-on-efficient-on-line-ranker-evaluation-online\/","title":{"rendered":"WSDM 2014 paper on efficient on-line ranker evaluation online"},"content":{"rendered":"<p>\u201cRelative Confidence Sampling for Efficient On-Line Ranker Evaluation\u201d by Masrour Zoghi, Shimon Whiteson, Maarten de Rijke and Remi Munos is\u00a0<a href=\"http:\/\/staff.science.uva.nl\/~mdr\/content\/publications\/wsdm2014-evaluation.pdf\" rel=\"self\">available<\/a>\u00a0online now.<\/p>\n<p>A key challenge in information retrieval is that of\u00a0<i>on-line ranker evaluation<\/i>: determining which one of a finite set of rankers performs the best in expectation on the basis of user clicks on presented document lists. When the presented lists are constructed using\u00a0<i>interleaved comparison methods<\/i>, which interleave lists proposed by two different candidate rankers, then the problem of minimizing the total\u00a0<i>regret<\/i>\u00a0accumulated while evaluating the rankers can be formalized as a\u00a0<i>K-armed dueling bandits problem<\/i>. In this paper, we propose a new method called\u00a0<i>relative confidence sampling<\/i>\u00a0(RCS) that aims to reduce cumulative regret by being less conservative than existing methods in eliminating rankers from contention. In addition, we present an empirical comparison between RCS and two state-of-the-art methods,\u00a0<i>relative upper confidence bound<\/i>\u00a0and<i>SAVAGE<\/i>. The results demonstrate that RCS can substantially outperform these alternatives on several large learning to rank datasets.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u201cRelative Confidence Sampling for Efficient On-Line Ranker Evaluation\u201d by Masrour Zoghi, Shimon Whiteson, Maarten de Rijke and Remi Munos is\u00a0available\u00a0online now. A key challenge in information retrieval is that of\u00a0on-line ranker evaluation: determining which one of a finite set of rankers performs the best in expectation on the basis of user clicks on presented document&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[],"_links":{"self":[{"href":"https:\/\/staff.fnwi.uva.nl\/m.derijke\/wp-json\/wp\/v2\/posts\/224"}],"collection":[{"href":"https:\/\/staff.fnwi.uva.nl\/m.derijke\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/staff.fnwi.uva.nl\/m.derijke\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/staff.fnwi.uva.nl\/m.derijke\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/staff.fnwi.uva.nl\/m.derijke\/wp-json\/wp\/v2\/comments?post=224"}],"version-history":[{"count":1,"href":"https:\/\/staff.fnwi.uva.nl\/m.derijke\/wp-json\/wp\/v2\/posts\/224\/revisions"}],"predecessor-version":[{"id":225,"href":"https:\/\/staff.fnwi.uva.nl\/m.derijke\/wp-json\/wp\/v2\/posts\/224\/revisions\/225"}],"wp:attachment":[{"href":"https:\/\/staff.fnwi.uva.nl\/m.derijke\/wp-json\/wp\/v2\/media?parent=224"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/staff.fnwi.uva.nl\/m.derijke\/wp-json\/wp\/v2\/categories?post=224"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/staff.fnwi.uva.nl\/m.derijke\/wp-json\/wp\/v2\/tags?post=224"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}