ur CIKM 2014 short paper “Vertical-aware click model-based effectiveness metrics” by Ilya Markov, Eugene Kharitonov, Vadim Nikulin, Pavel Serdyukov, Maarten de Rijke and Fabio Crestani is available now. You can find it online here.

Today’s web search systems present users with heterogeneous information coming from sources of different types, also known as verticals. Evaluating such systems is an important but complex task, which is still far from being solved. In this paper we examine the hypothesis that the use of models that capture user search behavior on heterogeneous result pages helps to improve the quality of offline metrics. We propose two vertical-aware metrics based on user click models for federated search and evaluate them using query logs of the Yandex search engine. We show that depending on the type of vertical, the proposed metrics have higher correlation with online user behavior than other state-of-the-art techniques.