Artificial intelligence & Information retrieval

Month: September 2014

CIKM 2014 paper on time-sensitive personalized query auto-completion online

Our CIKM 2014 paper “Time-Sensitive Personalized Query Auto-Completion” by Fei Cai, Shangsong Liang and Maarten de Rijke is online now.

Query auto-completion (QAC) is a prominent feature of modern search engines. It is aimed at saving user’s time and enhancing the search experience. Current QAC models mostly rank matching QAC candidates according to their past popularity, i.e., frequency. However, query popularity changes over time and may vary drastically across users. Hence, rankings of QAC candidates should be adjusted accordingly. In previous work time-sensitive QAC models and user-specific QAC models have been developed separately. Both types of QAC model lead to important improvements over models that are neither time-sensitive nor personalized. We propose a hybrid QAC model that considers both of these aspects: time-sensitivity and personalization.

Using search logs, we return the top N QAC candidates by predicted popularity based on their recent trend and cyclic behavior. We use auto-correlation to detect query periodicity by long-term time-series analysis, and anticipate the query popularity trend based on observations within an optimal time window returned by a regression model. We rerank the returned top N candidates by integrating their similarities with a user’s preceding queries (both in the current session and in previous sessions by the same user) on a character level to produce a final QAC list. Our experimental results on two real-world datasets show that our hybrid QAC model outperforms state-of-the-art time-sensitive QAC baseline, achieving total improvements of between 3% and 7% in terms of MRR.

One week to go: AMS Data #1: Coffee and Data Science

A week from today, we’ll see the launch of a new Amsterdam Data Science event series: AMS Data: Coffee and Data Science.


What is a better way to start your morning than with data and coffee? Amsterdam Data Science is organizing an event on September 25 (9:00 – 12:00pm) to bring the Amsterdam area data science community together. Go ahead and visit the Amsterdam Data Science site for details.

CIKM 2014 paper on vertical aware click model-based effectiveness metrics available online

ur CIKM 2014 short paper “Vertical-aware click model-based effectiveness metrics” by Ilya Markov, Eugene Kharitonov, Vadim Nikulin, Pavel Serdyukov, Maarten de Rijke and Fabio Crestani is available now. You can find it online here.

Today’s web search systems present users with heterogeneous information coming from sources of different types, also known as verticals. Evaluating such systems is an important but complex task, which is still far from being solved. In this paper we examine the hypothesis that the use of models that capture user search behavior on heterogeneous result pages helps to improve the quality of offline metrics. We propose two vertical-aware metrics based on user click models for federated search and evaluate them using query logs of the Yandex search engine. We show that depending on the type of vertical, the proposed metrics have higher correlation with online user behavior than other state-of-the-art techniques.

CIKM 2014 paper on multileaved comparisons for fast online evaluation online

Our CIKM 2014 paper “Multileaved comparisons for fast online evaluation” by Anne Schuth, Floor Sietsma, Shimon Whiteson, Damien Lefortier and Maarten de Rijke is available online now.

Evaluation methods for information retrieval systems come in three types: offline evaluation, using static data sets annotated for relevance by human judges; user studies, usually conducted in a lab-based setting; and online evaluation, using implicit signals such as clicks from actual users. For the latter, preferences between rankers are typically inferred from implicit signals via interleaved comparison methods, which combine a pair of rankings and display the result to the user. We propose a new approach to online evaluation called multileaved comparisons that is useful in the prevalent case where designers are interested in the relative performance of more than two rankers. Rather than combining only a pair of rankings, multileaved comparisons combine an arbitrary number of rankings. The resulting user clicks then give feedback about how all these rankings compare to each other. We propose two specific multileaved comparison methods. The first, called team draft multileave, is an extension of team draft interleave. The second, called optimized multileave, is an extension of optimized interleave and is designed to handle cases where a large number of rankers must be multileaved. We present experimental results that demonstrate that both team draft multileave and optimized multileave can accurately determine all pairwise preferences among a set of rankers using far less data than the interleaving methods that they extend.

© 2022 Maarten de Rijke

Theme by Anders NorenUp ↑