AutoEval: An Evaluation Methodology for Evaluating Query Suggestions Using Query Logs

Advanced Language Technologies for Digital Libraries

Niu

et al. 2011

Self Cite

Abstract. Search applications have become very popular over the last two decades, one of the main drivers being the advent of the Web. Nevertheless, searching on the Web is very different to searching on smaller, often more structured collections such as digital libraries, local Web sites, and intranets. One way of helping the searcher locating the right information for a specific information need in such a collection is by providing well-structured domain knowledge to assist query modification and navigation. There are two main challenges which we will both address in this chapter: acquiring the domain knowledge and adapting it automatically to the specific interests of the user community. We will outline how in digital libraries a domain model can automatically be acquired using search engine query logs and how it can be continuously updated using methods resembling ant colony behaviour.

Section: Automatic Evaluation Method: Autoevalmentioning

confidence: 99%

Section: The Domain Modelmentioning

confidence: 99%

Moving towards Adaptive Search in Digital Libraries

Kruschwitz

Advanced Language Technologies for Digital Libraries

Niu

et al. 2011

Self Cite

“…The validity of the framework has been confirmed with a user study [2]. The evaluation is performed on arbitrary intervals, e.g.…”

Section: The Evaluation Frameworkmentioning

confidence: 77%

Enriching Query Flow Graphs with Click Information

Information Retrieval Technology

Kruschwitz

Adeyanju

et al. 2011

Self Cite

Abstract. The increased availability of large amounts of data about user search behaviour in search engines has triggered a lot of research in recent years. This includes developing machine learning methods to build knowledge structures that could be exploited for a number of tasks such as query recommendation. Query flow graphs are a successful example of these structures, they are generated from the sequence of queries typed in by a user in a search session. In this paper we propose to modify the query flow graph by incorporating clickthrough information from the search logs. Click information provides evidence of the success or failure of the search journey and therefore can be used to enrich the query flow graph to make it more accurate and useful for query recommendation. We propose a method of adjusting the weights on the edges of the query flow graph by incorporating the number of clicked documents after submitting a query.We explore a number of weighting functions for the graph edges using click information. Applying an automated evaluation framework to assess query recommendations allows us to perform automatic and reproducible evaluation experiments. We demonstrate how our modified query flow graph outperforms the standard query flow graph. The experiments are conducted on the search logs of an academic organisation's search engine and validated in a second experiment on the log files of another Web site.

“…We use an automated evaluation framework, AutoEval, which measures the effectiveness of query recommendations over a period of time based on actual query logs [2]. The evaluation framework was validated with a user study which showed that scores from the automated evaluation correlated highly with user evaluation of the recommendations.…”

Section: Evaluation Methodologymentioning

confidence: 99%

Adaptation of the concept hierarchy model with search logs for query recommendation on intranets

Adeyanju

Song

Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval

et al. 2012

Self Cite

A concept hierarchy created from a document collection can be used for query recommendation on Intranets by ranking terms according to the strength of their links to the query within the hierarchy. A major limitation is that this model produces the same recommendations for identical queries and rebuilding it from scratch periodically can be extremely inefficient due to the high computational costs. We propose to adapt the model by incorporating query refinements from search logs. Our intuition is that the concept hierarchy built from the collection and the search logs provide complementary conceptual views on the same search domain, and their integration should continually improve the effectiveness of recommended terms. Two adaptation approaches using query logs with and without click information are compared. We evaluate the concept hierarchy models (static and adapted versions) built from the Intranet collections of two academic institutions and compare them with a state-of-theart log-based query recommender, the Query Flow Graph, built from the same logs. Our adaptive model significantly outperforms its static version and the query flow graph when tested over a period of time on data (documents and search logs) from two institutions' Intranets.