Improving Automatic Query Classification via Semi-Supervised Learning

Abstract

Accurate topical classification of user queries allows for increased effectiveness and efficiency in general-purpose web search systems.Such classification becomes critical if the system is to return results not just from a general web collection but from topic-specific back-end databases as well. Maintaining sufficient classification recall is very difficult as web queries are typically short, yielding few features per query. This feature sparseness coupled with the high query volumes typical for a large-scale search service makes manual and supervised learning approaches alone insufficient. We use an application of computational linguistics to develop an approach for mining the vast amount of unlabeled data in web query logs to improve automatic topical web query classification. We show that our approach in combination with manual matching and supervised learning allows us to classify a substantially larger proportion of queries than any single technique. We examine the performance of each approach on a real web query stream and show that our combined method accurately classifies 46% of queries, outperforming the recall of best single approach by nearly 20%, with a 7% improvement in overall effectiveness.

Sources of evidence for vertical selection

“…Therefore, some prior work in query-classification into topical categories is relevant to vertical selection [13,14,2,1,10]. Because queries are terse, many query-classification approaches augment the query with features beyond the query string, possibly derived from query-logs or corpora of documents associated with the target classes.…”

“…Because queries are terse, many query-classification approaches augment the query with features beyond the query string, possibly derived from query-logs or corpora of documents associated with the target classes. Bietzel et al use a large (unlabeled) query-log and a technique known as selectional preferencethe query "interest rates" belongs to target category finance because "interest" and "rates" are distributionally similar to the term "finance" [1,2]. Shen et al [13] and other participants of the KDD 2005 Cup [11] use corpus-based evidence.…”

“…We evaluate corpus features on two types of collections: collections of verticalsampled documents and surrogate collections representative of verticals constructed by sampling a non-vertical resource, the Wikipedia. 1 We evaluate several simple baselines, each focused on a single source of evidence and a supervised approach that combines our three feature types. An error analysis shows the contribution of each feature type and reveals unique challenges in vertical selection.…”

Classification-based Retrieval Methods to Enhance Information Discovery on the Web

“…Finally, the queries were classified into the subject taxonomy using the classifiers through a consensus function. The main limitations of the proposed method are the dependency of the classification to the quality of the training data, the human effort involved in the training data construction and the semi-automatic nature of the approach which limits the scale of the method applications [8] and [9] and [15].…”

Acquiring knowledge about human goals from Search Query Logs

“…user-click behavior or part-ofspeech information, to perform the categorization task. (Beitzel et al, 2005) proposed a method for automatic query classification by leveraging unlabeled data within a semi-supervised learning framework. Their semi-supervised approach facilitated the augmentation of labeled training samples for the classification task.…”