A hierarchical Dirichlet model for taxonomy expansion for search engines

Abstract

Emerging trends and products pose a challenge to modern search engines since they must adapt to the constantly changing needs and interests of users. For example, vertical search engines, such as Amazon, eBay, Walmart, Yelp and Yahoo! Local, provide business category hierarchies for people to navigate through millions of business listings. The category information also provides important ranking features that can be used to improve search experience. However, category hierarchies are often manually crafted by some human experts and they are far from complete. Manually constructed category hierarchies cannot handle the everchanging and sometimes long-tail user information needs. In this paper, we study the problem of how to expand an existing category hierarchy for a search/navigation system to accommodate the information needs of users more comprehensively. We propose a general framework for this task, which has three steps: 1) detecting meaningful missing categories; 2) modeling the category hierarchy using a hierarchical Dirichlet model and predicting the optimal tree structure according to the model; 3) reorganizing the corpus using the complete category structure, i.e., associating each webpage with the relevant categories from the complete category hierarchy. Experimental results demonstrate that our proposed framework generates a high-quality category hierarchy and significantly boosts the retrieval performance.

AutoKnow

“…Edge-F1 Ancestor-F1 Substr [5] 10.7 52.9 HiDir [33] 40.5 66.4 MSejrKu [29] 53 sequence of (y 1 , y 2 , ..., y L ), where y i ∈ {B, I, O, E}, representing "begin", "inside", "outside", "end" respectively. Table 2 illustrates an example of sequential labels obtained using OpenTag [37]: "ice cream" is labeled as product type, and "black cherry cheesecake" as product flavor.…”

TaxoExpan: Self-supervised Taxonomy Expansion with Position-Enhanced Graph Neural Network

“…Discussion. In this work, we follow previous studies [2,22,48] and assume each concept in N 0 ∪ C has an initial embedding vector learned from this concept's surface name, or if available, its definition sentences [39] and associated web pages [51]. We also note that our problem formulation assumes those relations in the existing taxonomy are not modified.…”

“…In this study, we assume each query concept has an initial feature vector learned based on some text associated with this concept. Such text can be as simple as the concept surface name, or in some prior studies [22,51], the definition sentences and clicked web pages about the concept. We represent each query concept n i using its initial feature vector denoted as n i .…”

Octet

“…Non-Goals. Although Octet works for terms regardless of their granularity, we keep T unchanged as in [11,33] since we would like to keep the high-level expert-curated hypernym pairs intact and focus on discovering fine-grained terms. Following convention [11], we do not identify the hypernym relationship between newly discovered types (…”

“…In particular, online catalog taxonomies serve Figure 1: The most relevant taxonomy nodes are shown on the left when a user searches "k cups" on Amazon.com. as a building block of e-commerce websites (e.g., Amazon.com) and business directory services (e.g., Yelp.com) for both customer-facing and internal applications, such as query understanding, item categorization [18], browsing, recommendation [9], and search [33]. Fig.…”