A Fully Hyperbolic Neural Model for Hierarchical Multi-Class Classification

Abstract

Label inventories for fine-grained entity typing have grown in size and complexity. Nonetheless, they exhibit a hierarchical structure. Hyperbolic spaces offer a mathematically appealing approach for learning hierarchical representations of symbolic data. However, it is not clear how to integrate hyperbolic components into downstream tasks. This is the first work that proposes a fully hyperbolic model for multi-class multi-label classification, which performs all operations in hyperbolic space.We evaluate the proposed model on two challenging datasets and compare to different baselines that operate under Euclidean assumptions. Our hyperbolic model infers the latent hierarchy from the class distribution, captures implicit hyponymic relations in the inventory, and shows performance on par with state-ofthe-art methods on fine-grained classification with remarkable reduction of the parameter size. A thorough analysis sheds light on the impact of each component in the final prediction and showcases its ease of integration with Euclidean layers. 1

Modeling Fine-Grained Entity Types with Box Embeddings

“…We also list the numbers from prior work in Table 2. HY XLarge (López and Strube, 2020), a hyperbolic model designed to learn hierarchical structure in entity types, exceeds the performance of the models with similar sizes such as Choi et al (2018) and Xiong et al (2019) especially in macrorecall. In the ultra-fine class, both our box-based model and HY XLarge achieve higher macro-F1 compared to their vector-based counterparts.…”

“…However, OE can only handle binary entailment decisions, and POE cannot model negative correlations between types, a critical limitation in its use as a probabilistic model; these shortcomings directly led to the development of box embeddings. Hyperbolic embeddings (Nickel and Kiela, 2017; López and Strube, 2020) can also model hierarchical relationships as can hyperbolic entailment cones (Ganea et al, 2018); however, these approaches lack a probabilistic interpretation.…”

“…These types follow some kind of hierarchical structure (Weischedel and Brunstein, 2005;Ling and Weld, 2012;Gillick et al, 2014;, so effective models for these tasks frequently engage with this hierarchy explicitly. Prior systems incorporate this structure via hierarchical losses Xu and Barbosa, 2018;Chen et al, 2020) or by embedding types into a high-dimensional Euclidean or hyperbolic space (Yogatama et al, 2015;López and Strube, 2020). However, the former approach requires prior knowledge of the type hierarchy, which is unsuitable for a recent class of large type sets where the hierarchy is not explicit (Choi et al, 2018;Onoe and Durrett, 2020a).…”

“…We omit the test number ofLópez and Strube (2020), since they report results broken down into coarse, fine, and ultra-fine types instead of an aggregated F1 value. However, based on the development results, their approach substantially underperforms the past work ofOnoe and Durrett (2019) regardless.…”

Fine-Grained Entity Typing via Label Noise Reduction and Data Augmentation

“…where λ is the relative weight of these two losses 3 . Baselines For Ultra-Fine dataset, we compare with following baselines: Onoe and Durrett ( 2019) which offers two multi-classifiers using BERT and ELMo as encoder respectively, Choi et al (2018) which is a multi-classifier using GloVe+LSTM as encoder, Xiong et al (2019) which is a multiclassifier using GloVe+LSTM as encoder and exploits label co-occurrence via introducing associated labels to enrich the label representation, López and Strube (2020) which is a hyperbolic multiclassifier using GloVe. For OntoNotes dataset, in addition to the baselines for Ultra-Fine, we also compare with which offers a multi-classifier using BERT as encoder, Lin and Ji (2019) which offers a multi-classifier using ELMo as encoder and exploits label co-occurrence via requiring the latent representation to reconstruct the co-occurrence association and which offers a multi-classifier using ELMo as encoder and exploits label hierarchy via designing a hierarchy-aware loss function.…”