Hierarchy-based semantic embeddings for single-valued & multi-valued categorical variables

Abstract

In low-resource domains, it is challenging to achieve good performance using existing machine learning methods due to a lack of training data and mixed data types (numeric and categorical). In particular, categorical variables with high cardinality pose a challenge to machine learning tasks such as classification and regression because training requires sufficiently many data points for the possible values of each variable. Since interpolation is not possible, nothing can be learned for values not seen in the training set. This paper presents a method that uses prior knowledge of the application domain to support machine learning in cases with insufficient data. We propose to address this challenge by using embeddings for categorical variables that are based on an explicit representation of domain knowledge (KR), namely a hierarchy of concepts. Our approach is to 1. define a semantic similarity measure between categories, based on the hierarchy—we propose a purely hierarchy-based measure, but other similarity measures from the literature can be used—and 2. use that similarity measure to define a modified one-hot encoding. We propose two embedding schemes for single-valued and multi-valued categorical data. We perform experiments on three different use cases. We first compare existing similarity approaches with our approach on a word pair similarity use case. This is followed by creating word embeddings using different similarity approaches. A comparison with existing methods such as Google, Word2Vec and GloVe embeddings on several benchmarks shows better performance on concept categorisation tasks when using knowledge-based embeddings. The third use case uses a medical dataset to compare the performance of semantic-based embeddings and standard binary encodings. Significant improvement in performance of the downstream classification tasks is achieved by using semantic information.

Time Is Ripe for Targeting Per- and Polyfluoroalkyl Substances-Induced Hormesis: Global Aquatic Hotspots and Implications for Ecological Risk Assessment

“…Categorical variables such as species and habitat were coded as numeric variables using the target encoding approach that could estimate the probability of variables without increasing the dimensionality. 33 The multicollinearity test was conducted using SPSS v.17.0 software (IBM Corporation, Armonk, New York, USA) to examine whether there was a correlation among explanatory variables. In addition, the robustness of variable importance ranking was tested using randomized 5-fold crossvalidation with 1000 replications.…”

Prediction of adverse biological effects of chemicals using knowledge graph embeddings

“…An in-depth analysis of this is an interesting direction for future research. This could be solved by embedding the hierarchy separately, e.g., [50], or imposing restrictions on the embeddings, such as a minimum distance constraint.…”

Prediction of Adverse Biological Effects of Chemicals Using Knowledge Graph Embeddings

“…An in-depth analysis of this is an interesting direction for future research. This could be solved by embedding the hierarchy separately, e.g., [49], or imposing restrictions on the embeddings, such as a minimum distance constraint.…”