Subgroup Discovery in Unstructured Data

Abstract

Subgroup discovery is a descriptive and exploratory data mining technique to identify subgroups in a population that exhibit interesting behavior with respect to a variable of interest. Subgroup discovery has numerous applications in knowledge discovery and hypothesis generation, yet it remains inapplicable for unstructured, high-dimensional data such as images. This is because subgroup discovery algorithms rely on defining descriptive rules based on (attribute, value) pairs, however, in unstructured data, an attribute is not well defined. Even in cases where the notion of attribute intuitively exists in the data, such as a pixel in an image, due to the high dimensionality of the data, these attributes are not informative enough to be used in a rule. In this paper, we introduce the subgroup-aware variational autoencoder, a novel variational autoencoder that learns a representation of unstructured data which leads to subgroups with higher quality. Our experimental results demonstrate the effectiveness of the method at learning subgroups with high quality while supporting the interpretability of the concepts.Keywords Subgroup Discovery • Unstructured Data • Pattern Mining Despite numerous works in the literature, all subgroup discovery techniques suffer from a limitation -they are only applicable to structured data, i.e., data with well-defined attributes. A typical instance of this would be data found relational databases. However, in many settings, the data to be dealt with is unstructured and potentially high dimensional such as images, videos, text documents, and time series. Subgroups are defined through a conjunction of attribute and value pairs. For high dimensional data, such as an image, this implies that the subgroups will be defined based on the

Inference on tree‐structured subgroups with subgroup size and subgroup effect relationship in clinical trials

“…The problems become more acute as the more patients a subgroup includes, the lower average treatment effect a subgroup tends to have. 16 Therefore, without taking subgroup size into account, existing methods that pursue the enhanced treatment effect often favor candidate subgroups with small subgroup size. Recently, several methods 15,17 have been proposed to account for subgroup size in the selection of promising subgroups, but they are not designed for inference on subgroups.…”