A Survey of Inductive Biases for Factorial Representation-Learning

Abstract

With the resurgence of interest in neural networks, representation learning has re-emerged as a central focus in artificial intelligence. Representation learning refers to the discovery of useful encodings of data that make domain-relevant information explicit. Factorial representations identify underlying independent causal factors of variation in data. A factorial representation is compact and faithful, makes the causal factors explicit, and facilitates human interpretation of data. Factorial representations support a variety of applications, including the generation of novel examples, indexing and search, novelty detection, and transfer learning.This article surveys various constraints that encourage a learning algorithm to discover factorial representations. I dichotomize the constraints in terms of unsupervised and supervised inductive bias. Unsupervised inductive biases exploit assumptions about the environment, such as the statistical distribution of factor coefficients, assumptions about the perturbations a factor should be invariant to (e.g. a representation of an object can be invariant to rotation, translation or scaling), and assumptions about how factors are combined to synthesize an observation. Supervised inductive biases are constraints on the representations based on additional information connected to observations. Supervisory labels come in variety of types, which vary in how strongly they constrain the representation, how many factors are labeled, how many observations are labeled, and whether or not we know the associations between the constraints and the factors they are related to.This survey brings together a wide variety of models that all touch on the problem of learning factorial representations and lays out a framework for comparing these models based on the strengths of the underlying supervised and unsupervised inductive biases.

Measuring Disentanglement: A Review of Metrics

“…Before analyzing metrics, we discuss what constitutes a disentangled representation. While there is no unanimously accepted definition of disentanglement, most agree on two main aspects [9,14,18,26,30,31]: First, the representation has to be distributed. This means that an input is a composition of explanatory factors and corresponds to a single point in the representation space.…”

“…Factors must be conceptually independent, but should also be statistically independent [2]. This condition is hard to satisfy in real-world data sets where certain factor realizations tend to co-occur more than others [14]. For example, in a data set of fruit images, we could be interested in two conceptually different factors: fruit type and color.…”

On Disentanglement in Gaussian Process Variational Autoencoders

“…This motivates the learning of task-agnostic representations that fare well on a wide variety of problems. Recent research suggests that disentangled representations could be suitable in this sense [Bengio et al, 2013, Ridgeway, 2016, Tschannen et al, 2018.…”

Invariance-based Multi-Clustering of Latent Space Embeddings for Equivariant Learning

“…We are interested in fleshing out a canonical reconstruction that is invariant under such group transformations while indepen-However, vanilla VAE models easily confound invariant representation with non-equivariant variational perturbations. Various papers have called attention to the inability of generative models like GAN (Goodfellow et al, 2014) and VAE (Kingma and Welling, 2014) in faithful factorization of latent representations (Ridgeway, 2016). Notably, the unsupervised InfoGAN (Chen et al, 2016), β-VAE (Higgins et al, 2017), β-TCVAE (Chen et al, 2018), factor-VAE (Kim and Mnih, 2018), and the semi-supervised DC-IGN (Kulkarni et al, 2015) are some examples of the necessary effort needed in learning factorized representations with modified generative models.…”