Optimizing training trajectories in variational autoencoders via latent Bayesian optimization approach ^*

Abstract

Unsupervised and semi-supervised ML methods such as variational autoencoders (VAE) have become widely adopted across multiple areas of physics, chemistry, and materials sciences due to their capability in disentangling representations and ability to find latent manifolds for classification and/or regression of complex experimental data. Like other ML problems, VAEs require hyperparameter tuning, e.g., balancing the Kullback–Leibler (KL) and reconstruction terms. However, the training process and resulting manifold topology and connectivity depend not only on hyperparameters, but also their evolution during training. Because of the inefficiency of exhaustive search in a high-dimensional hyperparameter space for the expensive-to-train models, here we have explored a latent Bayesian optimization (zBO) approach for the hyperparameter trajectory optimization for the unsupervised and semi-supervised ML and demonstrated for joint-VAE with rotational invariances. We have demonstrated an application of this method for finding joint discrete and continuous rotationally invariant representations for MNIST and experimental data of a plasmonic nanoparticles material system. The performance of the proposed approach has been discussed extensively, where it allows for any high dimensional hyperparameter trajectory optimization of other ML models.

A dynamic Bayesian optimized active recommender system for curiosity-driven partially “Human-in-the-loop” automated experiments

“…al. 47 attempted the approach to project into a low-dimensional latent space with variational autoencoder; and Oh et. al.…”

Combining variational autoencoders and physical bias for improved microscopy data analysis ∗

“…However, tuning this high-dimensional iterative dependent hyper-parameter would be a very time-consuming task (given the sh-VAE model is computationally very expensive) which need to be done prior to the training. However, one approach we took earlier in tuning such trajectory hyperparameter is through latent Bayesian optimization approach [35] but would still take a significant amount of time to conduct such optimization task. Here, bypassing the need and the mentioned challenges of prior tuning of particularly high-dimensional hyperparameters (e.g.…”

“…The development of the rotationally invariant autoencoders extended these approaches to general orientation and discovery of chemical transformation pathways in disordered systems [31][32][33][34]. Also, such rotationally invariant autoencoders have been tuned with Bayesian optimization to maximize uncovering of the features from complex microscopic data [35]. However, for the time being these methods relied on the descriptors centered at the individual atomic units, whereas unsupervised discovery allowing for translation invariance has remained far more complex.…”

Machine Learning-Based Sales Prediction Using Bayesian Optimized XGBoost Algorithms

“…Buckell et al incorporated the displayed preference data into the model to calibrate the market forecast [13]. Furthermore, numerous researchers have employed Bayesian optimization techniques to optimize hyperparameters [14][15][16].…”