On Minimal Variations for Unsupervised Representation Learning

Abstract

Learning arguably involves the discovery and memorization of abstract rules. The aim of this paper is to study associative memory mechanisms. Our model is based on high-dimensional matrices consisting of outer products of embeddings, which relates to the inner layers of transformer language models. We derive precise scaling laws with respect to sample size and parameter size, and discuss the statistical efficiency of different estimators, including optimization-based algorithms. We provide extensive numerical experiments to validate and interpret theoretical results, including fine-grained visualizations of the stored memory associations.

Review of synergy between machine learning and first principles models for asset integrity management

“…Unsupervised learning algorithms aim to find hidden patterns or clusters in the data, dimensionality reduction, or discover the underlying probability distribution of the data. Common techniques used in unsupervised learning include clustering algorithms (e.g., k-means, hierarchical clustering), dimensionality reduction (e.g., principal component analysis), and generative models (e.g., Gaussian mixture models) (Dike, Zhou, Deveerasetty, & Wu, 2018;Cabannes, Bietti, & Balestriero, 2023).…”