Tabular Transformers for Modeling Multivariate Time Series

Abstract

Tabular datasets are ubiquitous in data science applications. Given their importance, it seems natural to apply state-ofthe-art deep learning algorithms in order to fully unlock their potential. Here we propose neural network models that represent tabular time series that can optionally leverage their hierarchical structure. This results in two architectures for tabular time series: one for learning representations that is analogous to BERT and can be pre-trained end-to-end and used in downstream tasks, and one that is akin to GPT and can be used for generation of realistic synthetic tabular sequences. We demonstrate our models on two datasets: a synthetic credit card transaction dataset, where the learned representations are used for fraud detection and synthetic data generation, and on a real pollution dataset, where the learned encodings are used to predict atmospheric pollutant concentrations. Code and data are available at https://github. com/IBM/TabFormer.

SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training

“…TABERT [48], a more elaborate neural approach inspired by the large language transformer model BERT [9], is trained on semi-structured test data to perform language-specific tasks. Several other studies utilize tabular data, but their problem settings are outside of our scope [3,21,31,32,35].…”

The GatedTabTransformer. An enhanced deep learning architecture for tabular modeling

“…Additional embedding techniques and manipulated representations of the data (e.g. TaBERT [26], TabFormer [27]) can be incorporated as pre-processing steps.…”

NLP Based Anomaly Detection for Categorical Time Series

“…These methods can often be made to accommodate categorical data through one-hot encoding, but in the authors' experience quality of models in this family rapidly degrades as the fraction of the series' variables that are categorical increases. The field in which anomaly detection in categorical time series is most developed in is intrusion detection in network security and fraud detection [4,9]. The authors in [9] utilize a transformer architecture for fraud detection inspired by an analogy between finite sequences of discrete variables and words in the domain of Natural Language Processing (NLP).…”

“…The field in which anomaly detection in categorical time series is most developed in is intrusion detection in network security and fraud detection [4,9]. The authors in [9] utilize a transformer architecture for fraud detection inspired by an analogy between finite sequences of discrete variables and words in the domain of Natural Language Processing (NLP).…”