SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training

Somepalli, Gowthami; Goldblum, Micah; Schwarzschild, Avi; Bruss, C. Bayan; Goldstein, Tom

doi:10.48550/arxiv.2106.01342

Cited by 46 publications

(79 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…4.2.1) and transformers-based (Sec. 4.2.2) groups exhibit superior predictive performance compared to plain deep neural networks on various data sets (Gorishniy et al, 2021;Ke et al, 2018Ke et al, , 2019Somepalli et al, 2021). This underlines the importance of special-purpose architectures for tabular data.…”

Section: Summary and Trendsmentioning

confidence: 96%

“…For heterogeneous tabular data, these techniques are often difficult to apply. However, some frameworks for learning with tabular data, such as VIME (Yoon et al, 2020) and SAINT (Somepalli et al, 2021), use data augmentation strategies in the embedding space.…”

Section: Missing or Complex Irregular Spatial Dependenciesmentioning

confidence: 99%

“…SAINT (Self-Attention and Intersample Attention Transformer) (Somepalli et al, 2021) is a hybrid attention approach, combining self-attention (Vaswani et al, 2017) with inter-sample attention over multiple rows. When handling missing or noisy data, this mechanism allows the model to borrow the corresponding information from similar samples, which improves the model's robustness.…”

Section: Transformer-based Modelsmentioning

confidence: 99%

“…One of the first methods in this category was the Regularization Learning Network (RLN) proposed by Shavitt and Segal (2018), which uses a learned regularization scheme. The main idea is to apply trainable regularization coefficients to each single weight in a neural network and, by this, lower the high sensitivity: (Somepalli et al, 2021).…”

Section: Regularization Modelsmentioning

confidence: 99%

“…Variables can be correlated or independent, and the features have no positional information. Hence, it is necessary to discover and exploit correlation without relying on spatial information (Somepalli et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Deep Neural Networks and Tabular Data: A Survey

Borisov¹,

Leemann²,

Seßler³

et al. 2021

Preprint

View full text Add to dashboard Cite

Heterogeneous tabular data are the most commonly used form of data and are essential for numerous critical and computationally demanding applications. On homogeneous data sets, deep neural networks have repeatedly shown excellent performance and have therefore been widely adopted. However, their application to modeling tabular data (inference or generation) remains highly challenging. This work provides an overview of state-of-the-art deep learning methods for tabular data. We start by categorizing them into three groups: data transformations, specialized architectures, and regularization models. We then provide a comprehensive overview of the main approaches in each group. A discussion of deep learning approaches for generating tabular data is complemented by strategies for explaining deep models on tabular data. Our primary contribution is to address the main research streams and existing methodologies in this area, while highlighting relevant challenges and open research questions. To the best of our knowledge, this is the first in-depth look at deep learning approaches for tabular data. This work can serve as a valuable starting point and guide for researchers and practitioners interested in deep learning with tabular data.

show abstract

Section: Summary and Trendsmentioning

confidence: 96%

Section: Missing or Complex Irregular Spatial Dependenciesmentioning

confidence: 99%

Section: Transformer-based Modelsmentioning

confidence: 99%

Section: Regularization Modelsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Deep Neural Networks and Tabular Data: A Survey

Borisov¹,

Leemann²,

Seßler³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

Convolutional Point Transformer

Kaul¹,

Mitton²,

Dai³

et al. 2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

We present CpT: Convolutional point Transformer -a novel neural network layer for dealing with the unstructured nature of 3D point cloud data. CpT is an improvement over existing MLP and convolution layers for point cloud processing, as well as existing 3D point cloud processing transformer layers. It achieves this feat due to its effectiveness in creating a novel and robust attention-based point set embedding through a convolutional projection layer crafted for processing dynamically local point set neighbourhoods. The resultant point set embedding is robust to the permutations of the input points. Our novel layer builds over local neighbourhoods of points obtained via a dynamic graph computation at each layer of the network's structure. It is fully differentiable and can be stacked just like convolutional layers to learn intrinsic properties of the points. Further, we propose a novel Adaptive Global Feature layer that learns to aggregate features from different representations into a better global representation of the point cloud. We evaluate our models on standard benchmark ModelNet40 classification and ShapeNet part segmentation datasets to show that our layer can serve as an effective addition for various point cloud processing tasks while effortlessly integrating into existing point cloud processing architectures to provide significant performance boosts.

show abstract