Deep Neural Networks and Tabular Data: A Survey

Abstract

Heterogeneous tabular data are the most commonly used form of data and are essential for numerous critical and computationally demanding applications. On homogeneous datasets, deep neural networks have repeatedly shown excellent performance and have therefore been widely adopted. However, their adaptation to tabular data for inference or data generation tasks remains highly challenging. To facilitate further progress in the field, this work provides an overview of state-of-the-art deep learning methods for tabular data. We categorize these methods into three groups: data transformations, specialized architectures, and regularization models. For each of these groups, our work offers a comprehensive overview of the main approaches. Moreover, we discuss deep learning approaches for generating tabular data and also provide an overview over strategies for explaining deep models on tabular data. Thus, our first contribution is to address the main research streams and existing methodologies in the mentioned areas while highlighting relevant challenges and open research questions. Our second contribution is to provide an empirical comparison of traditional machine learning methods with 11 deep learning approaches across five popular real-world tabular datasets of different sizes and with different learning objectives. Our results, which we have made publicly available as competitive benchmarks, indicate that algorithms based on gradient-boosted tree ensembles still mostly outperform deep learning models on supervised learning tasks, suggesting that the research progress on competitive deep learning models for tabular data is stagnating. To the best of our knowledge, this is the first in-depth overview of deep learning approaches for tabular data; as such, this work can serve as a valuable starting point to guide researchers and practitioners interested in deep learning with tabular data.

Machine learning for predicting neurodegenerative diseases in the general older population: a cohort study

“…This study found that the TabTransformer compared to other DNN (Densenet and FeedForward) and regularized Cox models showed a superior discriminative ability to predict NDs events in an older general population. Due to the attention-based layers, TabTransformer performs well with heterogeneous data, particularly in managing categorical input, which is not the case with other neural networks [18]. In time-dependent assessment, Tab-Transformer, compared to the other models, performed similarly in AUC and balanced accuracy, slightly worse in sensitivity and better in specificity at the 4th and 6th years of follow-up.…”

“…DNNs also can handle survival time if the DNN algorithm is tailored to censored data by with the appropriate censoring unbiased loss functions [15][16][17]. The disadvantages are that most DNNs do not perform appropriately with heterogeneous tabular data [18]. Researchers have recently developed algorithms with different structures that can deal with tabular-heterogeneous data to fill this gap [18] Still, these algorithms have not been widely investigated yet to time-to-event outcomes.…”

Discrimination Neural Network Model for Binary Classification Tasks on Tabular Data

“…Classification tasks on tabular data have been widely studied in the machine learning literature [23], [24]. One popular approach for such tasks is the use of decision tree-based algorithms, such as Random Forests [25] and XGBoost [26].…”

An in-Depth Analysis of the Software Features’ Impact on the Performance of Deep Learning-Based Software Defect Predictors

“…As long as there is correlation between the input information and the output, the models will discover it. In order to use deep learning, the input software features are written in tabular form, a data form that has been extensively researched and for which many models are available [9].…”