REaLTabFormer: Generating Realistic Relational and Tabular Data using Transformers

Solatorio, Aivin; Dupriez, Olivier

doi:10.48550/arxiv.2302.02041

Cited by 6 publications

(6 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, accurately learning and reproducing nonstandard distribution patterns is di cult and may yield generated samples that cannot appropriately represent the complexities inherent in the original data. Recent advancements in deep learning, particularly those centered on Transformer architectures, have demonstrated promising applications in handling tabular datasets [22]- [24]. A notable development involves the implementation of a transformer-based GAN for the generation of synthetic data in the text and sequence areas [25], [26].…”

Section: Discussionmentioning

confidence: 99%

Tabular Transformer Generative Adversarial Network for Heterogeneous distribution in healthcare

Kang,

Ko,

Ryu

2024

Preprint

View full text Add to dashboard Cite

In healthcare, the most common type of data is tabular data, which hold high significance and potential in the field of medical AI. However, privacy concerns have hindered their widespread use. Despite the emergence of synthetic data as a viable solution, the generation of healthcare tabular data (HTD) is complex owing to the extensive interdependencies between the variables within each record that incorporate diverse clinical characteristics, including sensitive information. To overcome these issues, this study proposed a tabular transformer generative adversarial network (TT-GAN) to generate synthetic data that can effectively consider the relationships between variables potentially present in the HTD dataset. Transformers can consider the relationships between the columns in each record using a multi-attention mechanism. In addition, to address the potential risk of restoring sensitive data in patient information, a Transformer was employed in a generative adversarial network (GAN) architecture, to ensure an implicit-based algorithm. To consider the heterogeneous characteristics of the continuous variables in the HTD dataset, the discretization and converter methodology were applied. The experimental results confirmed the superior performance of the TT-GAN than the Conditional Tabular GAN (CTGAN) and copula GAN. Discretization and converters were proven to be effective using our proposed Transformer algorithm. However, the application of the same methodology to Transformer-based models without discretization and converters exhibited a significantly inferior performance. The CTGAN and copula GAN indicated minimal effectiveness with discretization and converter methodologies. Thus, the TT-GAN exhibited considerable potential in healthcare, demonstrating its ability to generate artificial data that closely resembled real healthcare datasets. The ability of the algorithm to handle different types of mixed variables efficiently, including polynomial, discrete, and continuous variables, demonstrated its versatility and practicality in health care research and data synthesis.

show abstract

Section: Discussionmentioning

confidence: 99%

Tabular Transformer Generative Adversarial Network for Heterogeneous distribution in healthcare

Kang,

Ko,

Ryu

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…More recent works use deep learning models to generate synthetic tabular data. Solatorio et al [32] proposed RE-alTabFormer, a transformer-based framework for data generation. REalTabFormer uses an autoregressive model based on the GPT-2 [51] to generate non-relational tabular data, as shown in the proposed architecture in Figure 3.…”

Section: Figure 1: General Data Augmentation Workflow For Imbalanced ...mentioning

confidence: 99%

“…Concerning REalTabFormer, we established the training parameters and set the batch size to 64 and 10 epochs. However, the training was interrupted before this number of epochs due to the Early Stopping with Q ′ δ condition, as described in the [32]. In the case of GReaT, we chose to train the model for a single epoch driven by computational constraints.…”

Section: A Experimental Environmentmentioning

confidence: 99%

“…More recently, some works have proposed the use of generative deep learning models based on Generative Adversarial Networks (GANs) [29], Autoencoders [30], and Transformer networks [31] to learn data representations and subsequently generate data conditioned on certain less represented classes. A recent work [32] proposed the Realistic Relational and Tabular Data using Transformers (REalTabFormer), which employs an autoregressive model to generate non-relational tabular data. In addition to REalTabFormer, the Generation of Realistic Tabular data (GReaT) [33] also leverages a GPTbased model to generate realistic tabular data.…”

mentioning

confidence: 99%

“…In SMOTE-NC, continuous attributes are over-sampled by interpolating between two samples the same way in SMOTE and categorical attributes are over-sampled by selecting one of the k-nearest neighbours for each example in the feature space.More recent works use deep learning models to generate synthetic tabular data. Solatorio et al[32] proposed RE-alTabFormer, a transformer-based framework for data generation. REalTabFormer uses an autoregressive model based on the GPT-2[51] to generate non-relational tabular data, as shown in the proposed architecture in Figure3.…”

mentioning

confidence: 99%

See 2 more Smart Citations

GPT and Interpolation-Based Data Augmentation for Multiclass Intrusion Detection in IIoT

Melícias,

Ribeiro,

Rabadão

et al. 2024

IEEE Access

View full text Add to dashboard Cite

The absence of essential security protocols in Industrial Internet of Things (IIoT) networks introduces cybersecurity vulnerabilities and turns them into potential targets for various attack types. Although machine learning has been used for intrusion detection in the IIoT, datasets with representative data of common attacks of IIoT network traffic are limited and often imbalanced. Data augmentation techniques address these problems by creating artificial data in classes with fewer samples. In this work, we evaluate the use of data augmentation when training intrusion detection models based on IIoT traffic data. We compare Generative Pre-trained Transformers (GPT) and variations on the Synthetic Minority Over-sampling TEchnique (SMOTE) and evaluate their capability to enhance intrusion detection performance. We examine the performance of five intrusion detection algorithms when trained with augmented datasets to models trained with the original non-augmented dataset. To ensure a fair comparison, we evaluated the algorithms' performance in the different scenarios using the same test dataset, which does not contain synthetic data. The results show the need for a systematic evaluation before employing data augmentation, as its impact on classification performance depends on the algorithm, data, and used technique. While deep neural networks benefit from data augmentation, the eXtreme Gradient Boosting (XGBoost), which achieved superior performance in intrusion detection between all evaluated classifiers (with F1-Score over 91%), didn't have any performance improvement when trained with augmented data. The evaluation of data generated by GPT-based methods shows such methods (especially GReaT) generate invalid data for both numerical and categorical features in a way that leads to performance degradation in multiclass classification.

show abstract

Künstliche Intelligenz und sichere Gesundheitsdatennutzung im Projekt KI-FDZ: Anonymisierung, Synthetisierung und sichere Verarbeitung für Real-World-Daten

Prasser,

Riedel,

Wolter

et al. 2024

Bundesgesundheitsbl

View full text Add to dashboard Cite

ZusammenfassungDie zunehmende Digitalisierung des Gesundheitswesens ist verbunden mit einem stetig wachsenden Datenvolumen, das durch Sekundärnutzung wertvolle Erkenntnisse über Diagnostik, Behandlungsprozesse und die Versorgungsqualität liefern kann. Das Forschungsdatenzentrum Gesundheit (FDZ) soll hierfür eine Infrastruktur bereitstellen. Dabei sind sowohl der Schutz der Privatsphäre der Patientinnen und Patienten als auch optimale Auswertungsmöglichkeiten von zentraler Bedeutung. Künstliche Intelligenz (KI) bietet hierfür ein doppeltes Potenzial. Zum einen ermöglichen Methoden des Machine Learning die Verarbeitung großer Datenmengen und die Analyse komplexer Zusammenhänge. Zum anderen können mithilfe von KI erzeugte synthetische – also künstliche – Daten die Privatsphäre schützen.In diesem Beitrag wird das Projekt KI-FDZ vorgestellt, welches innovative Technologien erforscht, die eine sichere Bereitstellung von Sekundärdaten für Forschungszwecke gewährleisten können. Es wird ein mehrschichtiger Ansatz untersucht, bei dem Maßnahmen auf Datenebene auf unterschiedliche Weise mit der Verarbeitung in sicheren Umgebungen kombiniert werden können. Dazu werden unter anderem Anonymisierungs- und Synthetisierungsmethoden anhand von 2 konkreten Anwendungsbeispielen evaluiert. Zudem wird untersucht, wie das Erstellen von Pipelines für maschinelles Lernen und die Ausführung von KI-Algorithmen in sicheren Umgebungen gestaltet werden können. Vorläufige Ergebnisse deuten darauf hin, dass mit diesem Ansatz ein hohes Maß an Schutz bei gleichzeitig hoher Datenvalidität erreicht werden kann. Der im Projekt untersuchte Ansatz kann ein wichtiger Baustein für die sichere Sekundärnutzung von Gesundheitsdaten sein.

show abstract

REaLTabFormer: Generating Realistic Relational and Tabular Data using Transformers

Cited by 6 publications

References 0 publications

Tabular Transformer Generative Adversarial Network for Heterogeneous distribution in healthcare

Tabular Transformer Generative Adversarial Network for Heterogeneous distribution in healthcare

GPT and Interpolation-Based Data Augmentation for Multiclass Intrusion Detection in IIoT

Künstliche Intelligenz und sichere Gesundheitsdatennutzung im Projekt KI-FDZ: Anonymisierung, Synthetisierung und sichere Verarbeitung für Real-World-Daten

Contact Info

Product

Resources

About