Leveraging Automated Machine Learning for Text Classification: Evaluation of AutoML Tools and Comparison with Human Performance

Blohm, Matthias; Hanussek, Marc; Kintz, Maximilien

doi:10.5220/0010331411311136

Cited by 14 publications

(8 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…That is the main reason that AutoML [25], [26] is a convenient alternative to achieve outstanding performance while saving time and effort of searching for the appropriate parameters. An interesting analysis is given by [27] comparing four AutoML tools with human performance over 13 commonly used datasets, and the obtained results were impressive as they show that AutoML tools outperform the machine learning process achieved by human data scientists in 4 of 13 tasks.…”

Section: Related Workmentioning

confidence: 99%

Dialectal Arabic sentiment analysis based on tree-based pipeline optimization tool

Mihi¹,

Ali²,

Bazi

et al. 2022

IJECE

View full text Add to dashboard Cite

The heavy involvement of the Arabic internet users resulted in spreading data written in the Arabic language and creating a vast research area regarding natural language processing (NLP). Sentiment analysis is a growing field of research that is of great importance to everyone considering the high added potential for decision-making and predicting upcoming actions using the texts produced in social networks. Arabic used in microblogging websites, especially Twitter, is highly informal. It is not compliant with neither standards nor spelling regulations making it quite challenging for automatic machine-learning techniques. In this paper’s scope, we propose a new approach based on AutoML methods to improve the efficiency of the sentiment classification process for dialectal Arabic. This approach was validated through benchmarks testing on three different datasets that represent three vernacular forms of Arabic. The obtained results show that the presented framework has significantly increased accuracy than similar works in the literature.

show abstract

Section: Related Workmentioning

confidence: 99%

Dialectal Arabic sentiment analysis based on tree-based pipeline optimization tool

Mihi¹,

Ali²,

Bazi

et al. 2022

IJECE

View full text Add to dashboard Cite

show abstract

“…There are works published that compare the performance of various existing AutoML tools on text classification tasks. Blohm et al [8] attempt to answer the question of whether AutoML can be effectively applied to text classification or not. They compare the performance of four AutoML tools on thirteen text classification datasets and document how they perform as opposed to human-engineered models.…”

Section: Review Of Literaturementioning

confidence: 99%

TextBrew: Automated Model Selection and Hyperparameter Optimization for Text Classification

Desai¹,

Shah²,

Kothari³

et al. 2022

IJACSA

View full text Add to dashboard Cite

In building a machine learning solution, algorithm selection and hyperparameter tuning is the most time-consuming task. Automated Machine Learning is a solution to fully automate the process of finding the best model for a given task without actually having to try various models. This paper introduces a new AutoML system, TextBrew, explicitly built for the NLP task of text classification. Our system provides an automated method for selecting transformer models, tuning hyperparameters, and combining the best models into one by ensembling. Keeping in mind that new state-of-the-art models are being constantly introduced, TextBrew has been designed to be highly flexible and thus can support additional models easily. In our work, we experiment with multiple transformer models, each with numerous different hyperparameter settings, and select the most robust models. These models are then trained on multiple datasets to obtain accuracy scores, which are then used to build the metadataset to train the meta-model. Since text classification datasets are not as abundant, our system generates synthetic data to augment the meta-dataset using CopulaGAN, a deep generative model. The meta-model is an ensemble of five models, which predicts the best candidate model with an accuracy of 78.75%. The final model returned to the user is an ensemble of all the best models that can be trained under the given time constraint. Experiments on various datasets and comparisons with existing systems demonstrate the effectiveness of our system.

show abstract

“…Additionally, benchmarks have been done that compare several AutoML approaches and their performance on different problems, also including NLP tasks [1]. For instance, in one of our previous works we have performed an opensource AutoML benchmark for text classification [10]. The focus was on comparing the classification results of four different AutoML tools both among each other and with the results of machine learning experts.…”

Section: Related Workmentioning

confidence: 99%

Evaluation of Representation Models for Text Classification with AutoML Tools

Brändle¹,

Hanussek²,

Blohm³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Automated Machine Learning (AutoML) has gained increasing success on tabular data in recent years. However, processing unstructured data like text is a challenge and not widely supported by open-source AutoML tools. This work compares three manually created text representations and text embeddings automatically created by Au-toML tools. Our benchmark includes four popular open-source AutoML tools and eight datasets for text classification purposes. The results show that straightforward text representations perform better than AutoML tools with automatically created text embeddings.

show abstract

Leveraging Automated Machine Learning for Text Classification: Evaluation of AutoML Tools and Comparison with Human Performance

Cited by 14 publications

References 8 publications

Dialectal Arabic sentiment analysis based on tree-based pipeline optimization tool

Dialectal Arabic sentiment analysis based on tree-based pipeline optimization tool

TextBrew: Automated Model Selection and Hyperparameter Optimization for Text Classification

Evaluation of Representation Models for Text Classification with AutoML Tools

Contact Info

Product

Resources

About