AutoTransformer: Automatic Transformer Architecture Design for Time Series Classification

Ren, Yankun; Li, Longfei; Yang, Xinxing; Zhou, Jun

doi:10.1007/978-3-031-05933-9_12

Cited by 10 publications

(4 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…integrates deformable convolutional blocks and online knowledge distillation, as well as a random mask to reduce noise [137]. For each TSC dataset, AutoTransformer searches for the suitable network architecture using the neural architecture search (NAS) algorithm before feeding the output to the multi-headed attention blocks [138].…”

Section: Transformersmentioning

confidence: 99%

Deep Learning for Time Series Classification and Extrinsic Regression: A Current Survey

Foumani¹,

Miller²,

Tan³

et al. 2023

Preprint

View full text Add to dashboard Cite

Time Series Classification and Extrinsic Regression are important and challenging machine learning tasks. Deep learning has revolutionized natural language processing and computer vision and holds great promise in other fields such as time series analysis where the relevant features must often be abstracted from the raw data but are not known a priori. This paper surveys the current state of the art in the fast-moving field of deep learning for time series classification and extrinsic regression. We review different network architectures and training methods used for these tasks and discuss the challenges and opportunities when applying deep learning to time series data. We also summarize two critical applications of time series classification and extrinsic regression, human activity recognition and satellite earth observation.

show abstract

Section: Transformersmentioning

confidence: 99%

Deep Learning for Time Series Classification and Extrinsic Regression: A Current Survey

Foumani¹,

Miller²,

Tan³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…In recent years, the Transformer has also shown powerful ability to capture long-term dependencies and correlations in time series research. Various Transformer variants have been applied by researchers to solve problems in time series, including classification [22][23] [24], prediction [25][26] [27], and anomaly detection [28][29] [30], among others. In a recent study on classification tasks, Jayant et al [23] proposed a Transformer-based framework for driving behavior classification, analyzing driving behavior through smartphone telematics data belonging to multivariate time series, instead of using convolutional or recurrent network architectures.…”

Section: Transformer For Time Seriesmentioning

confidence: 99%

“…In a recent study on classification tasks, Jayant et al [23] proposed a Transformer-based framework for driving behavior classification, analyzing driving behavior through smartphone telematics data belonging to multivariate time series, instead of using convolutional or recurrent network architectures. Ren et al [24] proposed a data-driven design method for time series classification network architecture called AutoTransformer, which can find network structures suitable for different datasets from a novel search space.…”

Section: Transformer For Time Seriesmentioning

confidence: 99%

DSNet: A Dual-Scaled Network for Multivariate Time Series Classification with Learning Temporal Features

Zhang

2023

Preprint

View full text Add to dashboard Cite

Multivariate time series classification (MTSC) is a widely recognized and important task in machine learning. Current methods often fail to account for the correlation between holistic and local temporal features. In this paper, we propose a novel dual-scaled network called DSNet, which combines holistic and local features and merges temporal information through a dual-scaled gating mechanism. The DSNet model's holistic module introduces the encoder part of the native Transformer, while the local module mainly consists of our proposed residual convolution module (ResBlock) and Squeeze-And-Excite Block to extract local features. Our experimental results show that our proposed method achieves higher accuracy on 12 out of the 20 UEA MTSC archive datasets. Furthermore, our method has a higher average ranking of 1.55 compared to the other 7 state-of-the-art methods.

show abstract

“…AutoTransformer [148] first designs a customized search space for Time Series Classification (TSC) task and applies Gradient-based Differentiable Architecture Sampler (GDAS) [149], an improved DARTS method, to search for an efficient Transformer. The search space incorporates several structures and operations within the Transformer backbone, which can extract global and local features from the time series.…”

Section: H Autotransformermentioning

confidence: 99%

Neural Architecture Search for Transformers: A Survey

et al. 2022

View full text Add to dashboard Cite

Transformer-based Deep Neural Network architectures have gained tremendous interest due to their effectiveness in various applications across Natural Language Processing (NLP) and Computer Vision (CV) domains. These models are the de facto choice in several language tasks, such as Sentiment Analysis and Text Summarization, replacing Long Short Term Memory (LSTM) model. Vision Transformers (ViTs) have shown better model performance than traditional Convolutional Neural Networks (CNNs) in vision applications while requiring significantly fewer parameters and training time. The design pipeline of a neural architecture for a given task and dataset is extremely challenging as it requires expertise in several interdisciplinary areas such as signal processing, image processing, optimization and allied fields. Neural Architecture Search (NAS) is a promising technique to automate the architectural design process of a Neural Network in a data-driven way using Machine Learning (ML) methods. The search method explores several architectures without requiring significant human effort, and the searched models outperform the manually built networks. In this paper, we review Neural Architecture Search techniques, targeting the Transformer model and its family of architectures such as Bidirectional Encoder Representations from Transformers (BERT) and Vision Transformers. We provide an in-depth literature review of approximately 50 state-of-theart Neural Architecture Search methods and explore future directions in this fast-evolving class of problems.

show abstract

AutoTransformer: Automatic Transformer Architecture Design for Time Series Classification

Cited by 10 publications

References 21 publications

Deep Learning for Time Series Classification and Extrinsic Regression: A Current Survey

Deep Learning for Time Series Classification and Extrinsic Regression: A Current Survey

DSNet: A Dual-Scaled Network for Multivariate Time Series Classification with Learning Temporal Features

Neural Architecture Search for Transformers: A Survey

Contact Info

Product

Resources

About