MultiRocket: multiple pooling operators and transformations for fast and effective time series classification

Tan, Chang Wei; Dempster, Angus; Bergmeir, Christoph; Webb, Geoffrey I.

doi:10.1007/s10618-022-00844-1

Cited by 94 publications

(51 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, the superiorities of deep learning-based methods have been proved in feature representation learning, in which less prior knowledge is used. Many complicated neural networks, such as recurrent neural networks, convolutional neural networks [4], [6], [36], and self-attentive-based models, have been applied for time series classification tasks. Besides, mining graph structure among time series with graph convolutional networks for comprehensive analysis have also attracted some researchers [37].…”

Section: Time Series Classificationmentioning

confidence: 99%

TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders

Cheng¹,

Liu²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Enhancing the expressive capacity of deep learning-based time series models with self-supervised pre-training has become ever-increasingly prevalent in time series classification. Even though numerous efforts have been devoted to developing self-supervised models for time series data, we argue that the current methods are not sufficient to learn optimal time series representations due to solely unidirectional encoding over sparse point-wise input units. In this work, we propose TimeMAE, a novel self-supervised paradigm for learning transferrable time series representations based on transformer networks. The distinct characteristics of the TimeMAE lie in processing each time series into a sequence of non-overlapping sub-series via window-slicing partitioning, followed by random masking strategies over the semantic units of localized sub-series. Such a simple yet effective setting can help us achieve the goal of killing three birds with one stone, i.e., (1) learning enriched contextual representations of time series with a bidirectional encoding scheme;(2) increasing the information density of basic semantic units; (3) efficiently encoding representations of time series using transformer networks. Nevertheless, it is a non-trivial to perform reconstructing task over such a novel formulated modeling paradigm. To solve the discrepancy issue incurred by newly injected masked embeddings, we design a decoupled autoencoder architecture, which learns the representations of visible (unmasked) positions and masked ones with two different encoder modules, respectively. Furthermore, we construct two types of informative targets to accomplish the corresponding pretext tasks. One is to create a tokenizer module that assigns a codeword to each masked region, allowing the masked codeword classification (MCC) task to be completed effectively. Another one is to adopt a siamese network structure to generate target representations for each masked input unit, aiming at performing the masked representation regression (MRR) optimization. Comprehensively pre-trained, our model can efficiently learn transferrable time series representations, thus benefiting the classification of time series. We extensively perform experiments on five benchmark datasets to verify the effectiveness of the TimeMAE. Experimental results show that the TimeMAE can significantly surpass previous competitive baselines. Furthermore, we also demonstrate the universality of the learned representations by performing transfer learning experiments. For the reproducibility of our results, we make our experiment codes publicly available to facilitate the self-supervised representations of time series in https://github.com/Mingyue-Cheng/TimeMAE.

show abstract

Section: Time Series Classificationmentioning

confidence: 99%

TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders

Cheng¹,

Liu²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Furthermore, literatures 37,42,43 demonstrate that, under the features produced by the ROCKET, a linear classifier can develop higher classification accuracy than other classifiers, even for datasets where the number of features dwarfs both the number and length of samples (1D sequence).…”

Section: Random Convolution Kernel Transformmentioning

confidence: 99%

“…Regarding the above three problems, we proposed a new strategy to detect bolted flange looseness in the underwater environment using percussion-based method and Feature-reduced Multiple Random Convolution Kernel Transform (FM-ROCKET). The FM-ROCKET model is developed based on the Multi-ROCKET model 37 The rest of this paper is organized as follows: Section 'Feature extraction: MFCC' introduces the MFCC which we use to compare with the proposed method. Section 'The proposed method: FM-ROCKET' elaborates relative theoretical background and the proposed method.…”

Section: Introductionmentioning

confidence: 99%

Underwater bolted flange looseness detection using percussion-induced sound and Feature-reduced Multi-ROCKET model

Chen

Zheng

Zhu

et al. 2023

Structural Health Monitoring

View full text Add to dashboard Cite

Recently, in the field of structural health monitoring, the detection of bolted connection looseness through percussion-based method and machine learning technology has received much attention due to the advantages of removing the requirement of sensor installation and potential for automation. However, there are few such research which are performed in the underwater environment. The paper proposes a new method, Feature-reduced Multiple Random Convolution Kernel Transform (FM-ROCKET), to identify the looseness level of the underwater bolted connections based on the percussion-induced sound (audio signal). By integrating deep learning (DL) and shallow learning, the FM-ROCKET model uses the 1D convolutional layer (a DL method) to extract features from the percussion-induced audio signal and adopts the rigid classifier (linear classifier, a shallow learning method) to classify the features. Five different preload levels of the bolted flange are considered. A hammer is utilized to tap the flange surface and the continuous percussion-induced audio signal is collected by a smartphone in an underwater environment. After the audio signal segmentation, single-hit audio signals are fed into the FM-ROCKET model. To verify the effectiveness of the proposed method, three case studies are conducted on two flanges. In case study I, the proposed method slightly outperforms other DL-based methods under different training/test splitting ratios. In case studies II and III, the proposed method is far more effective than other DL-based methods on independent and different test sets. The results demonstrate the superiority of the FM-ROCKET model in the underwater detection of bolted flange looseness. To the best of our knowledge, this article is the first attempt to address the detection of bolted flange looseness in the underwater environment by combining percussion-based method, DL, and shallow learning.

show abstract

“…Accordingly, numerous efforts [36,49] have been devoted to the MTSC problem over the last decades. In general, most of these current works can be divided into two categories: pattern-based and feature-based models.…”

Section: Introductionmentioning

confidence: 99%

FormerTime: Hierarchical Multi-Scale Representations for Multivariate Time Series Classification

Cheng¹,

Liu²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Deep learning-based algorithms, e.g., convolutional networks, have significantly facilitated multivariate time series classification (MTSC) task. Nevertheless, they suffer from the limitation in modeling longrange dependence due to the nature of convolution operations.Recent advancements have shown the potential of transformers to capture long-range dependence. However, it would incur severe issues, such as fixed scale representations, temporal-invariant and quadratic time complexity, with transformers directly applicable to the MTSC task because of the distinct properties of time series data. To tackle these issues, we propose FormerTime, an hierarchical representation model for improving the classification capacity for the MTSC task. In the proposed FormerTime, we employ a hierarchical network architecture to perform multi-scale feature maps. Besides, a novel transformer encoder is further designed, in which an efficient temporal reduction attention layer and a well-informed contextual positional encoding generating strategy are developed. To sum up, FormerTime exhibits three aspects of merits: (1) learning hierarchical multi-scale representations from time series data, (2) inheriting the strength of both transformers and convolutional networks, and (3) tacking the efficiency challenges incurred by the self-attention mechanism. Extensive experiments performed on 10 publicly available datasets from UEA archive verify the superiorities of the FormerTime compared to previous competitive baselines. CCS CONCEPTS• Computing methodologies → Neural networks; • Mathematics of computing → Time series analysis.

show abstract

MultiRocket: multiple pooling operators and transformations for fast and effective time series classification

Cited by 94 publications

References 25 publications

TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders

TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders

Underwater bolted flange looseness detection using percussion-induced sound and Feature-reduced Multi-ROCKET model

FormerTime: Hierarchical Multi-Scale Representations for Multivariate Time Series Classification

Contact Info

Product

Resources

About