WOODS: Benchmarks for Out-of-Distribution Generalization in Time Series Tasks

Abstract

Machine learning models often fail to generalize well under distributional shifts. Understanding and overcoming these failures have led to a research field of Out-of-Distribution (OOD) generalization. Despite being extensively studied for static computer vision tasks, OOD generalization has been underexplored for time series tasks. To shine light on this gap, we present WOODS: eight challenging open-source time series benchmarks covering a diverse range of data modalities, such as videos, brain recordings, and sensor signals. We revise the existing OOD generalization algorithms for time series tasks and evaluate them using our systematic framework. Our experiments show a large room for improvement for empirical risk minimization and OOD generalization algorithms on our datasets, thus underscoring the new challenges posed by time series tasks. Code and documentation are available at https://woods-benchmarks.github.io/

Comparing Handcrafted Features and Deep Neural Representations for Domain Generalization in Human Activity Recognition

“…Adam [ 55 ] was adopted as the optimizer used for the training process. To reduce bias [ 16 ], results were averaged over nine combinations of three different batch sizes (64, 128, and 256) and three learning rates (0.0008, 0.001, and 0.003). To account for class imbalance, the percentage of instances per class in the training set was given to the cross-entropy loss function as class weights.…”

“…Gagnon et al. [ 16 ] included a HAR dataset in a benchmark to compare domain generalization methods applied to deep neural networks. The results indicate a 9.07% drop in accuracy from 93.35% ID to 84.28% OOD on a dataset where different devices worn in different positions characterize the possible domains.…”

“…As HAR naturally includes many kinds of possible domains, it can be considered an excellent sandbox to study the OOD generalization ability of learning algorithms (Domain Generalization), being previously used for this purpose [ 16 ].…”

Exploring Regularization Methods for Domain Generalization in Accelerometer-Based Human Activity Recognition

“…Some of the aforementioned regularization methods have been investigated as a potential solution to the OOD generalization problem in HAR. Gagnon et al [ 35 ] included a HAR dataset in their Domain Generalization benchmark. Their results indicate a 9.07% drop in accuracy from 93.35% In-Distribution (ID) to 84.28% OOD on a dataset where different devices worn in different positions characterize the possible domains.…”

Enhancing Evolving Domain Generalization through Dynamic Latent Representations

“…To address the issue, most previous DG methods focus on extracting domain-invariant features across several stationary source domains (Sun and Saenko 2016;Li et al 2018;Sagawa et al 2020;Arjovsky et al 2019). Nevertheless, domains could also be non-stationary and evolve along with certain structures (Wang et al 2022;Qin, Wang, and Li 2022;Yao et al 2022a;Gagnon-Audet et al 2022). For instance, banks assess whether a person is likely to default on a loan by examining factors such as income, career, and marital status.…”