Category of Self-Supervised Learning for Time Series

Generative-based Methods on SSL4TS

In this category, the pretext task is to generate the expected data based on a given view of the data. In the context of time series modeling, the commonly used pretext tasks include using the past series to forecast the future windows or specific time stamps, using the encoder and decoder to reconstruct the input, and forecasting the unseen part of the masked time series. This section sorts out the existing self-supervised representation learning methods in time series modeling from the perspectives of autoregressive-based forecasting, autoencoder-based reconstruction, and diffusion-based generation. It should be noted that autoencoder-based reconstruction task is also viewed as an unsupervised framework. In the context of SSL, we mainly use the reconstruction task as a pretext task, and the final goal is to obtain the representations through autoencoder models. The illustration of the generative-based SSL for time series is shown in Fig. 3.

Autoregressive-based forecasting

• Timeseries anomaly detection using temporal hierarchical one-class network, in NeurIPS, 2020. [paper]
• Self-supervised transformer for sparse and irregularly sampled multivariate clinical time-series, in ACM Transactions on Knowledge Discovery from Data, 2022. [paper]
• Graph neural network-based anomaly detection in multivariate time series, in AAAI, 2021. [paper]
• Semisupervised time series classification model with self-supervised learning, in Engineering Applications of Artificial Intelligence, 2022. [paper]

Autoencoder-based reconstruction

• TimeNet: Pre-trained deep recurrent neural network for time series classification, in arXiv, 2017. [paper]
• Unsupervised pre-training of a deep LSTM-based stacked autoencoder for multivariate time series forecasting problems, in Scientific Reports, 2019. [paper]
• Autowarp: Learning a warping distance from unlabeled time series using sequence autoencoders, in NeurIPS, 2018. [paper]
• Practical approach to asynchronous multivariate time series anomaly detection and localization, in KDD, 2021. [paper]
• Learning representations for time series clustering, in NeurIPS, 2019. [paper]
• USAD: Unsupervised anomaly detection on multivariate time series, in KDD, 2020 [paper]
• Learning sparse latent graph representations for anomaly detection in multivariate time series, in KDD, 2022. [paper]
• Wind turbine fault detection using a denoising autoencoder with temporal information, in IEEE/ASME Transactions on Mechatronics, 2018 [paper]
• Denoising temporal convolutional recurrent autoencoders for time series classification, in Information Sciences, 2022. [paper]
• Pre-training enhanced spatial-temporal graph neural network for multivariate time series forecasting, in KDD, 2022. [paper]
• A transformer-based framework for multivariate time series representation learning, in KDD, 2021. [paper]
• Multi-variate time series forecasting on variable subsets, in KDD, 2022. [paper]
• TARNet: Task-aware reconstruction for time-series transformer, in KDD, 2022. [paper]
• Learning latent seasonal-trend representations for time series forecasting, in NeurIPS, 2022. [paper] [repo]
• Multivariate time series anomaly detection and interpretation using hierarchical inter-metric and temporal embedding, in KDD, 2021. [paper]
• Robust anomaly detection for multivariate time series through stochastic recurrent neural network, in KDD, 2019. [paper]
• GRELEN: Multivariate time series anomaly detection from the perspective of graph relational learning, in IJCAI, 2022. [paper]
• Deep variational graph convolutional recurrent network for multivariate time series anomaly detection, in ICML, 2022. [paper]
• Heteroscedastic temporal variational autoencoder for irregularly sampled time series, in ICLR, 2022. [paper]
• Learning from irregularly-sampled time series: A missing data perspective, in ICML, 2020. [paper]
• TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders, in arXiv, 2023. [paper] [code]

Diffusion-based generation

• CSDI: Conditional score-based diffusion models for probabilistic time series imputation, in NeurIPS, 2021. [paper]
• Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting, in ICML, 2021. [paper]
• Generative time series forecasting with diffusion, denoise, and disentanglement, in NeurIPS, 2022. [paper]
• ImDiffusion: Imputed diffusion models for multivariate time series anomaly detection, in arXiv, 2023. [paper]
• Diffusion-based time series imputation and forecasting with structured state space models, in Transactions on Machine Learning Research, 2022. [paper]
• Diffload: Uncertainty quantification in load forecasting with diffusion model, in arXiv, 2023. [paper]
• DiffSTG: Probabilistic spatio-temporal graph forecasting with denoising diffusion models, in SIGSPATIAL, 2023. [paper]
• PriSTI: A Conditional Diffusion Framework for Spatiotemporal Imputation, in ICDE, 2024. [paper] [code]

Contrastive-based Methods on SSL4TS

Contrastive learning is a widely used self-supervised learning strategy, showing a strong learning ability in computer vision and natural language processing. Unlike discriminative models that learn a mapping rule to true labels and generative models that try to reconstruct inputs, contrastive-based methods aim to learn data representations by contrasting between positive and negative samples. Specifically, positive samples should have similar representations, while negative samples have different representations. Therefore, the selection of positive samples and negative samples is very important to contrastive-based methods. This section sorts out and summarizes the existing contrastive-based methods in time series modeling according to the selection of positive and negative samples. The illustration of the contrastive-based SSL for time series is shown in Fig. 4.

Sampling contrast

• Unsupervised scalable representation learning for multivariate time series, in NeurIPS, 2019. [paper]
• Unsupervised representation learning for time series with temporal neighborhood coding, in ICLR, 2021. [paper]
• Neighborhood contrastive learning applied to online patient monitoring, in ICML, 2021. [paper]

Prediction contrast

• Representation learning with contrastive predictive coding, in arXiv, 2018. [paper]
• Detecting anomalies within time series using local neural transformations, in arXiv, 2022. [paper]
• Contrastive predictive coding for anomaly detection in multi-variate time series data, in arXiv, 2022. [paper]
• Time series change point detection with self-supervised contrastive predictive coding, in WWW, 2021. [paper]
• Time Series Anomaly Detection using Skip-Step Contrastive Predictive Coding, in *NeurIPS Workshop: Self-Supervised Lea