Skip to content

Latest commit

 

History

History
112 lines (89 loc) · 12.3 KB

README.md

File metadata and controls

112 lines (89 loc) · 12.3 KB
Raw

Transformers in Time Series

Awesome PRs Welcome Stars Visits Badge

A professionally curated list of awesome resources (paper, code, data, etc.) on Transformers in Time Series, which is first work to comprehensively and systematically summarize the recent advances of Transformers for modeling time series data to the best of our knowledge.

We will continue to update this list with newest resources. If you found any missed resources (paper/code) or errors, please feel free to open an issue or make a pull request.

Survey paper

Transformers in Time Series: A Survey

Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan and Liang Sun.

If you find this repository helpful for your work, please kindly cite our survey paper.

@article{wen2022tstransformers,
  title={Transformers in Time Series: A Survey},
  author={Wen, Qingsong and Zhou, Tian and Zhang, Chaoli and Chen, Weiqi and Ma, Ziqing and Yan, Junchi and Sun, Liang},
  journal={arXiv preprint arXiv:2202.07125},
  year={2022}
}

Taxonomy of Transformers for time series modeling


Application Domains of Time Series Transformers

Transformers in Forecasting

Time Series Forecasting

  • FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting, in ICML 2022. [paper]
  • Pyraformer: Low-Complexity Pyramidal Attention for Long-Range Time Series Modeling and Forecasting, in ICLR 2022. [paper]
  • Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting, in NeurIPS 2021. [paper] [official code]
  • Informer: Beyond efficient transformer for long sequence time-series forecasting, in AAAI 2021. [paper] [official code] [dataset]
  • Temporal fusion transformers for interpretable multi-horizon time series forecasting, in International Journal of Forecasting 2021. [paper] [code]
  • Probabilistic Transformer For Time Series Analysis, in NeurIPS 2021. [paper]
  • Adversarial sparse transformer for time series forecasting, in NeurIPS 2020. [paper] [code]
  • Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting, in NeurIPS 2019. [paper] [code]
  • SSDNet: State Space Decomposition Neural Network for Time Series Forecasting, in ICDM 2021, [paper]
  • From Known to Unknown: Knowledge-guided Transformer for Time-Series Sales Forecasting in Alibaba, in arXiv 2021. [paper]
  • TCCT: Tightly-coupled convolutional transformer on time series forecasting, in Neurocomputing 2022. [paper]
  • Triformer: Triangular, Variable-Specific Attentions for Long Sequence Multivariate Time Series Forecasting, in IJCAI 2022. [paper]
  • Learning to Rotate: Quaternion Transformer for Complicated Periodical Time Series Forecasting”, in KDD 2022.

Spatio-Temporal Forecasting

  • Spatio-temporal graph transformer networks for pedestrian trajectory prediction, in ECCV 2020. [paper] [official code]
  • Spatial-temporal transformer networks for traffic flow forecasting, in arXiv 2020. [paper] [official code]
  • Traffic transformer: Capturing the continuity and periodicity of time series for traffic forecasting, in Transactions in GIS 2022. [paper]

Event Forecasting

Transformers in Anomaly Detection

  • Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy, in ICLR 2022. [paper] [code]
  • TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data, in VLDB 2022. [paper] [official code]
  • Learning graph structures with transformer for multivariate time series anomaly detection in IoT, in IEEE Internet of Things Journal 2021. [paper] [official code]
  • Spacecraft Anomaly Detection via Transformer Reconstruction Error, in ICASSE 2019. [paper]
  • Unsupervised Anomaly Detection in Multivariate Time Series through Transformer-based Variational Autoencoder, in CCDC 2021. [paper]
  • Variational Transformer-based anomaly detection approach for multivariate time series, in Measurement 2022. [paper]

Transformers in Classification

  • A transformer-based framework for multivariate time series representation learning, in KDD 2021. [paper] [official code]
  • Voice2series: Reprogramming acoustic models for time series classification, in ICML 2021. [paper] [official code]
  • Gated Transformer Networks for Multivariate Time Series Classification, in arXiv 2021. [paper] [official code]
  • Self-attention for raw optical satellite time series classification, in ISPRS Journal of Photogrammetry and Remote Sensing 2020. [paper] [official code]
  • Self-supervised pretraining of transformers for satellite image time series classification, in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 2020. [paper]
  • Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Series, in ACM TKDD 2022. [paper]

Time Series Related Survey

  • Time series data augmentation for deep learning: a survey, in IJCAI 2021. [paper]
  • Neural temporal point processes: a review, in IJCAI 2021. [paper]
  • Time-series forecasting with deep learning: a survey, in Philosophical Transactions of the Royal Society A 2021. [paper]
  • Deep learning for time series forecasting: a survey, in Big Data 2021. [paper]
  • Neural forecasting: Introduction and literature overview, in arXiv 2020. [paper]
  • Deep learning for anomaly detection in time-series data: review, analysis, and guidelines, in Access 2021. [paper]
  • A review on outlier/anomaly detection in time series data, in ACM Computing Surveys 2021. [paper]
  • A unifying review of deep and shallow anomaly detection, in Proceedings of the IEEE 2021. [paper]
  • Deep learning for time series classification: a review, in Data Mining and Knowledge Discovery 2019. [paper]

Transformer/Attention Survey in Other Disciplines

  • A survey on visual transformer, in IEEE TPAMI 2022. [paper]
  • A General Survey on Attention Mechanisms in Deep Learning, in IEEE TKDE 2022. [paper]
  • Attention, please! A survey of neural attention models in deep learning, in Artificial Intelligence Review 2022. [paper]
  • Attention mechanisms in computer vision: A survey, in Computational Visual Media 2022. [paper]
  • Survey: Transformer based video-language pre-training, in AI Open 2022. [paper]
  • Transformers in vision: A survey, in ACM Computing Surveys 2021. [paper]
  • Pre-trained models: Past, present and future, in AI Open 2021. [paper]
  • An attentive survey of attention models, in ACM TIST 2021. [paper]
  • Attention in natural language processing, in IEEE TNNLS 2020. [paper]
  • Pre-trained models for natural language processing: A survey, in Science China Technological Sciences 2020. [paper]
  • Efficient transformers: A survey, in arXiv 2020. [paper]
  • A Survey of Transformers, in arXiv 2021. [paper]
  • A Survey of Vision-Language Pre-Trained Models, in arXiv 2022. [paper]
  • Video Transformers: A Survey, in arXiv 2022. [paper]
  • Transformer for Graphs: An Overview from Architecture Perspective, in arXiv 2022. [paper]
  • Transformers in Medical Imaging: A Survey, in arXiv 2022. [paper]
  • A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models, in arXiv 2022. [paper]