TL;DR List of scientific articles for deep learning and music: summary (few info), details (table - more info), details (bib - all info)

Deep Learning for Music (DL4M)

The role of this curated list is to gather scientific articles, thesis and reports that use deep learning approaches applied to music. The list is currently under construction but feel free to contribute to the missing fields and to add other resources. The resources provided here come from my review of the state-of-the-art for my PhD Thesis for which an article is being written. There are already surveys on deep learning for music generation, speech separation and speaker identification. However, these surveys do not cover music information retrieval tasks that are included in this repository.

Table of contents

• DL4M summary
• DL4M details
• Code without articles
• Statistics and visualisations
• How To Contribute
• FAQ
• Acronyms used
• Sources
• Contributors
• Other useful related lists
• Cited by

DL4M summary

Articles, Thesis and Reports	Code
An efficient approach for segmentation, feature extraction and classification of audio signals	No
Gabor frames and deep scattering networks in audio processing	No
Vision-based detection of acoustic timed events: A case study on clarinet note onsets	No
Neural net modeling of music	No
Deep learning techniques for music generation - A survey	No
A supervised learning approach to musical style recognition	No
XFlow: 1D <-> 2D cross-modal deep neural networks for audiovisual classification	No
Machine listening intelligence	No
Deep multimodal network for multi-label classification	No
A tutorial on deep learning for music information retrieval	GitHub
A comparison on audio signal preprocessing methods for deep neural networks on music tagging	GitHub
Text-based LSTM networks for automatic music composition	No
Towards playlist generation algorithms using RNNs trained on within-track transitions	No
Automatic tagging using deep convolutional neural networks	No
Transfer learning for music classification and regression tasks	GitHub
Convolutional recurrent neural networks for music classification	GitHub
Auralisation of deep convolutional neural networks: Listening to learned features	No
An evaluation of convolutional neural networks for music classification using spectrograms	No
The munich LSTM-RNN approach to the MediaEval 2014 Emotion In Music task	No
A machine learning approach to musical style recognition	No
Automatic chord estimation on seventhsbass chord vocabulary using deep neural network	No
Large vocabulary automatic chord estimation using deep neural nets: Design framework, system variations and limitations	No
Audio-based music classification with a pretrained convolutional network	No
Multiscale approaches to music audio feature learning	No
End-to-end learning for music audio	No
Basic filters for convolutional neural networks: Training or design?	No
Ensemble Of Deep Neural Networks For Acoustic Scene Classification	No
Robust downbeat tracking using an ensemble of convolutional networks	No
Downbeat tracking with multiple features and deep neural networks	No
Music signal processing using vector product neural networks	No
Deep learning for music genre classification	No
Multi-phase learning for jazz improvisation and interaction	No
Recognition of acoustic events using deep neural networks	No
Transforming musical signals through a genre classifying convolutional neural network	No
Audio to score matching by combining phonetic and duration information	GitHub
Musical networks: Parallel distributed perception and performance	No
Music boundary detection using neural networks on spectrograms and self-similarity lag matrices	No
Deep image features in music information retrieval	No
Interactive music generation with positional constraints using anticipation-RNNs	No
Deep rank-based transposition-invariant distances on musical sequences	No
GLSR-VAE: Geodesic latent space regularization for variational autoencoder architectures	No
DeepBach: A steerable model for Bach chorales generation	GitHub
Deep convolutional neural networks for predominant instrument recognition in polyphonic music	No
CNN architectures for large-scale audio classification	No
Classification of spatial audio location and content using convolutional neural networks	No
Bayesian meter tracking on learned signal representations	No
DeepSheet: A sheet music generator based on deep learning	No
Deep learning for music	No
From music audio to chord tablature: Teaching deep convolutional networks to play guitar	No
Rethinking automatic chord recognition with convolutional neural networks	No
Moving beyond feature design: Deep architectures and automatic feature learning in music informatics	No
Talking Drums: Generating drum grooves with neural networks	No
Singing voice separation with deep U-Net convolutional networks	No
Music emotion recognition via end-to-end multimodal neural networks	No
Learning temporal features using a deep neural network and its application to music genre classification	No
On the potential of simple framewise approaches to piano transcription	No
Deep learning, audio adversaries, and music content analysis	No
Deep learning and music adversaries	GitHub
Chord label personalization through deep learning of integrated harmonic interval-based representations	No
Feature learning for chord recognition: The deep chroma extractor	GitHub
A fully convolutional deep auditory model for musical chord recognition	No
End-to-end musical key estimation using a convolutional neural network	No
MediaEval 2017 AcousticBrainz genre task: Multilayer perceptron approach	No
Classification-based singing melody extraction using deep convolutional neural networks	No
The representation of pitch in a neural net model of chord classification	No
Unsupervised feature learning for audio classification using convolutional deep belief networks	No
Multi-level and multi-scale feature aggregation using pre-trained convolutional neural networks for music auto-tagging	No
Multi-level and multi-scale feature aggregation using sample-level deep convolutional neural networks for music classification	GitHub
Sample-level deep convolutional neural networks for music auto-tagging using raw waveforms	No
Singing voice detection with deep recurrent neural networks	No
Creation by refinement: A creativity paradigm for gradient descent learning networks	No
Algorithms for music composition by neural nets: Improved CBR paradigms	No
Audio musical genre classification using convolutional neural networks and pitch and tempo transformations	No
Automatic instrument recognition in polyphonic music using convolutional neural networks	No
Automatic musical pattern feature extraction using convolutional neural network	No
A deep bidirectional long short-term memory based multi-scale approach for music dynamic emotion prediction	No
Harmonic and percussive source separation using a convolutional auto encoder	No
Event localization in music auto-tagging	GitHub
Deep convolutional networks on the pitch spiral for musical instrument recognition	GitHub
Stacked convolutional and recurrent neural networks for music emotion recognition	No
A deep learning approach to source separation and remixing of hiphop music	No
Neural network based model for classification of music type	No
A software framework for musical data augmentation	No
Music Genre Classification Using Masked Conditional Neural Networks	No
SampleRNN: An unconditional end-to-end neural audio generation model	GitHub
Monaural Singing Voice Separation with Skip-Filtering Connections and Recurrent Inference of Time-Frequency Mask	GitHub
Generating data to train convolutional neural networks for classical music source separation	GitHub
Monaural score-informed source separation for classical music using convolutional neural networks	GitHub
Neural network music composition by prediction: Exploring the benefits of psychoacoustic constraints and multi-scale processing	No
Local-feature-map integration using convolutional neural networks for music genre classification	No
A deep bag-of-features model for music auto-tagging	No
Learning sparse feature representations for music annotation and retrieval	No
A convolutional-kernel based approach for note onset detection in piano-solo audio signals	No
Multi-label music genre classification from audio, text, and images using deep features	GitHub
A deep multimodal approach for cold-start music recommendation	GitHub
Melody extraction and detection through LSTM-RNN with harmonic sum loss	No
Representation learning of music using artist labels	No
Music-noise segmentation in spectrotemporal domain using convolutional neural networks	No
Musical instrument sound classification with deep convolutional neural network using feature fusion approach	No
Toward inverse control of physics-based sound synthesis	Website
Robust audio event recognition with 1-max pooling convolutional neural networks	No
DNN and CNN with weighted and multi-task loss functions for audio event detection	No
Environmental sound classification with convolutional neural networks	No
Score-informed syllable segmentation for a cappella singing voice with convolutional neural networks	GitHub
Experimenting with musically motivated convolutional neural networks	GitHub
End-to-end learning for music audio tagging at scale	GitHub
Designing efficient architectures for modeling temporal features with convolutional neural networks	GitHub
Timbre analysis of music audio signals with convolutional neural networks	GitHub
Monoaural audio source separation using deep convolutional neural networks	GitHub
Deep learning and intelligent audio mixing	No
Singing voice melody transcription using deep neural networks	No
Singing voice separation using deep neural networks and F0 estimation	Website
Learning to pinpoint singing voice from weakly labeled examples	No
Deep learning for event detection, sequence labelling and similarity estimation in music signals	No
Improved musical onset detection with convolutional neural networks	No
Musical onset detection with convolutional neural networks	No
Exploring data augmentation for improved singing voice detection with neural networks	GitHub
Music feature maps with convolutional neural networks for music genre classification	No
Singer traits identification using deep neural network	No
A hybrid recurrent neural network for music transcription	No
An end-to-end neural network for polyphonic music transcription	No
Deep karaoke: Extracting vocals from musical mixtures using a convolutional deep neural network	No
Automatic drum transcription for polyphonic recordings using soft attention mechanisms and convolutional neural networks	GitHub
Analysis of time-frequency representations for musical onset detection with convolutional neural network	No
Note onset detection in musical signals via neural-network-based multi-ODF fusion	No
Adversarial semi-supervised audio source separation applied to singing voice extraction	No
Taking the models back to music practice: Evaluating generative transcription models built using deep learning	GitHub
Music transcription modelling and composition using deep learning	GitHub
Folk music style modelling by recurrent neural networks with long short term memory units	GitHub
Convolutional neural network for robust pitch determination	No
Deep convolutional neural networks and data augmentation for acoustic event detection	Website
Generating nontrivial melodies for music as a service	No
Invariances and data augmentation for supervised music transcription	GitHub
A sequential network design for musical applications	No
A connectionist approach to algorithmic composition	No
Lyrics-based music genre classification using a hierarchical attention network	GitHub
Deep neural network based instrument extraction from music	No
Boundary detection in music structure analysis using convolutional neural networks	No
A hybrid DSP/deep learning approach to real-time full-band speech enhancement	GitHub
Deep content-based music recommendation	No
Convolutional methods for music analysis	No
Extending temporal feature integration for semantic audio analysis	No
Recognition and retrieval of sound events using sparse coding convolutional neural network	No
A two-stage approach to note-level transcription of a specific piano	No
Improving content-based and hybrid music recommendation using deep learning	No
Reducing model complexity for DNN based large-scale audio classification	No
Audio spectrogram representations for processing with convolutional neural networks	Website
Unsupervised learning of local features for music classification	No
Unsupervised feature learning based on deep models for environmental audio tagging	No
Surrey-CVSSP system for DCASE2017 challenge task4	GitHub
A study on LSTM networks for polyphonic music sequence modelling	Website
Attention and localization based on a deep convolutional recurrent model for weakly supervised audio tagging	GitHub
A deep representation for invariance and music classification	No
A deep neural network for modeling music	No

Go back to top

DL4M details

A human-readable table summarized version if displayed in the file dl4m.tsv. All details for each article are stored in the corresponding bib entry in dl4m.bib. Each entry has the regular bib field:

• author
• year
• title
• journal or booktitle

Each entry in dl4m.bib also displays additional information:

• link - HTML link to the PDF file
• code - Link to the source code if available
• archi - Neural network architecture
• layer - Number of layers
• task - The proposed tasks studied in the article
• dataset - The names of the dataset used
• dataaugmentation - The type of data augmentation technique used
• time - The computation time
• hardware - The hardware used
• note - Additional notes and information
• repro - Indication to what extent the experiments are reproducible

Go back to top

Code without articles

• Audio Classifier in Keras using Convolutional Neural Network
• Deep learning driven jazz generation using Keras & Theano
• End-to-end learning for music audio tagging at scale
• Music Genre classification on GTZAN dataset using CNNs
• Pitch Estimation of Choir Music using Deep Learning Strategies: from Solo to Unison Recordings
• Music Genre Classification with LSTMs
• CNN based Music Emotion Classification using TensorFlow
• Separating singing voice from music based on deep neural networks in Tensorflow

Go back to top

Statistics and visualisations

• 154 papers referenced. See the details in dl4m.bib.
• If you are applying DL to music, there are 319 other researchers in your field.
• 31 tasks investigated. See the list of tasks.
• 42 datasets used. See the list of datasets.
• 17 architectures used. See the list of architectures.
• Number of articles per year:
• Only 38 articles (24%) provide their source code. Repeatability is the key to good science, so check out the list of useful resources on reproducibility for MIR and ML.

Go back to top

How To Contribute

Contributions are welcome! Please refer to the CONTRIBUTING.md file.

Go back to top

FAQ

How are the articles sorted?

The articles are first sorted by decreasing year (to keep up with the latest news) and then alphabetically by the main author's family name.

Why are preprint from arXiv included in the list?

I want to have exhaustive research and the latest news on DL4M. However, one should take care of the information provided in the articles currently in review. If possible you should wait for the final accepted and peer-reviewed version before citing an arXiv paper. I regularly update the arXiv links to the corresponding published papers when available.

How much can I trust the results published in an article?

The list provided here does not guarantee the quality of the articles. You should either try to reproduce the experiments described or submit a request to ReScience. Use one article's conclusion at your own risks.

Go back to top

Acronyms used

A list of useful acronyms used in deep learning and music is stored in acronyms.md.

Go back to top

Sources

The list of conferences, journals and aggregators used to gather the proposed materials is stored in sources.md.

Go back to top

Contributors

• Yann Bayle (GitHub) - Instigator and principal maintainer
• Vincent Lostanlen (GitHub)
• Keunwoo Choi (GitHub)
• Bob L. Sturm (GitHub)
• Stefan Balke (GitHub)
• Jordi Pons (GitHub)

Go back to top

Other useful related lists and resources

Audio

• DL4MIR tutorial with keras - Tutorial for Deep Learning on Music Information Retrieval by Thomas Lidy
• Video talk from Ron Weiss - Ron Weiss (Google) Talk on Training neural network acoustic models on waveforms
• Slides on DL4M - A personal (re)view of the state-of-the-art by Jordi Pons
• DL4MIR tutorial - Python tutorials for learning to solve MIR tasks with DL
• Awesome Python Scientific Audio - Python resources for Audio and Machine Learning
• ISMIR resources - Community maintained list
• ISMIR Google group - Daily dose of general MIR
• Awesome Python - Audio section of Python resources
• Awesome Web Audio - WebAudio packages and resources
• Awesome Music - Music softwares
• Awesome Music Production - Music creation
• The Asimov Institute - 6 deep learning tools for music generation
• DLM Google group - Deep Learning in Music group
• MIR community on Slack - Link to subscribe to the MIR community's Slack
• Unclassified list of MIR-related links - Cory McKay's list of various links on DL, MIR, ...
• MIRDL - Unmaintained list of DL articles for MIR from Jordi Pons

Go back to top

Music datasets

• AudioContentAnalysis nearly exhaustive list of music-related datasets
• Teaching MIR
• Wikipedia's list of datasets for machine learning research
• Datasets for deep learning
• Awesome public datasets

Go back to top

Deep learning

• DLPaper2Code: Auto-generation of Code from Deep Learning Research Papers -
• Model Convertors - Convertors for DL frameworks and backend
• Deep architecture genealogy - Genealogy of DL architectures
• Deep Learning as an Engineer - Slides from Jan Schlüter
• Awesome Deep Learning - General deep learning resources
• Awesome Deep Learning Resources - Papers regarding deep learning and deep reinforcement learning
• Awesome RNNs - RNNs code, theory and applications
• Cheatsheets AI - Cheat Sheets for Keras, neural networks, scikit-learn,...
• DL PaperNotes - Summaries and notes on general deep learning research papers
• General lists
• Echo State Network
• DL in NLP - Best practices for using neural networks by Sebastian Ruder
• CNN overview - Stanford Course
• Dilated Recurrent Neural Networks - How to improve RNNs?
• Encoder-Decoder in RNNs - How Does Attention Work in Encoder-Decoder Recurrent Neural Networks
• On the use of DL - Misc fun around DL
• ML from scratch - Python implementations of ML models and algorithms from scratch from Data Mining to DL

Go back to top

Cited by

If you use the information contained in this repository, please let us know! This repository is cited by:

• Alexander Schindler
• Meinard Müller, Christof Weiss, Stefan Balke
• AINewsFeed

Go back to top