NLP-Datasets

This is an attemp to compile a consolidated list of datasets available for different NLP tasks. This work is under progress.

Question Answering

1. DROP Allen NLP: https://allennlp.org/drop.html
2. Google Natural Questions: https://ai.google.com/research/NaturalQuestions
3. Stanford SQuAD 2.0: https://rajpurkar.github.io/SQuAD-explorer/
4. Stanford SQuAD: https://rajpurkar.github.io/SQuAD-explorer/
5. Microsoft MARCO: http://www.msmarco.org/dataset.aspx
6. CMU RACE: http://www.cs.cmu.edu/~glai1/data/race/
7. University of Washington TriviaQA: http://nlp.cs.washington.edu/triviaqa/
8. Microsoft WikiQA: https://www.microsoft.com/en-us/download/details.aspx?id=52419
9. CNN/ Dailymail: https://cs.nyu.edu/~kcho/DMQA/
10. NewsQA: https://datasets.maluuba.com/NewsQA/dl
11. TREC: http://trec.nist.gov/data/qamain.html

Conversational QA

1. CoQA: https://stanfordnlp.github.io/coqa/
2. CSQA: https://amritasaha1812.github.io/CSQA/download/
3. QuAC: https://quac.ai

Dialog Generation

A compilation for all types of dialog generation datasets: https://breakend.github.io/DialogDatasets/

Knowledge Graph Completion

1. FB15K (FreeBase): https://github.com/ttrouill/complex/tree/master/datasets
2. WN18 (WordNet): https://github.com/ttrouill/complex/tree/master/datasets

Machine Translation

1. WMT: http://www.statmt.org/wmt16/translation-task.html#download

Sentiment Analysis

1. IMDb Movie Review: http://ai.stanford.edu/~amaas/data/sentiment/
2. Movie review data (Cornell): http://www.cs.cornell.edu/people/pabo/movie-review-data/
3. Yelp dataset: https://www.yelp.com/dataset/challenge

Linked Open Data

1. RDF data in HDT format: http://www.rdfhdt.org/datasets/
2. Wikidata: https://www.wikidata.org/wiki/Wikidata:Database_download

Image Caption Generation

1. MS COCO: http://cocodataset.org/#home
2. Flikr8K: http://nlp.cs.illinois.edu/HockenmaierGroup/Framing_Image_Description/KCCA.html
3. Flikr30K: http://shannon.cs.illinois.edu/DenotationGraph/
4. PASCAL : http://vision.cs.uiuc.edu/pascal-sentences/
5. Visual Genome: http://visualgenome.org
6. InstaPIC: https://github.com/cesc-park/attend2u
7. YFCC100M: http://yfcc100m.appspot.com

Visual QA:

1. VQA http://visualqa.org/download.html

Language Grounding / Visual Reasoning

1. CLEVR https://cs.stanford.edu/people/jcjohns/clevr/
2. NLVR http://lic.nlp.cornell.edu/nlvr/
3. FigureQA https://datasets.maluuba.com/FigureQA

Visual Commonsense Reasoning

1. VCR https://visualcommonsense.com/download/