code/collect_data/data_collection.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:c9b7ff4ae9d62d1c8d9a1c5187ed201eaa52e64aef2840d110a1442e01a33f52"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%load_ext watermark"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%watermark -a 'Sebastian Raschka' -d -v"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Sebastian Raschka 07/12/2014 \n",
        "\n",
        "CPython 3.4.2\n",
        "IPython 2.3.0\n"
       ]
      }
     ],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "\n",
      "<font size=\"1.5em\">[More information](http://nbviewer.ipython.org/github/rasbt/python_reference/blob/master/ipython_magic/watermark.ipynb) about the `watermark` magic command extension.</font>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br>\n",
      "<br>"
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Data Collection"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br>\n",
      "<br>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<hr>"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Sections"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<hr>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "- [Downloading the Dataset](#Downloading-the-Dataset)\n",
      "- [Compiling a Title-Artist Table](#Compiling-a-Title-Artist-Table)\n",
      "- [Downloading Lyrics](#Downloading-Lyrics)\n",
      "    - [Adding lyrics to the DataFrame](#Adding-lyrics-to-the-DataFrame)\n",
      "    - [Remove Rows where Lyrics are not available](#Remove-Rows-where-Lyrics-are-not-available)\n",
      "- [Language Filter](Language-Filter)\n",
      "    - [Remove all non-English lyrics](#Remove-all-non-English-lyrics)\n",
      "- [Create a filtered dataset](#Create-a-filtered-dataset)\n",
      "- [Randomly partition the dataset into separate training and validation sets](#Randomly-partition-the-dataset-into-separate-training-and-validation-sets)\n",
      "- [Make new CSV tables for the Training and Validation dataset](#Make-new-CSV-tables-for-the-Training-and-Validation-dataset)\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br>\n",
      "<br>"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Downloading the Dataset"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "[[back to top](#Sections)]"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "A subset of 10,000 songs in  `HDF5` format was downloaded from the [Million Song Dataset](http://labrosa.ee.columbia.edu/millionsong/pages/getting-dataset). A feature list of the file contents can be found [here](http://labrosa.ee.columbia.edu/millionsong/pages/field-list)."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The following snippet flattens the directory tree that the Million Song subset comes in:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import os, sys\n",
      "\n",
      "dir_tree = '/Users/sebastian/Desktop/MillionSongSubset/'\n",
      "\n",
      "for dir_path, dir_names, file_names in os.walk(dir_tree):\n",
      "    for file_name in file_names:\n",
      "        try:\n",
      "            os.rename(os.path.join(dir_path, file_name), os.path.join(dir_tree, file_name))\n",
      "        except OSError:\n",
      "            print (\"Could not move %s \" % os.join(dir_path, file_name))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 3
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br>\n",
      "<br>"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Compiling a Title-Artist Table"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "[[back to top](#Sections)]"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now, we create a a pandas DataFrame with the three feature columns `file`, `artist`, and `title`, where the `artist` and `title` are our input for the lyrics search, and the `file` name is merely serves for identification purposes."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import os\n",
      "import pandas as pd\n",
      "\n",
      "def make_artist_table(base):\n",
      "\n",
      "# Get file names\n",
      "\n",
      "    files = [os.path.join(base,fn) for fn in os.listdir(base) if fn.endswith('.h5')]\n",
      "    data = {'file':[], 'artist':[], 'title':[]}\n",
      "\n",
      "    # Add artist and title data to dictionary\n",
      "    for f in files:\n",
      "        store = pd.HDFStore(f)\n",
      "        title = store.root.metadata.songs.cols.title[0]\n",
      "        artist = store.root.metadata.songs.cols.artist_name[0]\n",
      "        data['file'].append(os.path.basename(f))\n",
      "        data['title'].append(title.decode(\"utf-8\"))\n",
      "        data['artist'].append(artist.decode(\"utf-8\"))\n",
      "        store.close()\n",
      "    \n",
      "    # Convert dictionary to pandas DataFrame\n",
      "    df = pd.DataFrame.from_dict(data, orient='columns')\n",
      "    df = df[['file', 'artist', 'title']]\n",
      "    return df"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 16
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "base = '/Users/sebastian/Desktop/MillionSongSubset/'\n",
      "df = make_artist_table(base)\n",
      "\n",
      "df.tail()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>file</th>\n",
        "      <th>artist</th>\n",
        "      <th>title</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>9996 </th>\n",
        "      <td> TRBIJMU12903CF892B.h5</td>\n",
        "      <td>                Moonspell</td>\n",
        "      <td>                   The Hanged Man</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>9997 </th>\n",
        "      <td> TRBIJNF128F14815A7.h5</td>\n",
        "      <td>           Danny Williams</td>\n",
        "      <td> The Wonderful World Of The Young</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>9998 </th>\n",
        "      <td> TRBIJNK128F93093EC.h5</td>\n",
        "      <td>            Winston Reedy</td>\n",
        "      <td>                  Sentimental Man</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>9999 </th>\n",
        "      <td> TRBIJRN128F425F3DD.h5</td>\n",
        "      <td> Myrick \"Freeze\" Guillory</td>\n",
        "      <td>                Zydeco In D-Minor</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>10000</th>\n",
        "      <td> TRBIJYB128F14AE326.h5</td>\n",
        "      <td>      Seventh Day Slumber</td>\n",
        "      <td>                   Shattered Life</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 17,
       "text": [
        "                        file                    artist  \\\n",
        "9996   TRBIJMU12903CF892B.h5                 Moonspell   \n",
        "9997   TRBIJNF128F14815A7.h5            Danny Williams   \n",
        "9998   TRBIJNK128F93093EC.h5             Winston Reedy   \n",
        "9999   TRBIJRN128F425F3DD.h5  Myrick \"Freeze\" Guillory   \n",
        "10000  TRBIJYB128F14AE326.h5       Seventh Day Slumber   \n",
        "\n",
        "                                  title  \n",
        "9996                     The Hanged Man  \n",
        "9997   The Wonderful World Of The Young  \n",
        "9998                    Sentimental Man  \n",
        "9999                  Zydeco In D-Minor  \n",
        "10000                    Shattered Life  "
       ]
      }
     ],
     "prompt_number": 17
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br>\n",
      "<br>"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Downloading Lyrics"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "[[back to top](#Sections)]"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "First, we add a new column for the lyrics to our DataFrame."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df['lyrics'] = pd.Series('', index=df.index)\n",
      "df.tail()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>file</th>\n",
        "      <th>artist</th>\n",
        "      <th>title</th>\n",
        "      <th>lyrics</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>9996 </th>\n",
        "      <td> TRBIJMU12903CF892B.h5</td>\n",
        "      <td>                Moonspell</td>\n",
        "      <td>                   The Hanged Man</td>\n",
        "      <td> </td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>9997 </th>\n",
        "      <td> TRBIJNF128F14815A7.h5</td>\n",
        "      <td>           Danny Williams</td>\n",
        "      <td> The Wonderful World Of The Young</td>\n",
        "      <td> </td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>9998 </th>\n",
        "      <td> TRBIJNK128F93093EC.h5</td>\n",
        "      <td>            Winston Reedy</td>\n",
        "      <td>                  Sentimental Man</td>\n",
        "      <td> </td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>9999 </th>\n",
        "      <td> TRBIJRN128F425F3DD.h5</td>\n",
        "      <td> Myrick \"Freeze\" Guillory</td>\n",
        "      <td>                Zydeco In D-Minor</td>\n",
        "      <td> </td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>10000</th>\n",
        "      <td> TRBIJYB128F14AE326.h5</td>\n",
        "      <td>      Seventh Day Slumber</td>\n",
        "      <td>                   Shattered Life</td>\n",
        "      <td> </td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 20,
       "text": [
        "                        file                    artist  \\\n",
        "9996   TRBIJMU12903CF892B.h5                 Moonspell   \n",
        "9997   TRBIJNF128F14815A7.h5            Danny Williams   \n",
        "9998   TRBIJNK128F93093EC.h5             Winston Reedy   \n",
        "9999   TRBIJRN128F425F3DD.h5  Myrick \"Freeze\" Guillory   \n",
        "10000  TRBIJYB128F14AE326.h5       Seventh Day Slumber   \n",
        "\n",
        "                                  title lyrics  \n",
        "9996                     The Hanged Man         \n",
        "9997   The Wonderful World Of The Young         \n",
        "9998                    Sentimental Man         \n",
        "9999                  Zydeco In D-Minor         \n",
        "10000                    Shattered Life         "
       ]
      }
     ],
     "prompt_number": 20
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Then, we use the following code to download the song lyrics from [LyricWikia](http://lyrics.wikia.com/) based on the artist and title names in the pandas DataFrame."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Sebastian Raschka, 2014\n",
      "# \n",
      "# Script to download lyrics from http://lyrics.wikia.com/\n",
      "\n",
      "import urllib\n",
      "import lxml.html\n",
      "\n",
      "class Song(object):\n",
      "    def __init__(self, artist, title):\n",
      "        self.artist = self.__format_str(artist)\n",
      "        self.title = self.__format_str(title)\n",
      "        self.url = None\n",
      "        self.lyric = None\n",
      "        \n",
      "    def __format_str(self, s):\n",
      "        # remove paranthesis and contents\n",
      "        s = s.strip()\n",
      "        try:\n",
      "            # strip accent\n",
      "            s = ''.join(c for c in unicodedata.normalize('NFD', s)\n",
      "                         if unicodedata.category(c) != 'Mn')\n",
      "        except:\n",
      "            pass\n",
      "        s = s.title()\n",
      "        return s\n",
      "        \n",
      "    def __quote(self, s):\n",
      "         return urllib.parse.quote(s.replace(' ', '_'))\n",
      "\n",
      "    def __make_url(self):\n",
      "        artist = self.__quote(self.artist)\n",
      "        title = self.__quote(self.title)\n",
      "        artist_title = '%s:%s' %(artist, title)\n",
      "        url = 'http://lyrics.wikia.com/' + artist_title\n",
      "        self.url = url\n",
      "        \n",
      "    def update(self, artist=None, title=None):\n",
      "        if artist:\n",
      "            self.artist = self.__format_str(artist)\n",
      "        if title:\n",
      "            self.title = self.__format_str(title)\n",
      "        \n",
      "    def lyricwikia(self):\n",
      "        self.__make_url()\n",
      "        try:\n",
      "            doc = lxml.html.parse(self.url)\n",
      "            lyricbox = doc.getroot().cssselect('.lyricbox')[0]\n",
      "        except (IOError, IndexError) as e:\n",
      "            self.lyric = ''\n",
      "            return self.lyric\n",
      "        lyrics = []\n",
      "\n",
      "        for node in lyricbox:\n",
      "            if node.tag == 'br':\n",
      "                lyrics.append('\\n')\n",
      "            if node.tail is not None:\n",
      "                lyrics.append(node.tail)\n",
      "        self.lyric =  \"\".join(lyrics).strip()    \n",
      "        return self.lyric"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 24
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "If this script doesn't work for you, you can find some alternatives to download lyrics from other websites in my [datacollect repository](https://github.com/rasbt/datacollect/tree/master/collect_lyrics)."
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Example:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "song = Song(artist='John Mellencamp', title='Jack and Diane')\n",
      "lyr = song.lyricwikia()\n",
      "print(lyr)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "A little ditty about Jack and Diane\n",
        "Two American kids growin' up in the heartland\n",
        "Jackie gonna be a football star\n",
        "Diane's a debutante, backseat of Jackie's car\n",
        "\n",
        "Suckin' on a chili dog outside the Tastee-Freez\n",
        "Diane's sittin' on Jackie's lap\n",
        "He's got his hands between her knees\n",
        "Jackie say, \"Hey Diane, let's run off behind the shady trees\n",
        "Dribble off those Bobbie Brooks, let me do what I please.\"\n",
        "And say uh\n",
        "\n",
        "Oh yeah, life goes on\n",
        "Long after the thrill of livin' is gone, they say uh\n",
        "Oh yeah, life goes on\n",
        "Long after the thrill of livin' is gone, they walk on\n",
        "\n",
        "Jackie sits back, collects his thoughts for the moment\n",
        "Scratches his head and does his best James Dean\n",
        "\"Well then there Diane, we oughta run off to the city.\"\n",
        "Diane says, \"Baby, you ain't missin' nothing.\"\n",
        "And Jackie say uh\n",
        "\n",
        "Oh yeah, life goes on\n",
        "Long after the thrill of livin' is gone\n",
        "Oh yeah, they say life goes on\n",
        "Long after the thrill of livin' is gone\n",
        "\n",
        "Gonna let it rock\n",
        "Let it roll\n",
        "Let the Bible Belt come and save my soul\n",
        "Hold on to sixteen as long as you can\n",
        "Changes come around real soon\n",
        "Make us women and men\n",
        "\n",
        "Oh yeah, life goes on\n",
        "Long after the thrill of livin' is gone\n",
        "Oh yeah, they say life goes on\n",
        "Long after the thrill of livin' is gone\n",
        "\n",
        "A little ditty about Jack and Diane\n",
        "Two American kids doin' the best they can\n"
       ]
      }
     ],
     "prompt_number": 25
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br>\n",
      "<br>"
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Adding lyrics to the DataFrame"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "[[back to top](#Sections)]"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import pyprind"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 26
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "pbar = pyprind.ProgBar(df.shape[0])\n",
      "for row_id in df.index:\n",
      "    song = Song(artist=df.loc[row_id]['artist'], title=df.loc[row_id]['title'])\n",
      "    lyr = song.lyricwikia()\n",
      "    df.loc[row_id,'lyrics'] = lyr\n",
      "    pbar.update()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "0%                          100%\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "[                              ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#                             ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 3266.223 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[##                            ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 3393.513 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[###                           ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 3319.358 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[####                          ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 3210.384 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#####                         ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 3085.919 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[######                        ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 2986.598 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#######                       ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 2883.762 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[########                      ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 2757.830 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#########                     ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 2624.769 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[##########                    ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 2490.598 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[###########                   ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 2371.431 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[############                  ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 2247.478 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#############                 ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 2132.358 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[##############                ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 2008.513 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[###############               ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 1880.316 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[################              ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 1751.628 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#################             ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 1738.960 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[##################            ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 1714.301 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[###################           ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 1662.368 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[####################          ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 1572.287 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#####################         ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 1442.812 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[######################        ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 1314.451 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#######################       ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 1169.392 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[########################      ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 1034.134 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#########################     ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 878.730 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[##########################    ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 718.852 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[###########################   ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 548.439 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[############################  ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 370.707 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[############################# ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 188.025 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[##############################]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 0.000 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[##############################]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 0.000 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\n",
        "Total time elapsed: 5709.172 sec\n"
       ]
      }
     ],
     "prompt_number": 27
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print('downloaded Lyrics for %s songs' %sum(df.lyrics!=''))\n",
      "df.head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "downloaded Lyrics for 3142 songs\n"
       ]
      },
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>file</th>\n",
        "      <th>artist</th>\n",
        "      <th>title</th>\n",
        "      <th>lyrics</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> subset_msd_summary_file.h5</td>\n",
        "      <td>         Mastodon</td>\n",
        "      <td> Deep Sea Creature</td>\n",
        "      <td> Knowing right, learning wrong\\nWhat you're fee...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td>      TRAAAAW128F429D538.h5</td>\n",
        "      <td>           Casual</td>\n",
        "      <td>  I Didn't Mean To</td>\n",
        "      <td> Verse One:\\n\\nAlright I might\\nHave had a litt...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td>      TRAAABD128F429CF47.h5</td>\n",
        "      <td>     The Box Tops</td>\n",
        "      <td>         Soul Deep</td>\n",
        "      <td> Darling, I don't know much\\nBut I know I love ...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td>      TRAAADZ128F9348C2E.h5</td>\n",
        "      <td> Sonora Santanera</td>\n",
        "      <td>   Amor De Cabaret</td>\n",
        "      <td>                                                  </td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td>      TRAAAEF128F4273421.h5</td>\n",
        "      <td>         Adam Ant</td>\n",
        "      <td>   Something Girls</td>\n",
        "      <td> Adam Ant/Marco Pirroni\\nEvery girl is a someth...</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 28,
       "text": [
        "                         file            artist              title  \\\n",
        "0  subset_msd_summary_file.h5          Mastodon  Deep Sea Creature   \n",
        "1       TRAAAAW128F429D538.h5            Casual   I Didn't Mean To   \n",
        "2       TRAAABD128F429CF47.h5      The Box Tops          Soul Deep   \n",
        "3       TRAAADZ128F9348C2E.h5  Sonora Santanera    Amor De Cabaret   \n",
        "4       TRAAAEF128F4273421.h5          Adam Ant    Something Girls   \n",
        "\n",
        "                                              lyrics  \n",
        "0  Knowing right, learning wrong\\nWhat you're fee...  \n",
        "1  Verse One:\\n\\nAlright I might\\nHave had a litt...  \n",
        "2  Darling, I don't know much\\nBut I know I love ...  \n",
        "3                                                     \n",
        "4  Adam Ant/Marco Pirroni\\nEvery girl is a someth...  "
       ]
      }
     ],
     "prompt_number": 28
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.to_csv('/Users/sebastian/Desktop/df_lyr_backup.csv')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 29
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Remove Rows where Lyrics are not available"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "[[back to top](#Sections)]"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "If lyrics were not available, this can be due to one of the following reasons\n",
      "- the URL was not parsed correctly\n",
      "- the song does not exist in the LyricWikia database\n",
      "- the song is an instrumental song"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df = df[df.lyrics!='']"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 30
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br>\n",
      "<br>"
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Language Filter"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "[[back to top](#Sections)]"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now, we remove all lyrics that are not in English. Basically, we say that if the song contains more English than non-English words (> 50%), then it is an English song. We use this relatively high cutoff-ratio of 0.5, since a songtext likely contains also names and other special words that are not part of a common English dictionary."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Example: "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import nltk\n",
      "\n",
      "def eng_ratio(text):\n",
      "    ''' Returns the ratio of non-English to English words from a text '''\n",
      "\n",
      "    english_vocab = set(w.lower() for w in nltk.corpus.words.words()) \n",
      "    text_vocab = set(w.lower() for w in text.split() if w.lower().isalpha()) \n",
      "    unusual = text_vocab.difference(english_vocab)\n",
      "    diff = len(unusual)/len(text_vocab)\n",
      "    return diff\n",
      "    \n",
      "text = 'This is a test fahrrad'\n",
      "\n",
      "print(eng_ratio(text))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.2\n"
       ]
      }
     ],
     "prompt_number": 32
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br>\n",
      "<br>"
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Remove all non-English lyrics"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "[[back to top](#Sections)]"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "before = df.shape[0]\n",
      "for row_id in df.index:\n",
      "    text = df.loc[row_id]['lyrics']\n",
      "    diff = eng_ratio(text)\n",
      "    if diff >= 0.5:\n",
      "        df = df[df.index != row_id]\n",
      "after = df.shape[0]\n",
      "rem = before - after\n",
      "print('%s have been removed.' %rem)\n",
      "print('%s songs remain in the dataset.' %after)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "372 have been removed.\n",
        "2770 songs remain in the dataset.\n"
       ]
      }
     ],
     "prompt_number": 33
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.to_csv('/Users/sebastian/Desktop/df_lyr_backup2.csv', index=False)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 34
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br>\n",
      "<br>"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Create a filtered dataset"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "[[back to top](#Sections)]"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now, we copy all songs for which the lyrics exist to a new directory."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import os\n",
      "import shutil\n",
      "\n",
      "new_dir = '/Users/sebastian/Desktop/h5_filtered/'\n",
      "if not os.path.exists(new_dir):\n",
      "    os.mkdir(new_dir)\n",
      "\n",
      "h1 = '/Users/sebastian/Desktop/MillionSongSubset/'\n",
      "filepaths1 = [os.path.join(h1, f) for f in os.listdir(h1) if f.endswith('.h5')]\n",
      "\n",
      "filepaths = filepaths1\n",
      "\n",
      "for f in filepaths:\n",
      "    base = os.path.basename(f)\n",
      "    if base in df.file.values:\n",
      "        target = os.path.join(new_dir, base)\n",
      "        shutil.copyfile(f, target)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 35
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br>\n",
      "<br>"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Randomly partition the dataset into separate training and validation sets"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "[[back to top](#Sections)]"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In this step, the dataset is reduced to a \"reasonable\" amount for the manual labeling step: 1000 songs for the training dataset and 200 songs for the validation dataset."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import random\n",
      "\n",
      "h2 = '/Users/sebastian/Desktop/h5_filtered/'\n",
      "filepaths2 = [os.path.join(h2, f) for f in os.listdir(h2) if f.endswith('.h5')]\n",
      "random.shuffle(filepaths2)\n",
      "\n",
      "train_dir = '../../dataset/training/h5_train/'\n",
      "valid_dir = '../../dataset/validation/h5_valid/'\n",
      "aux_dir = '../../dataset/auxiliary/h5_aux/'\n",
      "\n",
      "for d in (train_dir, valid_dir, aux_dir):\n",
      "    if not os.path.exists(d):\n",
      "        os.mkdir(d)\n",
      "\n",
      "for f in filepaths2[:1000]:\n",
      "    base = os.path.basename(f)\n",
      "    target = os.path.join(train_dir, base)\n",
      "    shutil.copyfile(f, target)\n",
      "  \n",
      "for i in filepaths2[1000:1200]:\n",
      "    base = os.path.basename(f)\n",
      "    target = os.path.join(valid_dir, base)\n",
      "    shutil.copyfile(f, target)\n",
      "    \n",
      "for i in filepaths2[1200:]:\n",
      "    base = os.path.basename(f)\n",
      "    target = os.path.join(aux_dir, base)\n",
      "    shutil.copyfile(f, target)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 38
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br>\n",
      "<br>"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Make new CSV tables for the Training and Validation dataset"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "[[back to top](#Sections)]"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_train = make_artist_table('../../dataset/training/h5_train')\n",
      "df_train['lyrics'] = pd.Series('', index=df_train.index)\n",
      "\n",
      "pbar = pp.ProgBar(df_train.shape[0])\n",
      "for row_id in df_train.index:\n",
      "    song = Song(artist=df_train.loc[row_id]['artist'], title=df_train.loc[row_id]['title'])\n",
      "    lyr = song.lyricwikia()\n",
      "    df_train.loc[row_id]['lyrics'] = lyr\n",
      "    pbar.update()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "0%                          100%\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "[                              ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#                             ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 43.546 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[##                            ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 47.001 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[###                           ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 45.831 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[####                          ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 42.677 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#####                         ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 46.097 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[######                        ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 44.529 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#######                       ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 41.693 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[########                      ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 41.735 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#########                     ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 38.929 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[##########                    ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 36.802 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[###########                   ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 37.586 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[############                  ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 35.876 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#############                 ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 34.957 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[##############                ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 32.475 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[###############               ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 29.877 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[################              ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 28.157 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#################             ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 26.522 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[##################            ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 25.121 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[###################           ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 22.755 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[####################          ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 21.051 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#####################         ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 18.714 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[######################        ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 16.362 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#######################       ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 14.150 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[########################      ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 12.033 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#########################     ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 9.884 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[##########################    ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 8.029 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[###########################   ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 6.031 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[############################  ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 4.007 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[############################# ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 2.028 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[##############################]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\n",
        "Total time elapsed: 60.924 sec\n"
       ]
      }
     ],
     "prompt_number": 88
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_valid = make_artist_table('../../dataset/validation/h5_validate')\n",
      "df_valid['lyrics'] = pd.Series('', index=df_valid.index)\n",
      "\n",
      "pbar = pp.ProgBar(df_valid.shape[0])\n",
      "for row_id in df_valid.index:\n",
      "    song = Song(artist=df_valid.loc[row_id]['artist'], title=df_valid.loc[row_id]['title'])\n",
      "    lyr = song.lyricwikia()\n",
      "    df_valid.loc[row_id]['lyrics'] = lyr\n",
      "    pbar.update()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "0%                          100%\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "[                              ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#                             ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 4.313 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[##                            ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 4.090 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[###                           ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 3.963 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[####                          ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 3.769 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#####                         ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 4.998 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[######                        ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 4.637 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#######                       ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 4.226 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[########                      ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 3.966 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#########                     ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 3.727 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[##########                    ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 3.917 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[###########                   ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 3.662 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[############                  ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 3.425 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#############                 ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 3.124 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[##############                ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 2.929 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[###############               ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 2.728 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[################              ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 2.651 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#################             ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 2.553 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[##################            ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 2.346 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[###################           ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 2.254 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[####################          ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 2.113 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#####################         ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 1.897 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[######################        ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 1.700 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#######################       ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 1.484 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[########################      ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 1.277 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[#########################     ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 1.007 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[##########################    ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 0.809 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[###########################   ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 0.616 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[############################  ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 0.366 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[############################# ]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        " | ETA[sec]: 0.181 "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\r",
        "[##############################]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "\n",
        "Total time elapsed: 5.997 sec\n"
       ]
      }
     ],
     "prompt_number": 89
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_train.to_csv('../../dataset/training/train_lyrics_1000.csv')\n",
      "df_valid.to_csv('../../dataset/validation/valid_lyrics_200.csv')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 90
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br>\n",
      "<br>"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Adding year information"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "[[back to top](#Sections)]"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import pandas as pd"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 39
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df = pd.read_csv('../../dataset/training/train_lyrics_1000.csv')\n",
      "df.head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>file</th>\n",
        "      <th>artist</th>\n",
        "      <th>title</th>\n",
        "      <th>lyrics</th>\n",
        "      <th>mood</th>\n",
        "      <th>year</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> TRAAAAW128F429D538.h5</td>\n",
        "      <td>        Casual</td>\n",
        "      <td>        I Didn't Mean To</td>\n",
        "      <td> Verse One:\\n\\nAlright I might\\nHave had a litt...</td>\n",
        "      <td>   sad</td>\n",
        "      <td> 1994</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> TRAAAEF128F4273421.h5</td>\n",
        "      <td>      Adam Ant</td>\n",
        "      <td>         Something Girls</td>\n",
        "      <td> Adam Ant/Marco Pirroni\\nEvery girl is a someth...</td>\n",
        "      <td> happy</td>\n",
        "      <td> 1982</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> TRAAAFD128F92F423A.h5</td>\n",
        "      <td>           Gob</td>\n",
        "      <td>          Face the Ashes</td>\n",
        "      <td> I've just erased it's been a while, I've got a...</td>\n",
        "      <td>   sad</td>\n",
        "      <td> 2007</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> TRAABJV128F1460C49.h5</td>\n",
        "      <td> Lionel Richie</td>\n",
        "      <td> Tonight Will Be Alright</td>\n",
        "      <td> Little darling \\nWhere you've been so long \\nI...</td>\n",
        "      <td> happy</td>\n",
        "      <td> 1986</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> TRAABLR128F423B7E3.h5</td>\n",
        "      <td>    Blue Rodeo</td>\n",
        "      <td>                Floating</td>\n",
        "      <td> Lead Vocal by Greg\\n\\nWell, these late night c...</td>\n",
        "      <td>   sad</td>\n",
        "      <td> 1987</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 46,
       "text": [
        "                    file         artist                    title  \\\n",
        "0  TRAAAAW128F429D538.h5         Casual         I Didn't Mean To   \n",
        "1  TRAAAEF128F4273421.h5       Adam Ant          Something Girls   \n",
        "2  TRAAAFD128F92F423A.h5            Gob           Face the Ashes   \n",
        "3  TRAABJV128F1460C49.h5  Lionel Richie  Tonight Will Be Alright   \n",
        "4  TRAABLR128F423B7E3.h5     Blue Rodeo                 Floating   \n",
        "\n",
        "                                              lyrics   mood  year  \n",
        "0  Verse One:\\n\\nAlright I might\\nHave had a litt...    sad  1994  \n",
        "1  Adam Ant/Marco Pirroni\\nEvery girl is a someth...  happy  1982  \n",
        "2  I've just erased it's been a while, I've got a...    sad  2007  \n",
        "3  Little darling \\nWhere you've been so long \\nI...  happy  1986  \n",
        "4  Lead Vocal by Greg\\n\\nWell, these late night c...    sad  1987  "
       ]
      }
     ],
     "prompt_number": 46
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import os\n",
      "\n",
      "df['year'] = pd.Series('', index=df.index)\n",
      "\n",
      "base = '../../dataset/training/h5_train/'\n",
      "files = [os.path.join(base,fn) for fn in os.listdir(base) if fn.endswith('.h5')]\n",
      "for row_id in df.index:\n",
      "    filename = df.loc[row_id]['file']\n",
      "    filepath = os.path.join(base,filename)\n",
      "    store = pd.HDFStore(filepath)\n",
      "    year = store.root.musicbrainz.songs.cols.year[0]\n",
      "    df.loc[row_id]['year'] = year"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 48
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df[['file', 'artist', 'title','lyrics','year']].tail()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>file</th>\n",
        "      <th>artist</th>\n",
        "      <th>title</th>\n",
        "      <th>lyrics</th>\n",
        "      <th>year</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>995</th>\n",
        "      <td> TRBIGRY128F42597B3.h5</td>\n",
        "      <td>                Sade</td>\n",
        "      <td>          All About Our Love</td>\n",
        "      <td> Its all about our love\\nSo shall it be forever...</td>\n",
        "      <td> 2000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>996</th>\n",
        "      <td> TRBIIEU128F9307C88.h5</td>\n",
        "      <td>     New Found Glory</td>\n",
        "      <td> Don't Let Her Pull You Down</td>\n",
        "      <td> It's time that I rain on your parade\\nWatch as...</td>\n",
        "      <td> 2009</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>997</th>\n",
        "      <td> TRBIIJY12903CE4755.h5</td>\n",
        "      <td>      Mindy McCready</td>\n",
        "      <td>         Ten Thousand Angels</td>\n",
        "      <td> Speakin of the devil\\nLook who just walked in\\...</td>\n",
        "      <td> 1996</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>998</th>\n",
        "      <td> TRBIIOT128F423C594.h5</td>\n",
        "      <td>        Joy Division</td>\n",
        "      <td>              Leaders Of Men</td>\n",
        "      <td> Born from some mother's womb\\nJust like any ot...</td>\n",
        "      <td> 1978</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>999</th>\n",
        "      <td> TRBIJYB128F14AE326.h5</td>\n",
        "      <td> Seventh Day Slumber</td>\n",
        "      <td>              Shattered Life</td>\n",
        "      <td> This wanting more from me is tearing me, it's ...</td>\n",
        "      <td> 2005</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 49,
       "text": [
        "                      file               artist                        title  \\\n",
        "995  TRBIGRY128F42597B3.h5                 Sade           All About Our Love   \n",
        "996  TRBIIEU128F9307C88.h5      New Found Glory  Don't Let Her Pull You Down   \n",
        "997  TRBIIJY12903CE4755.h5       Mindy McCready          Ten Thousand Angels   \n",
        "998  TRBIIOT128F423C594.h5         Joy Division               Leaders Of Men   \n",
        "999  TRBIJYB128F14AE326.h5  Seventh Day Slumber               Shattered Life   \n",
        "\n",
        "                                                lyrics  year  \n",
        "995  Its all about our love\\nSo shall it be forever...  2000  \n",
        "996  It's time that I rain on your parade\\nWatch as...  2009  \n",
        "997  Speakin of the devil\\nLook who just walked in\\...  1996  \n",
        "998  Born from some mother's womb\\nJust like any ot...  1978  \n",
        "999  This wanting more from me is tearing me, it's ...  2005  "
       ]
      }
     ],
     "prompt_number": 49
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.to_csv('../../dataset/training/train_lyrics_1000.csv', index=False)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 22
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Missing year labels were manually added based on information from http://www.allmusic.com"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}