Meetup Predict song composition with Spotify data.ipynb

{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Welcome to this months Central London Data Science Meetup! \n\nIf you've ever read a data science related blog before, you'll probably have read either:\n* 'AI is the new electricity' (I'm looking at you Andrew Ng)\n* '90% of a data scientist time is spent sourcing and then cleaning the data'\n\nIn tonights notebook we will be delving into predicting the genre of songs using spotify data. From this you'll see that a data scientist really does spend a lot of time collecting and wrangling data.\n\n**Note** The code you will be working with is commented out using `#`. To uncomment and run code highlight it and press `command` + `/`\n\nSo without further ado lets get started!"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport re\nimport json\nimport numpy as np \nimport pandas as pd \nfrom sklearn.model_selection import train_test_split\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.metrics import f1_score\nfrom sklearn.metrics import recall_score\nfrom sklearn.metrics import precision_score\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as mpatches","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Sourcing data online\n\nSourcing data for a project can be tricky, one option is to scower the internet looking for datasets to download. Another option is to use an Application Programming Interface (API).\n\n\n### APIs\nAn API allows different applications to share data between each another. By calling an API we gain access to data held on a server.\n\n### So why are these good for a data scientist?\n\nSay you wanted to keep up-to-date the National Leage promotion race (abosulte nail biter). You could search for the hashtag #LeytonOrient and copy and paste each of the tweets into a document. This will take forever.\n\nThe next best thing would be to email twitter and ask for a dataset of the twitter stream, but again this would waste even more peoples peoples time. \n\nInstead, companies create these access points known as APIs that let you query available data. This reduces up the time needed to collect the data you need for your project. Great!\n\n### What to expect from an API\nInformation returned from an API can come in a couple of formats, but the one that we'll be using today is JavaScript Object Notation- JSON. \n\nJSON is ubiquitous throughout the web. It's human readable, lightweight and can be interpreted by a tonne of languages, including python. \n"},{"metadata":{},"cell_type":"markdown","source":"# Sourcing our data\nSo now that we know what an API is and what to expect from it, the next hurdle is understanding how to get it and what it'll contain. \n\nWith this in mind let's jump over to the spotify API documentation- https://developer.spotify.com/documentation/web-api/reference/\n\nIn order to source and naviagate the data needed for a project you will need to understand what each key and value represent, so get used to jumping between your code and API documentation.\n\nHave a look for yourself across all the documentation- (It might give you a bit of inspiration for your own project).\n\nFor our project we will be living on these two pages: \n\n* https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-analysis/\n\n* https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/\n\n\nThese pages are incredibly useful- we get a description of what the data represent, but a map of how to access it.\n\nIn the interest of time, (and also ensuring we don't bring CodeNodes internet crashing down). All the data for tonights excercise is stored locally in the environment. \n\nHowever, at the end of the exercise we'll have a mini tutorial on how to call an API by yourself.\n\nLet's jump into a single songs audio analysis.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"single_song_path = '../input/singlesong/single_song.json'\nwith open(single_song_path, 'rb') as f:\n    song_json = json.load(f)\n# Uncomment this     \n# print(song_json)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Kinda gross, right? Let's make this a bit easier on the eyes.\n\nTry using the `json.dumps()` function."},{"metadata":{"trusted":true},"cell_type":"code","source":"# print(json.dumps(None, indent=4, sort_keys=True))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Still a bit intimadating though...\n\nWhen we load in a json object- python interprets it as a dictionary. So lets use some python 101 to navigate the struture.\n\nA dictionary consists of a collection of key-value pairs. Each key-value pair maps the key to its associated value.\n\nYou can build a dictionary by wrapping a `key` joined by a colon `:` to an asscoicated `value` using curly braces `{}`.\n\nLet's build a simple dictionary"},{"metadata":{"trusted":true},"cell_type":"code","source":"dictionary = {'dog': 'woof',\n              'cat': 'meow',\n              'lazer': 'zapppp',\n              'list_of_things': ['a', 3, dict()],\n              'numbers': 10012}\ndictionary","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we have a dictionary built of key-value pairs. You can access each value stored in the dictionary using the keys."},{"metadata":{"trusted":true},"cell_type":"code","source":"dictionary.keys()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here's how we would access the value associated with the key `dog`"},{"metadata":{"trusted":true},"cell_type":"code","source":"dictionary['dog']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now the super cool thing about dictionary object is that you can store a tonne of inforation in a variety of formats. Lets check out `list_of_things`."},{"metadata":{"trusted":true},"cell_type":"code","source":"dictionary['list_of_things']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It returns a `list`, we can either associate this value with a variable and work with it elsewhere like so...."},{"metadata":{"trusted":true},"cell_type":"code","source":"list_of_things = dictionary['list_of_things']\nlist_of_things[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Or access the list object directly..."},{"metadata":{"trusted":true},"cell_type":"code","source":"dictionary['list_of_things'][0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Cool! So now we have the basics of how to naviagate our way across a dictionary, let's put it to use.\n\nHere are the keys to our audio analysis"},{"metadata":{"trusted":true},"cell_type":"code","source":"song_json.keys()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets look at the meta data values..."},{"metadata":{"trusted":true},"cell_type":"code","source":"song_json['meta']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"OK, this stuff is a bit boring, let's get into the data that we'll be using for our model.\nHead on over to the [doumentation](https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-analysis/) and check out what `segments` represents.\n\n**\"Audio segments attempts to subdivide a song into many segments, with each segment containing a roughly consitent sound throughout its duration.\"**"},{"metadata":{"trusted":true},"cell_type":"code","source":"song_json['segments']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Segments returns a list of dictionaries, each of these dictionaries contains string data and lists.\n\nGreat, our dictionary contains a list of other dictionaries, that isn't confusing at all...\n\nWhat we want is the timbre values.\n\n**Timbre is the quality of a musical note or sound that distinguishes different types of musical instruments, or voices. Timbre vectors are best used in comparison with each other.**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Access the first element (dictionary) in the list\nfirst_segment = song_json['segments'][0]\nfirst_segment","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Then access the timbre values\nfirst_segment['timbre']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can combine the two lines and pass it through a for loop to get all the timbre values across each segment for this song...\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# song_timbres = []\n# for segment in song_json[None]:\n#     song_timbres.append(segment[None])\n# song_timbres","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Neat, we've just navigated and stored the relevant data for our model.\n\nHowever, this is only one song, we'll need to get a few more samples to play with.\n\nWe've collected about 50 songs from some well known genres:\n* jazz\n* blues\n* funk,\n* metal\n* classical\n* hiphop\n* pop\n* electronic\n\nAnd streamlined the data to include the information we'll need for the rest of the exercise.\n\nLet's use our knowledge of dictionary naviagation to wrangle the data into a Dataframe so we can model it.\n\nWithin the `musicdata` folder we have a few JSON files, each one contains songs and their segment data from a single genre."},{"metadata":{"trusted":true},"cell_type":"code","source":"training_data_path = '../input/musicdata/'\nos.listdir(training_data_path)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We want to combine all these data sources into one big dataframe.\n\nEach one has the same structure, let's look at an example fo how to get the data."},{"metadata":{"trusted":true},"cell_type":"code","source":"with open(os.path.join(training_data_path,'hiphop.json'), 'rb') as f:\n    hiphop = json.load(f)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Each key is a unique identifer for a song known as a URI\nhiphop.keys()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hiphop['spotify:track:3MnwLa9KRUiv2gNFtWPvib'].keys()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hiphop['spotify:track:3MnwLa9KRUiv2gNFtWPvib']['artist']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hiphop['spotify:track:3MnwLa9KRUiv2gNFtWPvib']['song']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hiphop['spotify:track:3MnwLa9KRUiv2gNFtWPvib']['meta'].keys()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hiphop['spotify:track:3MnwLa9KRUiv2gNFtWPvib']['meta']['segments'][0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hiphop['spotify:track:3MnwLa9KRUiv2gNFtWPvib']['meta']['segments'][0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To give you an idea on the end goal of this wrangling- Check out how the dataframe will look like later...\n\n![](https://github.com/Blair-Young/PredictingMusicGenresFromSpotifyData/blob/master/images/Screen%20Shot%202019-04-21%20at%2015.43.03.png?raw=true)"},{"metadata":{},"cell_type":"markdown","source":"Lets create a couple of functions to extract this data."},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_song_name(json_data, song_uri):\n    '''Returns song name from song URI key\n     Args:\n     * json_data- (JSON) \n     * song_uri- (str) URI\n     \n     Return\n     * (str)- Song name\n     '''\n    return\n\ndef get_artist_name(json_data, song_uri):\n    '''Returns Artist name from song URI key\n     Args:\n     * json_data- (JSON) \n     * song_uri- (str) URI\n     \n     Return:\n     * (str)- Artist name\n     '''\n\n    return\n\ndef get_timbre_values(json_data, song_uri):\n    '''Returns timbre values from a song\n    Args:\n    * json_data- (JSON) \n    * song_uri- (str) URI \n    \n    Return:\n    * (list) Each element is a list of timbre values\n    '''\n    timbre_data = []\n    for segment in json_data[None][None][None]:\n        timbre_data.append(segment[None])\n    return timbre_data\n\ndef get_segment_start_time(json_data, song_uri):\n    '''Returns start times of segments from a song\n    Args:\n    * json_data- (JSON) \n    * song_uri- (str) URI \n    \n    Return:\n    * (list) Each element is float representing time in milliseconds\n    '''\n    start_times = []\n    for segment in json_data[None][None][None]:\n        start_times.append(segment[None])\n    return start_times\n\ndef get_segment_duration(json_data, song_uri):\n    '''Returns duration of segments from a song\n    Args:\n    * json_data- (JSON) \n    * song_uri- (str) URI \n    \n    Return:\n    * (list) Each element is float representing the duration of a segment\n    '''\n    durations = [] \n    for segment in json_data[None][None][None]:\n        durations.append(segment[None])\n    return durations","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"We have our extraction functions, lets get extracting!\n\nEverything that we return from these functions should go straight into a pandas DataFrame.\n\nFor anyone that hasn't used a pandas DataFrame, it's basically a table similiar to that of an Excel/Sheets spreadsheet.\nIf you want more information about them, check out https://pandas.pydata.org/"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_genre_data(genre_data, genre_type):\n    '''\n    Processes a JSON object of a single genre\n    Args:\n    * genre data (JSON)\n    * single genre (str) Name of genre\n    \n    Returns:\n    * pandas DataFrame containing training data and label for ML\n    '''\n    genre_dataframes = []\n    for song_uri in genre_data.keys():\n        # Extract the relevant data with our functions\n        timbres = None\n        start_times = None\n        durations = None\n        artist_name = None\n        song_name = None\n        # Create a dataframe per song\n        # We'll build the timbre parts first then add columns\n        song_df = pd.DataFrame(timbres)\n        song_df['start'] = None\n        song_df['durations'] = None\n        song_df['song_name'] = None\n        song_df['artist'] = None\n        # Remember to add the genre so we can use it for supervised learning later!\n        song_df['genre'] = None\n        # Now we need to store/append all the songs in a genre dataframe\n        genre_dataframes.append(None)\n    # Now concatenate the song dataframes into a single genre specific dataframe\n    genre_df = pd.concat(None)\n    return genre_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# get_genre_data(hiphop, 'hiphop').head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's scale it up so we can use it across all genres."},{"metadata":{"trusted":true},"cell_type":"code","source":"genre_data_path = '../input/musicdata/'\ngenre_list = os.listdir(genre_data_path)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# all_genre_list = []\n# for genre in genre_list:\n#     # Get rid of the pesky .DS_Store files with this clause\n#     if not genre.endswith('.DS_Store'):\n#         path = os.path.join(genre_data_path, genre)\n#         with open(path, 'rb') as f:\n#             genre_json = json.load(f)\n#         # Extract the genre from the file name    \n#         genre_label = genre.replace('.json', '')   \n#         # Apply our function\n#         genre_data = get_genre_data(genre_json, genre_label)\n#         all_genre_list.append(genre_data)\n\n# df = pd.concat(all_genre_list)\n# df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's tidy the name columns up.\n\nCheck out this page on what the timbre values correspond to\n\nhttps://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-analysis/#timbre\n\nFrom this we know what some of the values correspond to, but not all of them..."},{"metadata":{"trusted":true},"cell_type":"code","source":"# df.rename(columns={i: 'timbre_value_'+str(i) for i in range(0,12)}, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Rename the first three timbre values according to the documentation"},{"metadata":{"trusted":true},"cell_type":"code","source":"# df.rename(columns={'timbre_value_0':'loudness',\n#                    'timbre_value_1': 'brightness',\n#                    'timbre_value_2': 'flatness'}, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we'll wrap all this in a function. "},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_genre_df(genre_data_path):\n    genre_list = os.listdir(genre_data_path)\n    all_genre_list = []\n    for genre in genre_list:\n    # Get rid of the pesky .DS_Store files with this clause\n        if not genre.endswith(None):\n            path = os.path.join(genre_data_path, genre)\n            with open(path, 'rb') as f:\n                genre_json = json.load(f)\n            # Extract the genre from the file name    \n            genre_label = genre.replace(None, '')   \n            # Apply our function get_genre_data\n            genre_data = None\n            all_genre_list.append(None)\n\n    df = pd.concat(all_genre_list)\n    df.rename(columns={i: 'timbre_value_'+str(i) for i in range(0,12)}, inplace=True)\n    df.rename(columns={'timbre_value_0':'loudness',\n                   'timbre_value_1': 'brightness',\n                   'timbre_value_2': 'flatness'}, inplace=True)\n    return df\n\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# df = get_genre_df(genre_data_path)\n# df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Check out the last line of the documentation**\n\n* `Timbre vectors are best used in comparison with each other.`\n\nLooks like they've already been normalised for us, thanks Spotify.\n\nThis means we can get straight into the machine learning (about time!)\n\nOne more thing (I promise). We'll separate the pop music from the rest of the data, we'll try and use our model to break them down."},{"metadata":{"trusted":true},"cell_type":"code","source":"# df_pop = df[df['genre']==None]\n# df = df[df['genre']!=None]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# training_colummns = ['loudness', 'brightness', 'flatness', 'timbre_value_3',\n#                      'timbre_value_4', 'timbre_value_5', 'timbre_value_6', 'timbre_value_7',\n#                      'timbre_value_8', 'timbre_value_9', 'timbre_value_10',\n#                      'timbre_value_11']\n# X = df[training_colummns]\n# y = df['genre']\n# X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train a logistic regression model"},{"metadata":{"trusted":true},"cell_type":"code","source":"# clf = LogisticRegression(random_state=0, solver='lbfgs',\n#                          multi_class='multinomial')\n# clf.fit(X_train, y_train)\n# y_pred_log_reg = clf.predict(X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print('f1 score {}'.format(f1_score(y_test, y_pred_log_reg, average='weighted')))\n# print('recall score {}'.format(recall_score(y_test, y_pred_log_reg, average='weighted')))\n# print('precision score {}'.format(precision_score(y_test, y_pred_log_reg, average='weighted')))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"* Not amazing performance here, let's see which genre's the model is having problems with..."},{"metadata":{"trusted":true},"cell_type":"code","source":"# {key:value for key, value in zip(sorted(df['genre'].unique()), f1_score(y_test, y_pred_log_reg, average=None))}\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# log_reg_results = pd.DataFrame({'y_Actual':y_test,\n#                         'y_Predicted':y_pred_log_reg})\n# confusion_matrix_log_reg = pd.crosstab(log_reg_results['y_Actual'], log_reg_results['y_Predicted'], rownames=['Actual'], colnames=['Predicted'], margins = True)\n# confusion_matrix_log_reg","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Not great :S\nLet's bring in the cavalry..."},{"metadata":{"trusted":true},"cell_type":"code","source":"# rf = RandomForestClassifier()\n# rf.fit(X_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pred_rf = rf.predict(X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print('f1 score {}'.format(f1_score(y_test, y_pred_rf, average='weighted')))\n# print('recall score {}'.format(recall_score(y_test, y_pred_rf, average='weighted')))\n# print('precision score {}'.format(precision_score(y_test, y_pred_rf, average='weighted')))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# {key:value for key, value in zip(sorted(df['genre'].unique()), f1_score(y_test, y_pred_log_reg, average=None))}\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# y_pred_rf = rf.predict(X_test)\n# results_rf = pd.DataFrame({'y_Actual':y_test,\n#                            'y_Predicted':y_pred_rf})\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# confusion_matrix_rf = pd.crosstab(results_rf['y_Actual'], results_rf['y_Predicted'], rownames=['Actual'], colnames=['Predicted'], margins = True)\n# confusion_matrix_rf","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looks like the funk class is really letting itself down.\n\nIf we look closer we can see our model is having trouble distinguishing between funk and hiphop.\n\nThese genres are quite similar in terms of the timbre. A lot of samples will be used in hiphop that have definitely been derived from funk, so we'll let it slide for now. \n\nMaybe we can introduce nes features/more data later to boost our performance.\n\nBut right now, we'll continue with the project..."},{"metadata":{},"cell_type":"markdown","source":"# Now we have the model trained, we can now test it on pop songs\n\nCreate a predicted genre column for our pop dataframe"},{"metadata":{"trusted":true},"cell_type":"code","source":"# pop_timbre = df_pop[training_colummns]\n# df_pop['predicted_genre'] = rf.predict(None)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# df_pop['song_name'].unique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We'll pick a song from the list and see the breakdown of the composition"},{"metadata":{"trusted":true},"cell_type":"code","source":"# pop_song = df_pop[df_pop['song_name']=='CHopstix (with Travis Scott)']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# pop_song['predicted_genre'].value_counts().plot(kind='bar')\n\n# plt.title('Genre Composition for CHopstix by ScHoolboy Q with Travis Scott')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plt.rcParams[\"figure.figsize\"] = (10,10)\n\n# colors = {'hiphop':'m',\n#            'funk': 'g',\n#            'metal': 'k',\n#            'jazz':'y',\n#            'blues':'b',\n#            'classical':'r',\n#            'electronic': 'C1'}\n\n# for segment in range(len(pop_song)):\n#     prediction = pop_song.iloc[segment]['predicted_genre']\n#     start = pop_song.iloc[segment]['start']\n#     duration = pop_song.iloc[segment]['durations']\n#     plt.hlines(xmin=start, xmax= start+duration, y=1,\n#                colors=colors[prediction], linewidth= 200)\n# plt.yticks([])\n# plt.xlabel='Seconds'\n# patches = [mpatches.Patch(color=color, label=genre) for genre, color in colors.items()]\n# plt.legend(handles=patches, bbox_to_anchor=(0.5, -0.05),\n#            fancybox=True, shadow=True, ncol=7,\n#            loc='upper center')\n# plt.title('{} by {}'.format(pop_song.iloc[segment]['song_name'],\n#                             pop_song.iloc[segment]['artist']))\n\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is neat, we can see the majority of the song is made up of hiphop, electronic and a bit of funk.\nCheck the song out for yourself [here](https://www.youtube.com/watch?v=5xvxgUE_pTA)\n\nChopstiiiiicks, chopsticks, chopsticks....\n\nLyrical genius.[](http://)\n"},{"metadata":{},"cell_type":"markdown","source":"Pop music isn't necessarily a genre onto itself, it's whats popular right now. So it's kinda cool that we can see whats currently influencing current music.\n\nWe'll now test out our model with separate data from our original genres and see how it copes.\n\nWe have another dataset waiting in the wing- `testsongdata`...\n\nLet's use our original `get_genre_df` function to save some time wrangling."},{"metadata":{"trusted":true},"cell_type":"code","source":"# test_songs = get_genre_df('../input/testsongdata/')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# test_songs.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Predict the genre\n# test_songs_timbre = test_songs[training_colummns]\n# test_songs['predicted_genre'] = rf.predict(test_songs_timbre)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We'll create a function that either picks a random or chosen song  to process from our test_songs dataframe."},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_song_data(song_dataframe, genre=None, song_name=None):\n    if song_name:\n        song_df = song_dataframe[song_dataframe['song_name']==song_name]\n        return song_df\n    else:\n        genre_df = song_dataframe[song_dataframe['genre']==genre]\n        random_song = np.random.choice(genre_df['song_name'].unique())\n        random_song_data = genre_df[genre_df['song_name']==random_song]\n        return random_song_data\n        ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_song_composition_bar(song_data):\n    song_data['predicted_genre'].value_counts().plot(kind='bar')\n    plt.title('{} by {}'.format(song_data.iloc[0]['song_name'],\n                                song_data.iloc[0]['artist']))\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_song_composition_timeline(song_data):\n    plt.rcParams[\"figure.figsize\"] = (10,10)\n    colors = {'hiphop':'m',\n               'funk': 'g',\n               'metal': 'k',\n               'jazz':'y',\n               'blues':'b',\n               'classical':'r',\n               'electronic': 'C1'}\n\n    for segment in range(len(song_data)):\n        prediction = song_data.iloc[segment]['predicted_genre']\n        start = song_data.iloc[segment]['start']\n        duration = song_data.iloc[segment]['durations']\n        plt.hlines(xmin=start, xmax= start+duration, y=1,\n                   colors=colors[prediction], linewidth= 200)\n    plt.yticks([])\n    plt.xlabel='Seconds'\n    patches = [mpatches.Patch(color=color, label=genre) for genre, color in colors.items()]\n    plt.legend(handles=patches, bbox_to_anchor=(0.5, -0.05),\n               fancybox=True, shadow=True, ncol=7,\n               loc='upper center')\n    plt.title('{} by {}'.format(song_data.iloc[segment]['song_name'],\n                                song_data.iloc[segment]['artist']))\n\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def song_composition(song_dataframe, genre=None, song_name=None):\n    song_data = get_song_data(song_dataframe, genre, song_name=song_name)\n    get_song_composition_bar(None)\n    get_song_composition_timeline(None)\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# song_composition(test_songs, 'classical')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Try and explore and pick a song from the test dataframe"},{"metadata":{"trusted":true},"cell_type":"code","source":"# test_songs[test_songs['genre']=='metal']['song_name'].unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# song_composition(test_songs, genre=None, song_name='My Own Summer (Shove It)')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We'll go through each genre and pick a random song..."},{"metadata":{"trusted":true},"cell_type":"code","source":"# genre_types = ['metal','hiphop', 'funk', 'jazz', 'blues', 'classical', 'electronic']\n# for g in genre_types:\n#     print(g)\n#     song_composition(test_songs, g)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"Phew...\n\nOK that's the main part of the exercise complete, congratz! 🎉\n\nThese visualisation look pretty cool, you can even see the main parts like- intro, verses, chorus and even bridges!\n\nIf you have time to spare, or want to dive a bit deeper be our guest. 🍵🕯\n\nAlso, we mentioned about the `'90% of a data scientist time is spent sourcing and then cleaning the data'`\n\nIn this notebook we wrote `30` lines of code dedicated to ML, the rest (`284` lines) was getting and displaying the data.\n\nThat's approx `89.5%`, pretty close to the `90%` quote.\n\nHere's a few suggestions:\n\n* Try and cluster the data \n* Bring in the start time as a feature used to predict \n* Play around with the original data and see if you can extract other meaningful information for your model\n* What could we use to distinguish hiphop from funk better? Maybe time signatures, or tempo, if we had a bigger dataset we could try and bring in artist name?\n* Predict the verse, chorus, bidge, outros of a song or genre\n* Try other predictive models\n* Tune the hyperparams to optimise the current random forrest\n> * Explore other APIs\n"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# Just as promised here's a quick tutorial on calling an API\nA lot of companies open up their APIs to the public so it's always worth checking their dev pages.\n\nA great example of this is [TFL](https://api.tfl.gov.uk/)- The level of detail in this API is outrageous.\n\nIf your just looking to flex your API muscles then have a look at this website [apilist](https://apilist.fun/)\n\nThere's an API that gives us images of dogs, so of course we're going to use that...\n\nhttps://dog.ceo/dog-api/\n\nThe dataset is based on the Stanford Dogs Dataset so hopefully we get some dog images back.\n\n**NOTE** Kaggle notebooks do not support calling an API directly- So if you want to try this code out, just copy it into a local jupyter notebook/script.\n\n"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"```\n# use requests to interact with an API\nimport requests\n\n# The API path should be stated in the documentation \napi_path = 'https://dog.ceo/api/breeds/image/random'\n\n# use the get to collect data from the access point.\nr = requests.get(api_path).json()\n# Copy the url from the message and enjoy the view\nprint(r['message'])\n\n```\n"},{"metadata":{},"cell_type":"markdown","source":"See you all next time! 👋👋👋👋"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}