tf-hub-for-tf2-imdb-use-and-gnews.ipynb

{"cells":[{"metadata":{"colab_type":"text","id":"Eg62Pmz3o83v"},"cell_type":"markdown","source":"# IMDB USE and GNEWS Embedding Models\n\n_By Nick Brooks, February 2020_\n\n\n# TF Hub for TF2: Text classification with movie reviews (preview)\n\nSOURCE: https://github.com/tensorflow/hub/blob/master/examples/colab/tf2_text_classification.ipynb\n\nThis notebook classifies movie reviews as *positive* or *negative* using the text of the review. This is an example of *binary*—or two-class—classification, an important and widely applicable kind of machine learning problem. \n\nWe'll use the [IMDB dataset](https://www.tensorflow.org/api_docs/python/tf/keras/datasets/imdb) that contains the text of 50,000 movie reviews from the [Internet Movie Database](https://www.imdb.com/). These are split into 25,000 reviews for training and 25,000 reviews for testing. The training and testing sets are *balanced*, meaning they contain an equal number of positive and negative reviews. \n\nThis notebook uses [tf.keras](https://www.tensorflow.org/guide/keras), a high-level API to build and train models in TensorFlow, and [TensorFlow Hub](https://www.tensorflow.org/hub), a library and platform for transfer learning. For a more advanced text classification tutorial using `tf.keras`, see the [MLCC Text Classification Guide](https://developers.google.com/machine-learning/guides/text-classification/)."},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install tensorflow_datasets > /dev/null","execution_count":null,"outputs":[]},{"metadata":{"colab":{},"colab_type":"code","id":"2ew7HTbPpCJH","trusted":true},"cell_type":"code","source":"import time\nimport numpy as np\nimport gc\nimport pandas as pd\n\nimport tensorflow as tf\nimport tensorflow_hub as hub\nimport tensorflow_datasets as tfds\nfrom keras import backend as K\n\nfrom tensorflow.keras.layers import Dense, Input, BatchNormalization, Dropout, concatenate, GlobalAveragePooling1D\nfrom tensorflow.keras.models import Model, Sequential\nfrom tensorflow.keras.optimizers import Adam, SGD\n\nimport matplotlib as mpl\nimport matplotlib.pyplot as plt\nfrom sklearn.cluster import KMeans\nfrom wordcloud import WordCloud, STOPWORDS\nfrom sklearn import metrics\n\nprint(\"Version: \", tf.__version__)\nprint(\"Eager mode: \", tf.executing_eagerly())\nprint(\"Hub version: \", hub.__version__)\nprint(\"GPU is\", \"available\" if tf.test.is_gpu_available() else \"NOT AVAILABLE\")\n\nstart = time.time()\npd.options.display.max_colwidth = 1500","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# My Parameters\nBATCH_SIZE = 512\nSEED=42","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(\"Word Cloud Function..\")\nstopwords = set(STOPWORDS)\nsize = (20,10)\n\ndef cloud(text, title, stopwords=stopwords, size=size):\n    \"\"\"\n    Function to plot WordCloud\n    Includes: \n    \"\"\"\n    # Setting figure parameters\n    mpl.rcParams['figure.figsize']=(10.0,10.0)\n    mpl.rcParams['font.size']=12\n    mpl.rcParams['savefig.dpi']=100\n    mpl.rcParams['figure.subplot.bottom']=.1 \n    \n    # Processing Text\n    # Redundant when combined with my Preprocessing function\n    wordcloud = WordCloud(width=1600, height=800,\n                          background_color='black',\n                          stopwords=stopwords,\n                         ).generate(str(text))\n    \n    # Output Visualization\n    fig = plt.figure(figsize=size, dpi=80, facecolor='k',edgecolor='k')\n    plt.imshow(wordcloud,interpolation='bilinear')\n    plt.axis('off')\n    plt.title(title, fontsize=50,color='y')\n    plt.tight_layout(pad=0)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"iAsKG535pHep"},"cell_type":"markdown","source":"## Download the IMDB dataset\n\nThe IMDB dataset is available on [TensorFlow datasets](https://github.com/tensorflow/datasets). The following code downloads the IMDB dataset to your machine (or the colab runtime):"},{"metadata":{"colab":{},"colab_type":"code","id":"zXXx5Oc3pOmN","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"train_data, test_data = tfds.load(name=\"imdb_reviews\", split=[\"train\", \"test\"], \n                                  batch_size=-1, as_supervised=True)\n\ntrain_examples, train_labels = tfds.as_numpy(train_data)\ntest_examples, test_labels = tfds.as_numpy(test_data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"np.save(\"train_examples\", train_examples)\nnp.save(\"train_labels\", train_labels)\n\nnp.save(\"test_examples\", test_examples)\nnp.save(\"test_labels\", test_labels)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"l50X3GfjpU4r"},"cell_type":"markdown","source":"## Explore the data \n\nLet's take a moment to understand the format of the data. Each example is a sentence representing the movie review and a corresponding label. The sentence is not preprocessed in any way. The label is an integer value of either 0 or 1, where 0 is a negative review, and 1 is a positive review."},{"metadata":{"colab":{},"colab_type":"code","id":"y8qCnve_-lkO","trusted":true},"cell_type":"code","source":"print(\"Training entries: {}, test entries: {}\".format(len(train_examples), len(test_examples)))","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"RnKvHWW4-lkW"},"cell_type":"markdown","source":"Let's print first 10 examples."},{"metadata":{"trusted":true},"cell_type":"code","source":"input_len = [len(x) for x in np.concatenate((train_examples, test_examples), axis=0)]\nprint(\"Input Lengths:\\nAverage {:.1f} +/- {:.1f}\\nMax {} Min {}\".format(np.mean(input_len), np.std(input_len), np.max(input_len), np.min(input_len)))","execution_count":null,"outputs":[]},{"metadata":{"colab":{},"colab_type":"code","id":"QtTS4kpEpjbi","trusted":true},"cell_type":"code","source":"train_examples[:10]","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"IFtaCHTdc-GY"},"cell_type":"markdown","source":"Let's also print the first 10 labels."},{"metadata":{"colab":{},"colab_type":"code","id":"tvAjVXOWc6Mj","trusted":true},"cell_type":"code","source":"train_labels[:10]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Look at class balance..\nunique_elements, counts_elements = np.unique(train_labels, return_counts=True)\nprint(\"Frequency of unique values of the said array:\")\nprint(np.asarray((unique_elements, counts_elements)))","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"LLC02j2g-llC"},"cell_type":"markdown","source":"## Build the model\n\nThe neural network is created by stacking layers—this requires three main architectural decisions:\n\n* How to represent the text?\n* How many layers to use in the model?\n* How many *hidden units* to use for each layer?\n\nIn this example, the input data consists of sentences. The labels to predict are either 0 or 1.\n\nOne way to represent the text is to convert sentences into embeddings vectors. We can use a pre-trained text embedding as the first layer, which will have two advantages:\n*   we don't have to worry anout text preprocessing,\n*   we can benefit from transfer learning.\n\nFor this example we will use a model from [TensorFlow Hub](https://www.tensorflow.org/hub) called [google/tf2-preview/gnews-swivel-20dim/1](https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1).\n\nThere are three other models to test for the sake of this tutorial:\n* [google/tf2-preview/gnews-swivel-20dim-with-oov/1](https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim-with-oov/1) - same as [google/tf2-preview/gnews-swivel-20dim/1](https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1), but with 2.5% vocabulary converted to OOV buckets. This can help if vocabulary of the task and vocabulary of the model don't fully overlap.\n* [google/tf2-preview/nnlm-en-dim50/1](https://tfhub.dev/google/tf2-preview/nnlm-en-dim50/1) - A much larger model with ~1M vocabulary size and 50 dimensions.\n* [google/tf2-preview/nnlm-en-dim128/1](https://tfhub.dev/google/tf2-preview/nnlm-en-dim128/1) - Even larger model with ~1M vocabulary size and 128 dimensions."},{"metadata":{"colab_type":"text","id":"In2nDpTLkgKa"},"cell_type":"markdown","source":"Let's first create a Keras layer that uses a TensorFlow Hub model to embed the sentences, and try it out on a couple of input examples. Note that the output shape of the produced embeddings is a expected: `(num_examples, embedding_dimension)`."},{"metadata":{},"cell_type":"markdown","source":"## GNEWS Embeddings Model"},{"metadata":{"colab":{},"colab_type":"code","id":"_NUbzVeYkgcO","trusted":true},"cell_type":"code","source":"%%time\nmodel = \"https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1\"\nhub_layer = hub.KerasLayer(model, output_shape=[], input_shape=[], \n                           dtype=tf.string, trainable=True, name='gnews_embedding')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hub_layer(train_examples[:3])","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"dfSbV6igl1EH"},"cell_type":"markdown","source":"Let's now build the full model:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model(embed):\n    \n    model = Sequential([\n        Input(shape=[], dtype=tf.string),\n        embed,\n        Dropout(.2),\n        Dense(16, activation='relu'),\n        Dropout(.2),\n        Dense(1, activation='sigmoid')\n    ])\n    model.compile(Adam(), loss='binary_crossentropy', metrics=['accuracy'])\n    \n    return model\n\nmodel = build_model(hub_layer)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"es = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=4, verbose=1, mode='min')\ncheckpoint = tf.keras.callbacks.ModelCheckpoint('model.h5', monitor='val_loss', save_best_only=True)\n\nhistory = model.fit(\n                    train_examples,\n                    train_labels,\n                    epochs=40,\n                    batch_size=BATCH_SIZE,\n                    validation_split = .2,\n                    shuffle = True,\n                    callbacks = [checkpoint, es],\n                    verbose=1)\n\nmodel.load_weights('model.h5')\nresults = model.evaluate(test_examples, test_labels)\nprint(results)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history_dict = history.history\nhistory_dict.keys()\n\nacc = history_dict['accuracy']\nval_acc = history_dict['val_accuracy']\nloss = history_dict['loss']\nval_loss = history_dict['val_loss']\n\nepochs = range(1, len(acc) + 1)\n\nf, ax = plt.subplots(1,2, figsize = [11,4])\n\n# \"bo\" is for \"blue dot\"\nax[0].plot(epochs, loss, 'r', label='Training loss')\n# b is for \"solid blue line\"\nax[0].plot(epochs, val_loss, 'b', label='Validation loss')\nax[0].set_title('Training and validation loss')\nax[0].set_xlabel('Epochs')\nax[0].set_ylabel('Loss')\nax[0].legend()\n\nax[1].plot(epochs, acc, 'r', label='Training acc')\nax[1].plot(epochs, val_acc, 'b', label='Validation acc')\nax[1].set_title('Training and validation accuracy')\nax[1].set_xlabel('Epochs')\nax[1].set_ylabel('Accuracy')\nax[1].legend()\n\nplt.tight_layout()\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"# Test Predictions\ntest_pred = model.predict(test_examples, batch_size = BATCH_SIZE)\nresults_pd = pd.DataFrame.from_dict({'text': test_examples, 'pred': test_pred[:,0], 'ground_truth': test_labels})\nresults_pd['error'] = results_pd['ground_truth'] - results_pd['pred']\n\nprint(\"Look at False Negative\")\ndisplay(results_pd.sort_values(by = 'error', ascending=False).iloc[:10])\n\nprint(\"Look at False Positives\")\ndisplay(results_pd.sort_values(by = 'error', ascending=True).iloc[:10])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Clear Memory\nK.clear_session()\n\ndel history\ndel model\n_ = gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Universal Sentence Encoding Embeddings Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nmodule_url = 'https://tfhub.dev/google/universal-sentence-encoder-large/4'\nUSE_embed = hub.KerasLayer(module_url, trainable=False, name='USE_embedding')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"USE_embed(train_examples[:3])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model(embed):\n    \n    model = Sequential([\n        Input(shape=[], dtype=tf.string),\n        embed,\n        Dropout(.2),\n        Dense(16, activation='relu'),\n        Dropout(.2),\n        Dense(1, activation='sigmoid')\n    ])\n    model.compile(Adam(), loss='binary_crossentropy', metrics=['accuracy'])\n    \n    return model\n\nmodel = build_model(USE_embed)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"MAX_LEN = 2058\n\nsmall_train_examples = np.array([x[:MAX_LEN] for x in train_examples])\nsmall_test_examples = np.array([x[:MAX_LEN] for x in test_examples])","execution_count":null,"outputs":[]},{"metadata":{"colab":{},"colab_type":"code","id":"tXSGrjWZ-llW","trusted":true},"cell_type":"code","source":"es = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=4, verbose=1, mode='min')\ncheckpoint = tf.keras.callbacks.ModelCheckpoint('model.h5', monitor='val_loss', save_best_only=True)\n\nhistory = model.fit(\n                    small_train_examples,\n                    train_labels,\n                    epochs=40,\n                    batch_size=BATCH_SIZE,\n                    validation_split = .2,\n                    shuffle = True,\n                    callbacks = [checkpoint, es],\n                    verbose=1)\n\nmodel.load_weights('model.h5')\nresults = model.evaluate(small_test_examples, test_labels)\nprint(results)","execution_count":null,"outputs":[]},{"metadata":{"colab":{},"colab_type":"code","id":"nGoYf2Js-lle","trusted":true},"cell_type":"code","source":"history_dict = history.history\nhistory_dict.keys()\n\nacc = history_dict['accuracy']\nval_acc = history_dict['val_accuracy']\nloss = history_dict['loss']\nval_loss = history_dict['val_loss']\n\nepochs = range(1, len(acc) + 1)\n\nf, ax = plt.subplots(1,2, figsize = [11,4])\n\n# \"bo\" is for \"blue dot\"\nax[0].plot(epochs, loss, 'r', label='Training loss')\n# b is for \"solid blue line\"\nax[0].plot(epochs, val_loss, 'b', label='Validation loss')\nax[0].set_title('Training and validation loss')\nax[0].set_xlabel('Epochs')\nax[0].set_ylabel('Loss')\nax[0].legend()\n\nax[1].plot(epochs, acc, 'r', label='Training acc')\nax[1].plot(epochs, val_acc, 'b', label='Validation acc')\nax[1].set_title('Training and validation accuracy')\nax[1].set_xlabel('Epochs')\nax[1].set_ylabel('Accuracy')\nax[1].legend()\n\nplt.tight_layout()\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Test Predictions\ntest_pred = model.predict(small_test_examples, batch_size = BATCH_SIZE)\nresults_pd = pd.DataFrame.from_dict({'text': test_examples, 'pred': test_pred[:,0], 'ground_truth': test_labels})\nresults_pd['error'] = results_pd['ground_truth'] - results_pd['pred']\n\nprint(\"Look at False Negative\")\ndisplay(results_pd.sort_values(by = 'error', ascending=False).iloc[:10])\n\nprint(\"Look at False Positives\")\ndisplay(results_pd.sort_values(by = 'error', ascending=True).iloc[:10])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Universal Sentence Encoding Clustering"},{"metadata":{"trusted":true},"cell_type":"code","source":"# USE output shape..\nUSE_embed([small_train_examples[0]])['outputs'].numpy().shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nfull_labels = np.concatenate((train_labels, test_labels))\nfull_txt = np.concatenate((small_train_examples, small_test_examples))\n\nbatch_size = 500\nembeddings = []\n\nfor b in range(0, full_txt.shape[0] // batch_size):\n    embeddings.extend(USE_embed(full_txt[batch_size*b: batch_size*(b+1)])['outputs'].numpy())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Fit Kmeans "},{"metadata":{"trusted":true},"cell_type":"code","source":"kmeans = KMeans(n_clusters=3, random_state=SEED).fit(embeddings)\nprint(\"Silhouette Coefficient: %0.3f\"% metrics.silhouette_score(embeddings, kmeans.labels_, sample_size=1000))\n\n# Prepare DataFrame\ndf = pd.DataFrame.from_dict({\"Text\": full_txt,\n                             \"Labels\": np.concatenate((train_labels, test_labels)),\n                             \"Clusters\": kmeans.labels_})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Model Evaluation"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Homogeneity: %0.3f\" % metrics.homogeneity_score(full_labels, kmeans.labels_))\nprint(\"Completeness: %0.3f\" % metrics.completeness_score(full_labels, kmeans.labels_))\nprint(\"V-measure: %0.3f\" % metrics.v_measure_score(full_labels, kmeans.labels_))\nprint(\"Adjusted Rand-Index: %.3f\"\n      % metrics.adjusted_rand_score(full_labels, kmeans.labels_))\nprint(\"Silhouette Coefficient: %0.3f\"\n      % metrics.silhouette_score(embeddings, full_labels, sample_size=1000))\n\ndisplay(pd.crosstab(df['Clusters'], df['Labels']))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for c in sorted(df['Clusters'].unique()):\n    cloud(df.loc[df.Clusters == c,\"Text\"].astype(str).str.title().values, title=f\"Cluster ID: {c}\", size=[8,5])\n    display(df.loc[df.Clusters == c,:].sample(5, random_state=SEED))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Notebook Runtime: %0.2f Minutes\"%((time.time() - start)/60))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"colab":{"collapsed_sections":[],"name":"basic-text-classification.ipynb","private_outputs":true,"provenance":[],"toc_visible":true,"version":"0.3.2"},"kernelspec":{"display_name":"Python 3","name":"python3"}},"nbformat":4,"nbformat_minor":4}