Clustering_EN.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Clustering_EN.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/kobemawu/www/blob/master/Clustering_EN.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sESw_dKQL5mA",
        "colab_type": "text"
      },
      "source": [
        "# NLTK Corpus Clustering with scikit-learn package"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3nFpmxQOMIKS",
        "colab_type": "text"
      },
      "source": [
        "## Preparation\n",
        "First of all, let us import necessary libraries.\n",
        "* nltk\n",
        "* sklearn\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v9v_nPKQMTZd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.cluster import KMeans\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import pandas as pd\n",
        "import nltk\n",
        "import collections"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y2UGMZJaMgvM",
        "colab_type": "text"
      },
      "source": [
        "We will use the following datasets in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MaRmXeotMjKv",
        "colab_type": "code",
        "outputId": "bafc1706-7536-414d-b311-b62631f60281",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        }
      },
      "source": [
        "nltk.download(\"stopwords\")\n",
        "nltk.download(\"wordnet\")\n",
        "nltk.download(\"reuters\")\n",
        "nltk.download(\"punkt\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n",
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/wordnet.zip.\n",
            "[nltk_data] Downloading package reuters to /root/nltk_data...\n",
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7VtrTuH4MrHN",
        "colab_type": "text"
      },
      "source": [
        "Load the corpus from NLTK package and check out the contents."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zXcN4x3bMljo",
        "colab_type": "code",
        "outputId": "5c25f738-8df9-411c-bb0a-85f70a8904f8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        }
      },
      "source": [
        "from nltk.corpus import reuters as corpus\n",
        "\n",
        "for n,item in enumerate(corpus.words(corpus.fileids()[0])[:300]):\n",
        "    print(item, end=\" \")\n",
        "    if (n%25) ==24:\n",
        "      print(\" \")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "ASIAN EXPORTERS FEAR DAMAGE FROM U . S .- JAPAN RIFT Mounting trade friction between the U . S . And Japan has raised fears  \n",
            "among many of Asia ' s exporting nations that the row could inflict far - reaching economic damage , businessmen and officials said . They  \n",
            "told Reuter correspondents in Asian capitals a U . S . Move against Japan might boost protectionist sentiment in the U . S . And  \n",
            "lead to curbs on American imports of their products . But some exporters said that while the conflict would hurt them in the long -  \n",
            "run , in the short - term Tokyo ' s loss might be their gain . The U . S . Has said it will  \n",
            "impose 300 mln dlrs of tariffs on imports of Japanese electronics goods on April 17 , in retaliation for Japan ' s alleged failure to  \n",
            "stick to a pact not to sell semiconductors on world markets at below cost . Unofficial Japanese estimates put the impact of the tariffs at  \n",
            "10 billion dlrs and spokesmen for major electronics firms said they would virtually halt exports of products hit by the new taxes . \" We  \n",
            "wouldn ' t be able to do business ,\" said a spokesman for leading Japanese electronics firm Matsushita Electric Industrial Co Ltd & lt ;  \n",
            "MC . T >. \" If the tariffs remain in place for any length of time beyond a few months it will mean the complete  \n",
            "erosion of exports ( of goods subject to tariffs ) to the U . S .,\" said Tom Murtha , a stock analyst at the  \n",
            "Tokyo office of broker & lt ; James Capel and Co >. In Taiwan , businessmen and officials are also worried . \" We are  \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HXvSxPG0MuXr",
        "colab_type": "code",
        "outputId": "36dd9857-3666-4fe2-8434-b4b94ae06afa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        }
      },
      "source": [
        "for n,item in enumerate(corpus.words(corpus.fileids()[0])[:300]):\n",
        "    print(item, end=\" \")\n",
        "    if (n%25) ==24:\n",
        "      print(\" \")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "ASIAN EXPORTERS FEAR DAMAGE FROM U . S .- JAPAN RIFT Mounting trade friction between the U . S . And Japan has raised fears  \n",
            "among many of Asia ' s exporting nations that the row could inflict far - reaching economic damage , businessmen and officials said . They  \n",
            "told Reuter correspondents in Asian capitals a U . S . Move against Japan might boost protectionist sentiment in the U . S . And  \n",
            "lead to curbs on American imports of their products . But some exporters said that while the conflict would hurt them in the long -  \n",
            "run , in the short - term Tokyo ' s loss might be their gain . The U . S . Has said it will  \n",
            "impose 300 mln dlrs of tariffs on imports of Japanese electronics goods on April 17 , in retaliation for Japan ' s alleged failure to  \n",
            "stick to a pact not to sell semiconductors on world markets at below cost . Unofficial Japanese estimates put the impact of the tariffs at  \n",
            "10 billion dlrs and spokesmen for major electronics firms said they would virtually halt exports of products hit by the new taxes . \" We  \n",
            "wouldn ' t be able to do business ,\" said a spokesman for leading Japanese electronics firm Matsushita Electric Industrial Co Ltd & lt ;  \n",
            "MC . T >. \" If the tariffs remain in place for any length of time beyond a few months it will mean the complete  \n",
            "erosion of exports ( of goods subject to tariffs ) to the U . S .,\" said Tom Murtha , a stock analyst at the  \n",
            "Tokyo office of broker & lt ; James Capel and Co >. In Taiwan , businessmen and officials are also worried . \" We are  \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2lysFJt4M1Ux",
        "colab_type": "text"
      },
      "source": [
        "The total number of documents."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_Yz5UaciM0UG",
        "colab_type": "code",
        "outputId": "67ce2cf3-6221-4889-ecbd-cd95d045cef2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "len(corpus.fileids())"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "10788"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eu_yYroqM8-8",
        "colab_type": "text"
      },
      "source": [
        "You can train the model with first K number of documents or all documents."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JOSFXIzlM5wI",
        "colab_type": "code",
        "outputId": "2121e787-4375-4fd0-df82-6db35cfc005d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        }
      },
      "source": [
        "# First K documents\n",
        "# K=1000\n",
        "# docs=[corpus.words(fileid) for fileid in corpus.fileids()[:K]]\n",
        "\n",
        "# All documents\n",
        "docs=[corpus.words(fileid) for fileid in corpus.fileids()]\n",
        "\n",
        "print(docs[:5])\n",
        "print(\"num of docs:\", len(docs))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[['ASIAN', 'EXPORTERS', 'FEAR', 'DAMAGE', 'FROM', 'U', ...], ['CHINA', 'DAILY', 'SAYS', 'VERMIN', 'EAT', '7', '-', ...], ['JAPAN', 'TO', 'REVISE', 'LONG', '-', 'TERM', ...], ['THAI', 'TRADE', 'DEFICIT', 'WIDENS', 'IN', 'FIRST', ...], ['INDONESIA', 'SEES', 'CPO', 'PRICE', 'RISING', ...]]\n",
            "num of docs: 10788\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U17VbjPaNIAD",
        "colab_type": "text"
      },
      "source": [
        "## Data preprocessing\n",
        "First, let us define some stopwords. Here we consider English stopwords from the NLTK package and some noises that may affect our result.  \n",
        "(Optional) Try to ignore numbers and words through regular expression."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CHC1l7zDNFLx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# English stopwords defined by the NLTK package.\n",
        "en_stop = nltk.corpus.stopwords.words('english')\n",
        "\n",
        "# Ignore noises that might affect our result.\n",
        "en_stop = [\"``\",\"/\",\",.\",\".,\",\";\",\"--\",\":\",\")\",\"(\",'\"','&',\"'\",'),',',\"','-','.,','.,\"','.-',\"?\",\">\",\"<\"]                  \\\n",
        "         +[\"0\",\"1\",\"2\",\"3\",\"4\",\"5\",\"6\",\"7\",\"8\",\"9\",\"10\",\"11\",\"12\",\"86\",\"1986\",\"1987\",\"000\"]                                                      \\\n",
        "         +[\"said\",\"say\",\"u\",\"v\",\"mln\",\"ct\",\"net\",\"dlrs\",\"tonne\",\"pct\",\"shr\",\"nil\",\"company\",\"lt\",\"share\",\"year\",\"billion\",\"price\"]          \\\n",
        "         +en_stop"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7hdG75XqNYPa",
        "colab_type": "text"
      },
      "source": [
        "Next, let us define several preprocessing functions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f1jUH0P-NcoP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from nltk.corpus import wordnet as wn # import for lemmatize\n",
        "\n",
        "def preprocess_word(word, stopwordset):\n",
        "    \n",
        "    #1.convert words to lowercase (e.g., Python =>python)\n",
        "    word=word.lower()\n",
        "    \n",
        "    #2.remove \",\" and \".\"\n",
        "    if word in [\",\",\".\"]:\n",
        "        return None\n",
        "    \n",
        "    #3.remove stopwords  (e.g., the => (None)) \n",
        "    if word in stopwordset:\n",
        "        return None\n",
        "    \n",
        "    #4.lemmatize  (e.g., cooked=>cook)\n",
        "    lemma = wn.morphy(word)\n",
        "    if lemma is None:\n",
        "        return word\n",
        "\n",
        "    # lemmatized words could be in the stopwords set\n",
        "    elif lemma in stopwordset: \n",
        "        return None\n",
        "    else:\n",
        "        return lemma\n",
        "    \n",
        "\n",
        "def preprocess_document(document):\n",
        "    document=[preprocess_word(w, en_stop) for w in document]\n",
        "    document=[w for w in document if w is not None]\n",
        "    return document\n",
        "\n",
        "def preprocess_documents(documents):\n",
        "    return [preprocess_document(document) for document in documents]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nJZeCVgNNi5T",
        "colab_type": "text"
      },
      "source": [
        "Let us check out the preprocessing result."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2u2P8lmTNgLn",
        "colab_type": "code",
        "outputId": "b4a5260b-9d08-4a2f-a95b-7dead0ac95fc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        }
      },
      "source": [
        "# before\n",
        "print(docs[0][:25]) \n",
        "\n",
        "# after\n",
        "print(preprocess_documents(docs)[0][:25])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['ASIAN', 'EXPORTERS', 'FEAR', 'DAMAGE', 'FROM', 'U', '.', 'S', '.-', 'JAPAN', 'RIFT', 'Mounting', 'trade', 'friction', 'between', 'the', 'U', '.', 'S', '.', 'And', 'Japan', 'has', 'raised', 'fears']\n",
            "['asian', 'exporter', 'fear', 'damage', 'japan', 'rift', 'mounting', 'trade', 'friction', 'japan', 'raise', 'fear', 'among', 'many', 'asia', 'exporting', 'nation', 'row', 'could', 'inflict', 'far', 'reaching', 'economic', 'damage', 'businessmen']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kIwd7wGvNyWj",
        "colab_type": "text"
      },
      "source": [
        "## Clustering\n",
        "Document vectorization with tf-idf. We use the TfidfVectorizer that provided by the sklearn package (and set the hyperparameter)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QVtbDMmtNuQV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# define the vectorizer\n",
        "pre_docs=preprocess_documents(docs)\n",
        "pre_docs=[\"\".join(doc) for doc in pre_docs]\n",
        "print(pre_docs[0])\n",
        "\n",
        "vectorizer = TfidfVectorizer(max_features=200, token_pattern=u'(?u)\\\\b\\\\w+\\\\b' )\n",
        "\n",
        "\n",
        "# fit\n",
        "tf_idf = vectorizer.fit_transform(pre_docs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yKhgoWLeOCW_",
        "colab_type": "text"
      },
      "source": [
        "We use K-means to cluster our documents."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qVRkE4lsOBuL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# K-means setting\n",
        "num_clusters = 8\n",
        "km = KMeans(n_clusters=num_clusters, random_state = 0)\n",
        "\n",
        "# fit\n",
        "clusters = km.fit_predict(tf_idf)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aIcN2sKSOZdu",
        "colab_type": "text"
      },
      "source": [
        "Check out the clustering result."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HBFqXETzOXec",
        "colab_type": "code",
        "outputId": "5ebffb95-587c-433f-8074-5859399a5abe",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "for doc, cls in zip(preprocess_documents(doｃs)[0], clusters):\n",
        "    print(cls,doc)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1 asian\n",
            "1 exporter\n",
            "1 fear\n",
            "1 damage\n",
            "7 japan\n",
            "1 rift\n",
            "1 mounting\n",
            "2 trade\n",
            "1 friction\n",
            "7 japan\n",
            "1 raise\n",
            "1 fear\n",
            "1 among\n",
            "1 many\n",
            "1 asia\n",
            "1 exporting\n",
            "1 nation\n",
            "1 row\n",
            "1 could\n",
            "1 inflict\n",
            "1 far\n",
            "1 reaching\n",
            "1 economic\n",
            "1 damage\n",
            "1 businessmen\n",
            "1 official\n",
            "1 tell\n",
            "1 reuter\n",
            "1 correspondent\n",
            "1 asian\n",
            "1 capital\n",
            "1 move\n",
            "7 japan\n",
            "1 might\n",
            "1 boost\n",
            "1 protectionist\n",
            "1 sentiment\n",
            "1 lead\n",
            "1 curb\n",
            "1 american\n",
            "5 import\n",
            "1 product\n",
            "1 exporter\n",
            "1 conflict\n",
            "1 would\n",
            "1 hurt\n",
            "1 long\n",
            "1 run\n",
            "1 short\n",
            "1 term\n",
            "1 tokyo\n",
            "1 loss\n",
            "1 might\n",
            "1 gain\n",
            "1 impose\n",
            "1 300\n",
            "4 tariff\n",
            "5 import\n",
            "1 japanese\n",
            "1 electronics\n",
            "1 good\n",
            "1 april\n",
            "1 17\n",
            "1 retaliation\n",
            "7 japan\n",
            "1 allege\n",
            "1 failure\n",
            "1 stick\n",
            "1 pact\n",
            "1 sell\n",
            "1 semiconductor\n",
            "1 world\n",
            "1 market\n",
            "1 cost\n",
            "1 unofficial\n",
            "1 japanese\n",
            "1 estimate\n",
            "1 put\n",
            "1 impact\n",
            "4 tariff\n",
            "1 spokesman\n",
            "1 major\n",
            "1 electronics\n",
            "1 firm\n",
            "1 would\n",
            "1 virtually\n",
            "1 halt\n",
            "1 export\n",
            "1 product\n",
            "1 hit\n",
            "1 new\n",
            "1 tax\n",
            "1 able\n",
            "1 business\n",
            "1 spokesman\n",
            "1 leading\n",
            "1 japanese\n",
            "1 electronics\n",
            "1 firm\n",
            "1 matsushita\n",
            "1 electric\n",
            "1 industrial\n",
            "1 co\n",
            "1 ltd\n",
            "1 mc\n",
            "1 >.\n",
            "4 tariff\n",
            "1 remain\n",
            "1 place\n",
            "1 length\n",
            "1 time\n",
            "1 beyond\n",
            "1 month\n",
            "1 mean\n",
            "1 complete\n",
            "1 erosion\n",
            "1 export\n",
            "1 good\n",
            "1 subject\n",
            "4 tariff\n",
            "1 tom\n",
            "1 murtha\n",
            "1 stock\n",
            "1 analyst\n",
            "1 tokyo\n",
            "1 office\n",
            "1 broker\n",
            "1 james\n",
            "1 capel\n",
            "1 co\n",
            "1 >.\n",
            "6 taiwan\n",
            "1 businessmen\n",
            "1 official\n",
            "1 also\n",
            "1 worry\n",
            "1 aware\n",
            "1 seriousness\n",
            "1 threat\n",
            "7 japan\n",
            "1 serve\n",
            "1 warning\n",
            "1 us\n",
            "1 senior\n",
            "1 taiwanese\n",
            "2 trade\n",
            "1 official\n",
            "1 ask\n",
            "1 name\n",
            "6 taiwan\n",
            "2 trade\n",
            "2 trade\n",
            "1 surplus\n",
            "1 15\n",
            "1 last\n",
            "1 95\n",
            "1 surplus\n",
            "1 help\n",
            "1 swell\n",
            "6 taiwan\n",
            "1 foreign\n",
            "1 exchange\n",
            "1 reserves\n",
            "1 53\n",
            "1 among\n",
            "1 world\n",
            "1 large\n",
            "1 must\n",
            "1 quickly\n",
            "1 open\n",
            "1 market\n",
            "1 remove\n",
            "2 trade\n",
            "1 barrier\n",
            "1 cut\n",
            "5 import\n",
            "4 tariff\n",
            "1 allow\n",
            "5 import\n",
            "1 product\n",
            "1 want\n",
            "1 defuse\n",
            "1 problem\n",
            "1 possible\n",
            "1 retaliation\n",
            "1 paul\n",
            "1 sheen\n",
            "1 chairman\n",
            "1 textile\n",
            "1 exporter\n",
            "6 taiwan\n",
            "1 safe\n",
            "1 group\n",
            "1 >.\n",
            "1 senior\n",
            "1 official\n",
            "1 south\n",
            "1 korea\n",
            "2 trade\n",
            "1 promotion\n",
            "1 association\n",
            "2 trade\n",
            "1 dispute\n",
            "7 japan\n",
            "1 might\n",
            "1 also\n",
            "1 lead\n",
            "1 pressure\n",
            "1 south\n",
            "1 korea\n",
            "1 whose\n",
            "1 chief\n",
            "1 export\n",
            "1 similar\n",
            "7 japan\n",
            "1 last\n",
            "1 south\n",
            "1 korea\n",
            "2 trade\n",
            "1 surplus\n",
            "1 1985\n",
            "1 malaysia\n",
            "2 trade\n",
            "1 officer\n",
            "1 businessmen\n",
            "1 tough\n",
            "1 curb\n",
            "7 japan\n",
            "1 might\n",
            "1 allow\n",
            "1 hard\n",
            "1 hit\n",
            "1 producer\n",
            "1 semiconductor\n",
            "1 third\n",
            "1 country\n",
            "1 expand\n",
            "1 sales\n",
            "3 hong\n",
            "1 kong\n",
            "1 newspaper\n",
            "1 allege\n",
            "7 japan\n",
            "1 selling\n",
            "1 cost\n",
            "1 semiconductor\n",
            "1 electronics\n",
            "1 manufacturer\n",
            "1 view\n",
            "1 businessmen\n",
            "1 short\n",
            "1 term\n",
            "1 commercial\n",
            "1 advantage\n",
            "1 would\n",
            "1 outweigh\n",
            "1 pressure\n",
            "1 block\n",
            "5 import\n",
            "1 short\n",
            "1 term\n",
            "1 view\n",
            "1 lawrence\n",
            "1 mills\n",
            "1 director\n",
            "1 general\n",
            "1 federation\n",
            "3 hong\n",
            "1 kong\n",
            "1 industry\n",
            "1 whole\n",
            "1 purpose\n",
            "1 prevent\n",
            "5 import\n",
            "0 one\n",
            "1 day\n",
            "1 extend\n",
            "1 source\n",
            "1 much\n",
            "1 serious\n",
            "3 hong\n",
            "1 kong\n",
            "1 disadvantage\n",
            "1 action\n",
            "1 restrain\n",
            "2 trade\n",
            "1 last\n",
            "3 hong\n",
            "1 kong\n",
            "1 big\n",
            "1 export\n",
            "1 market\n",
            "1 accounting\n",
            "1 30\n",
            "1 domestically\n",
            "1 produce\n",
            "1 export\n",
            "1 australian\n",
            "1 government\n",
            "1 await\n",
            "1 outcome\n",
            "2 trade\n",
            "1 talks\n",
            "7 japan\n",
            "1 interest\n",
            "1 concern\n",
            "1 industry\n",
            "1 minister\n",
            "1 john\n",
            "1 button\n",
            "1 canberra\n",
            "1 last\n",
            "1 friday\n",
            "1 kind\n",
            "1 deterioration\n",
            "2 trade\n",
            "1 relations\n",
            "1 two\n",
            "1 country\n",
            "1 major\n",
            "1 trading\n",
            "1 partner\n",
            "1 serious\n",
            "1 matter\n",
            "1 button\n",
            "1 australia\n",
            "1 concern\n",
            "1 centre\n",
            "1 coal\n",
            "1 beef\n",
            "1 australia\n",
            "1 two\n",
            "1 large\n",
            "1 export\n",
            "7 japan\n",
            "1 also\n",
            "1 significant\n",
            "1 export\n",
            "1 country\n",
            "1 meanwhile\n",
            "1 japanese\n",
            "1 diplomatic\n",
            "1 manoeuvre\n",
            "1 solve\n",
            "2 trade\n",
            "1 stand\n",
            "1 continue\n",
            "7 japan\n",
            "1 ruling\n",
            "1 liberal\n",
            "1 democratic\n",
            "1 party\n",
            "1 yesterday\n",
            "1 outline\n",
            "1 package\n",
            "1 economic\n",
            "1 measure\n",
            "1 boost\n",
            "1 japanese\n",
            "1 economy\n",
            "1 measure\n",
            "1 propose\n",
            "1 include\n",
            "1 large\n",
            "1 supplementary\n",
            "1 budget\n",
            "1 record\n",
            "1 public\n",
            "1 works\n",
            "1 spending\n",
            "1 first\n",
            "1 half\n",
            "1 financial\n",
            "1 also\n",
            "1 call\n",
            "1 step\n",
            "1 spending\n",
            "1 emergency\n",
            "1 measure\n",
            "1 stimulate\n",
            "1 economy\n",
            "1 despite\n",
            "1 prime\n",
            "1 minister\n",
            "1 yasuhiro\n",
            "1 nakasone\n",
            "1 avow\n",
            "1 fiscal\n",
            "1 reform\n",
            "1 program\n",
            "1 deputy\n",
            "2 trade\n",
            "1 representative\n",
            "1 michael\n",
            "1 smith\n",
            "1 makoto\n",
            "1 kuroda\n",
            "7 japan\n",
            "1 deputy\n",
            "1 minister\n",
            "1 international\n",
            "2 trade\n",
            "1 industry\n",
            "1 miti\n",
            "1 due\n",
            "1 meet\n",
            "1 washington\n",
            "1 week\n",
            "1 effort\n",
            "1 end\n",
            "1 dispute\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ps1PgVeyOfbU",
        "colab_type": "text"
      },
      "source": [
        "## Hints\n",
        "\n",
        "There are many hyperparameters in the vectorizer and kmeans of scikit-learn. The vectorizer method also provides data preprocessing functions with hyperparameters (e.g., stop_words). The clustering result will change according to the change of these hyperparameters. You can try different hyperparameter settings to check out the result refer to the following URL.   \n",
        "* About TF-IDF   \n",
        "    https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html   \n",
        "* About K-means   \n",
        "    https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yRRWSoojOm1-",
        "colab_type": "text"
      },
      "source": [
        "## Try it yourself\n",
        "Modify the above code with the following methods to check out the differences:\n",
        "1. Try other vectorization methods (e.g., bag-of-words)\n",
        "2. Try other clustering methods (e.g., hierarchical clustering) or visualize the result of K-means."
      ]
    }
  ]
}