clustering_personal_business.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clustering\n",
    "\n",
    "The clustering solution tries to address the problem:\n",
    "\n",
    "*Mixing of personal and business expenses is a common problem for small business. Create an algorithm that can separate any potential personal expenses in the training data. Labels of personal and business expenses were deliberately not given as this is often the case in our system. There is no right answer so it is important you provide any assumptions you have made.*\n",
    "\n",
    "For this, we will use K-means clustering for simplicity.\n",
    "\n",
    "For distinguishing personal expenses from business expenses, we are going to make the following **assumptions** (**none of those are necessarily true in all cases**):\n",
    "\n",
    "- If the date of expenses is on weekeends or holidays, it is more likely to be a personal expense (notice, however, the flaw of this assumption is that an expense on Friday night might very well be personal)\n",
    "- Higher expense amount has more chance of being a business expense\n",
    "- The text fields along with the previous 2 criteria (expense description and category) can possibly give us some clue about the type of expense. For example, category **office supplies** and **Computer Hardware** has more chance of being a business expense, whereas **Meals and Entertainment** with a relatively low expense amount has a higher chance of being a personal expense, especially if it is on a weekend.\n",
    "- Employee ID might useful as well since certain employees might have certain habits of making personal or business type expenses.\n",
    "- Same goes for role.\n",
    "\n",
    "(Though not shown in this notebook, later I have found that excluding employee id and role does not alter the resulting clusters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Necessary imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from collections import defaultdict\n",
    "import itertools\n",
    "\n",
    "from sklearn import preprocessing\n",
    "from sklearn.cluster import KMeans"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read in the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train = pd.read_csv('training_data_example.csv')\n",
    "df_val = pd.read_csv('validation_data_example.csv')\n",
    "df_employee = pd.read_csv('employee.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ready the features to be used\n",
    "\n",
    "We will define a pre-processing function to ready our features. \n",
    "- It has an internal function to determine whether a prticular day is a holiday. It's kind of simplified and by no means an exact dertminer of a holiday. I am assuming weekends and any day after December 19 as a holiday.\n",
    "- It basically drops all columns except employee id, role and pre-tax amount\n",
    "- Text features (category and expense description will be processed separately)\n",
    "- Pre-tax amount is normalized to 0-1 scale"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pre_process(df, columns_to_drop=['date',\n",
    "                                     'category', \n",
    "                                     'tax amount',\n",
    "                                     'tax name',  \n",
    "                                     'expense description']):\n",
    "    \n",
    "    def holiday(x):\n",
    "        if x.weekday() >= 5:\n",
    "            return 1\n",
    "        if x.month == 12 and x.day >= 20:\n",
    "            return 1\n",
    "        return 0\n",
    "    \n",
    "    df['holiday'] = pd.to_datetime(df['date']).apply(lambda x: holiday(x)).astype(str)    \n",
    "    df = pd.merge(df, df_employee[['employee id', 'role']], how='inner', on=['employee id'])\n",
    "    df['employee id'] = df['employee id'].astype(str)\n",
    "    df = df.drop(columns_to_drop, axis=1)\n",
    "    \n",
    "    # one-hot encode the categorical variables\n",
    "    df = pd.get_dummies(df)\n",
    "    \n",
    "    df['pre-tax amount'] = preprocessing.minmax_scale(df[['pre-tax amount']])\n",
    "    \n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the pre-process features for training and validation data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = pre_process(df_train)\n",
    "x_val = pre_process(df_val)\n",
    "x_train, x_val = x_train.align(x_val, join='left', axis=1)\n",
    "x_val = x_val.fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pre-tax amount</th>\n",
       "      <th>employee id_1</th>\n",
       "      <th>employee id_3</th>\n",
       "      <th>employee id_4</th>\n",
       "      <th>employee id_5</th>\n",
       "      <th>employee id_6</th>\n",
       "      <th>employee id_7</th>\n",
       "      <th>holiday_0</th>\n",
       "      <th>holiday_1</th>\n",
       "      <th>role_CEO</th>\n",
       "      <th>role_Engineer</th>\n",
       "      <th>role_IT and Admin</th>\n",
       "      <th>role_Sales</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.018045</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.098246</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.115789</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.005514</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   pre-tax amount  employee id_1  employee id_3  employee id_4  employee id_5  \\\n",
       "0        0.018045              0              0              0              0   \n",
       "1        0.098246              0              0              0              0   \n",
       "2        1.000000              0              0              0              0   \n",
       "3        0.115789              1              0              0              0   \n",
       "4        0.005514              1              0              0              0   \n",
       "\n",
       "   employee id_6  employee id_7  holiday_0  holiday_1  role_CEO  \\\n",
       "0              0              1          1          0         0   \n",
       "1              0              1          1          0         0   \n",
       "2              0              1          1          0         0   \n",
       "3              0              0          1          0         1   \n",
       "4              0              0          0          1         1   \n",
       "\n",
       "   role_Engineer  role_IT and Admin  role_Sales  \n",
       "0              0                  0           1  \n",
       "1              0                  0           1  \n",
       "2              0                  0           1  \n",
       "3              0                  0           0  \n",
       "4              0                  0           0  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Processing the text fields\n",
    "\n",
    "For this purpose, we can make use of as much text as available.\n",
    "\n",
    "- We will concatenate **category** and **text description** together into a single field\n",
    "- Since K-means will internally use Euclidean distance, we will vecorize the field. We will try 2 different ways if vectorizing it:\n",
    "    - Using Scikit learn's TfIdfVectoirizer (which is just a frequency based embedding of the words)\n",
    "    - Using a pre-trained prediction based embedding of the words. This approach makes more sense because this sort of embeddings caoture useful relations between words.  We will use the **glove** embedding https://nlp.stanford.edu/projects/glove/ with 100 dimensional vector. We will use the technique describe here http://nadbordrozd.github.io/blog/2016/05/20/text-classification-with-word2vec/. The following class was taken from this reference. I tweaked it slightly to work with python 3.\n",
    "    - We will then compare the results of both of these approaches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reference: http://nadbordrozd.github.io/blog/2016/05/20/text-classification-with-word2vec/\n",
    "class TfidfEmbeddingVectorizer(object):\n",
    "    def __init__(self, word2vec):\n",
    "        self.word2vec = word2vec\n",
    "        self.word2weight = None\n",
    "        self.dim = len(word2vec.values())\n",
    "\n",
    "    def fit(self, X):\n",
    "        tfidf = TfidfVectorizer(analyzer=lambda x: x)\n",
    "        tfidf.fit(X)\n",
    "        # if a word was never seen - it must be at least as infrequent\n",
    "        # as any of the known words - so the default idf is the max of \n",
    "        # known idf's\n",
    "        max_idf = max(tfidf.idf_)\n",
    "        self.word2weight = defaultdict(\n",
    "            lambda: max_idf,\n",
    "            [(w, tfidf.idf_[i]) for w, i in tfidf.vocabulary_.items()])\n",
    "\n",
    "        return self\n",
    "\n",
    "    def transform(self, X):\n",
    "        return np.array([\n",
    "                np.mean([self.word2vec[w] * self.word2weight[w]\n",
    "                         for w in words if w in self.word2vec] or\n",
    "                        [np.zeros(self.dim)], axis=0)\n",
    "                for words in X\n",
    "            ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we will read in the embedding vectors from the relevant file and fit using the above TfidfEmbeddingVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"glove.6B.100d.txt\", \"rb\") as lines:\n",
    "    w2v = {line.split()[0]: np.array(map(float, line.split()[1:]))\n",
    "           for line in lines}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will create a new field concatenating category and expense description."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 242,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train['text_data'] = df_train['expense description'] + ' ' + df_train['category']\n",
    "df_val['text_data'] = df_val['expense description'] + ' ' + df_val['category']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's fit the embedding vectorizer, which considers TfIdf on top of the embedding vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 243,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<__main__.TfidfEmbeddingVectorizer at 0x117adfef0>"
      ]
     },
     "execution_count": 243,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_vectorizer = TfidfEmbeddingVectorizer(w2v)\n",
    "embedding_vectorizer.fit(df_train['text_data'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also fit using only the TfIdfVectorizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 244,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TfidfVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "        ngram_range=(1, 1), norm='l2', preprocessor=None, smooth_idf=True,\n",
       "        stop_words='english', strip_accents=None, sublinear_tf=False,\n",
       "        token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b', tokenizer=None, use_idf=True,\n",
       "        vocabulary=None)"
      ]
     },
     "execution_count": 244,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorizer = TfidfVectorizer(stop_words='english')\n",
    "vectorizer.fit(df_train['text_data'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's prepare the text vectors for the training and validation data, for both - using and not using pre-trained embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 245,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train_tfidf = vectorizer.transform(df_train['text_data']).toarray()\n",
    "x_val_tfidf = vectorizer.transform(df_val['text_data']).toarray()\n",
    "\n",
    "x_train_tfidf_embedding = embedding_vectorizer.transform(df_train['text_data'])\n",
    "x_val_tfidf_embedding = embedding_vectorizer.transform(df_val['text_data'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Concatenate the text vectors with the non-text feature vectors to get the final feature vectors (again, for both - using and not using pre-trained embeddings.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train_final_no_embedding = np.concatenate((x_train.values, x_train_tfidf), axis=1)\n",
    "x_val_final_no_embedding = np.concatenate((x_val.values, x_val_tfidf), axis=1)\n",
    "\n",
    "x_train_final_embedding = np.concatenate((x_train.values, x_train_tfidf_embedding), axis=1)\n",
    "x_val_final_embedding = np.concatenate((x_val.values, x_val_tfidf_embedding), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Apply K-means\n",
    "\n",
    "Now that we have the fetures ready, we will apply K-means to it.\n",
    "First, let's try with the one that has no pretrained embeddings considered (only tf-idf)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 261,
   "metadata": {},
   "outputs": [],
   "source": [
    "kmeans = KMeans(n_clusters=2, random_state=1)\n",
    "kmeans.fit(x_train_final_no_embedding)\n",
    "clusters_train = kmeans.predict(x_train_final_no_embedding)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 248,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 1,\n",
       "       1], dtype=int32)"
      ]
     },
     "execution_count": 248,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clusters_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 249,
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster1 = df_train[clusters_train==0]\n",
    "cluster2 = df_train[clusters_train==1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 250,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>category</th>\n",
       "      <th>employee id</th>\n",
       "      <th>expense description</th>\n",
       "      <th>pre-tax amount</th>\n",
       "      <th>tax name</th>\n",
       "      <th>tax amount</th>\n",
       "      <th>holiday</th>\n",
       "      <th>text_data</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11/1/2016</td>\n",
       "      <td>Travel</td>\n",
       "      <td>7</td>\n",
       "      <td>Taxi ride</td>\n",
       "      <td>40.0</td>\n",
       "      <td>NY Sales tax</td>\n",
       "      <td>3.55</td>\n",
       "      <td>0</td>\n",
       "      <td>Taxi ride Travel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11/15/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>1</td>\n",
       "      <td>Team lunch</td>\n",
       "      <td>235.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>30.55</td>\n",
       "      <td>0</td>\n",
       "      <td>Team lunch Meals and Entertainment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11/30/2016</td>\n",
       "      <td>Computer - Hardware</td>\n",
       "      <td>3</td>\n",
       "      <td>HP Laptop Computer</td>\n",
       "      <td>999.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>129.87</td>\n",
       "      <td>0</td>\n",
       "      <td>HP Laptop Computer Computer - Hardware</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>11/14/2016</td>\n",
       "      <td>Computer - Software</td>\n",
       "      <td>3</td>\n",
       "      <td>Microsoft Office</td>\n",
       "      <td>899.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>116.87</td>\n",
       "      <td>0</td>\n",
       "      <td>Microsoft Office Computer - Software</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>12/9/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>6</td>\n",
       "      <td>Coffee with Steve</td>\n",
       "      <td>300.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>39.00</td>\n",
       "      <td>0</td>\n",
       "      <td>Coffee with Steve Meals and Entertainment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>11/12/2016</td>\n",
       "      <td>Travel</td>\n",
       "      <td>4</td>\n",
       "      <td>Taxi ride</td>\n",
       "      <td>230.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>29.90</td>\n",
       "      <td>1</td>\n",
       "      <td>Taxi ride Travel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>11/21/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>7</td>\n",
       "      <td>Client dinner</td>\n",
       "      <td>200.0</td>\n",
       "      <td>NY Sales tax</td>\n",
       "      <td>17.75</td>\n",
       "      <td>0</td>\n",
       "      <td>Client dinner Meals and Entertainment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10/4/2016</td>\n",
       "      <td>Travel</td>\n",
       "      <td>6</td>\n",
       "      <td>Flight to Miami</td>\n",
       "      <td>200.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>26.00</td>\n",
       "      <td>0</td>\n",
       "      <td>Flight to Miami Travel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>10/12/2016</td>\n",
       "      <td>Computer - Hardware</td>\n",
       "      <td>7</td>\n",
       "      <td>Macbook Air Computer</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>NY Sales tax</td>\n",
       "      <td>177.41</td>\n",
       "      <td>0</td>\n",
       "      <td>Macbook Air Computer Computer - Hardware</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>12/30/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>4</td>\n",
       "      <td>Dinner with potential client</td>\n",
       "      <td>200.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>26.00</td>\n",
       "      <td>1</td>\n",
       "      <td>Dinner with potential client Meals and Enterta...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>11/6/2016</td>\n",
       "      <td>Computer - Hardware</td>\n",
       "      <td>6</td>\n",
       "      <td>iPhone</td>\n",
       "      <td>200.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>26.00</td>\n",
       "      <td>1</td>\n",
       "      <td>iPhone Computer - Hardware</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>11/7/2016</td>\n",
       "      <td>Travel</td>\n",
       "      <td>1</td>\n",
       "      <td>Airplane ticket to NY</td>\n",
       "      <td>200.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>26.00</td>\n",
       "      <td>0</td>\n",
       "      <td>Airplane ticket to NY Travel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>12/18/2016</td>\n",
       "      <td>Travel</td>\n",
       "      <td>6</td>\n",
       "      <td>Airplane ticket to NY</td>\n",
       "      <td>1500.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>195.00</td>\n",
       "      <td>1</td>\n",
       "      <td>Airplane ticket to NY Travel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>12/15/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>4</td>\n",
       "      <td>Dinner with client</td>\n",
       "      <td>200.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>26.00</td>\n",
       "      <td>0</td>\n",
       "      <td>Dinner with client Meals and Entertainment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>12/1/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>4</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>210.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>27.30</td>\n",
       "      <td>0</td>\n",
       "      <td>Dinner Meals and Entertainment</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          date                 category  employee id  \\\n",
       "0    11/1/2016                   Travel            7   \n",
       "1   11/15/2016  Meals and Entertainment            1   \n",
       "2   11/30/2016      Computer - Hardware            3   \n",
       "3   11/14/2016      Computer - Software            3   \n",
       "6    12/9/2016  Meals and Entertainment            6   \n",
       "7   11/12/2016                   Travel            4   \n",
       "8   11/21/2016  Meals and Entertainment            7   \n",
       "9    10/4/2016                   Travel            6   \n",
       "10  10/12/2016      Computer - Hardware            7   \n",
       "14  12/30/2016  Meals and Entertainment            4   \n",
       "15   11/6/2016      Computer - Hardware            6   \n",
       "16   11/7/2016                   Travel            1   \n",
       "18  12/18/2016                   Travel            6   \n",
       "19  12/15/2016  Meals and Entertainment            4   \n",
       "20   12/1/2016  Meals and Entertainment            4   \n",
       "\n",
       "             expense description  pre-tax amount      tax name  tax amount  \\\n",
       "0                      Taxi ride            40.0  NY Sales tax        3.55   \n",
       "1                     Team lunch           235.0  CA Sales tax       30.55   \n",
       "2             HP Laptop Computer           999.0  CA Sales tax      129.87   \n",
       "3               Microsoft Office           899.0  CA Sales tax      116.87   \n",
       "6              Coffee with Steve           300.0  CA Sales tax       39.00   \n",
       "7                      Taxi ride           230.0  CA Sales tax       29.90   \n",
       "8                  Client dinner           200.0  NY Sales tax       17.75   \n",
       "9                Flight to Miami           200.0  CA Sales tax       26.00   \n",
       "10          Macbook Air Computer          1999.0  NY Sales tax      177.41   \n",
       "14  Dinner with potential client           200.0  CA Sales tax       26.00   \n",
       "15                        iPhone           200.0  CA Sales tax       26.00   \n",
       "16         Airplane ticket to NY           200.0  CA Sales tax       26.00   \n",
       "18         Airplane ticket to NY          1500.0  CA Sales tax      195.00   \n",
       "19            Dinner with client           200.0  CA Sales tax       26.00   \n",
       "20                        Dinner           210.0  CA Sales tax       27.30   \n",
       "\n",
       "   holiday                                          text_data  \n",
       "0        0                                   Taxi ride Travel  \n",
       "1        0                 Team lunch Meals and Entertainment  \n",
       "2        0             HP Laptop Computer Computer - Hardware  \n",
       "3        0               Microsoft Office Computer - Software  \n",
       "6        0          Coffee with Steve Meals and Entertainment  \n",
       "7        1                                   Taxi ride Travel  \n",
       "8        0              Client dinner Meals and Entertainment  \n",
       "9        0                             Flight to Miami Travel  \n",
       "10       0           Macbook Air Computer Computer - Hardware  \n",
       "14       1  Dinner with potential client Meals and Enterta...  \n",
       "15       1                         iPhone Computer - Hardware  \n",
       "16       0                       Airplane ticket to NY Travel  \n",
       "18       1                       Airplane ticket to NY Travel  \n",
       "19       0         Dinner with client Meals and Entertainment  \n",
       "20       0                     Dinner Meals and Entertainment  "
      ]
     },
     "execution_count": 250,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cluster1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 251,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>category</th>\n",
       "      <th>employee id</th>\n",
       "      <th>expense description</th>\n",
       "      <th>pre-tax amount</th>\n",
       "      <th>tax name</th>\n",
       "      <th>tax amount</th>\n",
       "      <th>holiday</th>\n",
       "      <th>text_data</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>11/6/2016</td>\n",
       "      <td>Computer - Software</td>\n",
       "      <td>4</td>\n",
       "      <td>Dropbox Subscription</td>\n",
       "      <td>50.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>6.50</td>\n",
       "      <td>1</td>\n",
       "      <td>Dropbox Subscription Computer - Software</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>11/3/2016</td>\n",
       "      <td>Computer - Software</td>\n",
       "      <td>3</td>\n",
       "      <td>Dropbox Subscription</td>\n",
       "      <td>50.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>6.50</td>\n",
       "      <td>0</td>\n",
       "      <td>Dropbox Subscription Computer - Software</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>12/11/2016</td>\n",
       "      <td>Computer - Software</td>\n",
       "      <td>1</td>\n",
       "      <td>iCloud Subscription</td>\n",
       "      <td>15.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>1.95</td>\n",
       "      <td>1</td>\n",
       "      <td>iCloud Subscription Computer - Software</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>9/18/2016</td>\n",
       "      <td>Travel</td>\n",
       "      <td>1</td>\n",
       "      <td>Taxi ride</td>\n",
       "      <td>60.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>7.80</td>\n",
       "      <td>1</td>\n",
       "      <td>Taxi ride Travel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>9/30/2016</td>\n",
       "      <td>Office Supplies</td>\n",
       "      <td>3</td>\n",
       "      <td>Paper</td>\n",
       "      <td>200.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>26.00</td>\n",
       "      <td>0</td>\n",
       "      <td>Paper Office Supplies</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>12/3/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>5</td>\n",
       "      <td>Starbucks coffee</td>\n",
       "      <td>4.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>0.52</td>\n",
       "      <td>1</td>\n",
       "      <td>Starbucks coffee Meals and Entertainment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>12/8/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>4</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>180.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>23.40</td>\n",
       "      <td>0</td>\n",
       "      <td>Dinner Meals and Entertainment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>12/31/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>4</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>1000.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>130.00</td>\n",
       "      <td>1</td>\n",
       "      <td>Dinner Meals and Entertainment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>12/9/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>4</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>30.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>3.90</td>\n",
       "      <td>0</td>\n",
       "      <td>Dinner Meals and Entertainment</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          date                 category  employee id   expense description  \\\n",
       "4    11/6/2016      Computer - Software            4  Dropbox Subscription   \n",
       "5    11/3/2016      Computer - Software            3  Dropbox Subscription   \n",
       "11  12/11/2016      Computer - Software            1   iCloud Subscription   \n",
       "12   9/18/2016                   Travel            1             Taxi ride   \n",
       "13   9/30/2016          Office Supplies            3                 Paper   \n",
       "17   12/3/2016  Meals and Entertainment            5      Starbucks coffee   \n",
       "21   12/8/2016  Meals and Entertainment            4                Dinner   \n",
       "22  12/31/2016  Meals and Entertainment            4                Dinner   \n",
       "23   12/9/2016  Meals and Entertainment            4                Dinner   \n",
       "\n",
       "    pre-tax amount      tax name  tax amount holiday  \\\n",
       "4             50.0  CA Sales tax        6.50       1   \n",
       "5             50.0  CA Sales tax        6.50       0   \n",
       "11            15.0  CA Sales tax        1.95       1   \n",
       "12            60.0  CA Sales tax        7.80       1   \n",
       "13           200.0  CA Sales tax       26.00       0   \n",
       "17             4.0  CA Sales tax        0.52       1   \n",
       "21           180.0  CA Sales tax       23.40       0   \n",
       "22          1000.0  CA Sales tax      130.00       1   \n",
       "23            30.0  CA Sales tax        3.90       0   \n",
       "\n",
       "                                   text_data  \n",
       "4   Dropbox Subscription Computer - Software  \n",
       "5   Dropbox Subscription Computer - Software  \n",
       "11   iCloud Subscription Computer - Software  \n",
       "12                          Taxi ride Travel  \n",
       "13                     Paper Office Supplies  \n",
       "17  Starbucks coffee Meals and Entertainment  \n",
       "21            Dinner Meals and Entertainment  \n",
       "22            Dinner Meals and Entertainment  \n",
       "23            Dinner Meals and Entertainment  "
      ]
     },
     "execution_count": 251,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cluster2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Observations:\n",
    "\n",
    "- One cluster has 15 data ponits, whereas the other has 9\n",
    "- The first one has fewer holidays, all the computer hardware stuffs and meals and entertainment that re mostly related to client, team etc, indicating that these are business expense.\n",
    "- The seond cluster has more holidays, meals and entertainment that look more personal, even though it has the **Office Supplies** record which clearly in the wrong cluster.\n",
    "- Overall the clusters kind of make sense."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try with pre-trained embedding vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 268,
   "metadata": {},
   "outputs": [],
   "source": [
    "kmeans = KMeans(n_clusters=2, random_state=1)\n",
    "kmeans.fit(x_train_final_embedding)\n",
    "clusters_train = kmeans.predict(x_train_final_embedding)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 253,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 0,\n",
       "       0], dtype=int32)"
      ]
     },
     "execution_count": 253,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clusters_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 254,
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster1 = df_train[clusters_train==0]\n",
    "cluster2 = df_train[clusters_train==1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 255,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>category</th>\n",
       "      <th>employee id</th>\n",
       "      <th>expense description</th>\n",
       "      <th>pre-tax amount</th>\n",
       "      <th>tax name</th>\n",
       "      <th>tax amount</th>\n",
       "      <th>holiday</th>\n",
       "      <th>text_data</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>11/6/2016</td>\n",
       "      <td>Computer - Software</td>\n",
       "      <td>4</td>\n",
       "      <td>Dropbox Subscription</td>\n",
       "      <td>50.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>6.50</td>\n",
       "      <td>1</td>\n",
       "      <td>Dropbox Subscription Computer - Software</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>11/3/2016</td>\n",
       "      <td>Computer - Software</td>\n",
       "      <td>3</td>\n",
       "      <td>Dropbox Subscription</td>\n",
       "      <td>50.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>6.50</td>\n",
       "      <td>0</td>\n",
       "      <td>Dropbox Subscription Computer - Software</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>12/11/2016</td>\n",
       "      <td>Computer - Software</td>\n",
       "      <td>1</td>\n",
       "      <td>iCloud Subscription</td>\n",
       "      <td>15.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>1.95</td>\n",
       "      <td>1</td>\n",
       "      <td>iCloud Subscription Computer - Software</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>9/18/2016</td>\n",
       "      <td>Travel</td>\n",
       "      <td>1</td>\n",
       "      <td>Taxi ride</td>\n",
       "      <td>60.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>7.80</td>\n",
       "      <td>1</td>\n",
       "      <td>Taxi ride Travel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>9/30/2016</td>\n",
       "      <td>Office Supplies</td>\n",
       "      <td>3</td>\n",
       "      <td>Paper</td>\n",
       "      <td>200.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>26.00</td>\n",
       "      <td>0</td>\n",
       "      <td>Paper Office Supplies</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>12/3/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>5</td>\n",
       "      <td>Starbucks coffee</td>\n",
       "      <td>4.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>0.52</td>\n",
       "      <td>1</td>\n",
       "      <td>Starbucks coffee Meals and Entertainment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>12/8/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>4</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>180.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>23.40</td>\n",
       "      <td>0</td>\n",
       "      <td>Dinner Meals and Entertainment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>12/31/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>4</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>1000.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>130.00</td>\n",
       "      <td>1</td>\n",
       "      <td>Dinner Meals and Entertainment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>12/9/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>4</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>30.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>3.90</td>\n",
       "      <td>0</td>\n",
       "      <td>Dinner Meals and Entertainment</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          date                 category  employee id   expense description  \\\n",
       "4    11/6/2016      Computer - Software            4  Dropbox Subscription   \n",
       "5    11/3/2016      Computer - Software            3  Dropbox Subscription   \n",
       "11  12/11/2016      Computer - Software            1   iCloud Subscription   \n",
       "12   9/18/2016                   Travel            1             Taxi ride   \n",
       "13   9/30/2016          Office Supplies            3                 Paper   \n",
       "17   12/3/2016  Meals and Entertainment            5      Starbucks coffee   \n",
       "21   12/8/2016  Meals and Entertainment            4                Dinner   \n",
       "22  12/31/2016  Meals and Entertainment            4                Dinner   \n",
       "23   12/9/2016  Meals and Entertainment            4                Dinner   \n",
       "\n",
       "    pre-tax amount      tax name  tax amount holiday  \\\n",
       "4             50.0  CA Sales tax        6.50       1   \n",
       "5             50.0  CA Sales tax        6.50       0   \n",
       "11            15.0  CA Sales tax        1.95       1   \n",
       "12            60.0  CA Sales tax        7.80       1   \n",
       "13           200.0  CA Sales tax       26.00       0   \n",
       "17             4.0  CA Sales tax        0.52       1   \n",
       "21           180.0  CA Sales tax       23.40       0   \n",
       "22          1000.0  CA Sales tax      130.00       1   \n",
       "23            30.0  CA Sales tax        3.90       0   \n",
       "\n",
       "                                   text_data  \n",
       "4   Dropbox Subscription Computer - Software  \n",
       "5   Dropbox Subscription Computer - Software  \n",
       "11   iCloud Subscription Computer - Software  \n",
       "12                          Taxi ride Travel  \n",
       "13                     Paper Office Supplies  \n",
       "17  Starbucks coffee Meals and Entertainment  \n",
       "21            Dinner Meals and Entertainment  \n",
       "22            Dinner Meals and Entertainment  \n",
       "23            Dinner Meals and Entertainment  "
      ]
     },
     "execution_count": 255,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cluster1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 256,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>category</th>\n",
       "      <th>employee id</th>\n",
       "      <th>expense description</th>\n",
       "      <th>pre-tax amount</th>\n",
       "      <th>tax name</th>\n",
       "      <th>tax amount</th>\n",
       "      <th>holiday</th>\n",
       "      <th>text_data</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11/1/2016</td>\n",
       "      <td>Travel</td>\n",
       "      <td>7</td>\n",
       "      <td>Taxi ride</td>\n",
       "      <td>40.0</td>\n",
       "      <td>NY Sales tax</td>\n",
       "      <td>3.55</td>\n",
       "      <td>0</td>\n",
       "      <td>Taxi ride Travel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11/15/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>1</td>\n",
       "      <td>Team lunch</td>\n",
       "      <td>235.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>30.55</td>\n",
       "      <td>0</td>\n",
       "      <td>Team lunch Meals and Entertainment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11/30/2016</td>\n",
       "      <td>Computer - Hardware</td>\n",
       "      <td>3</td>\n",
       "      <td>HP Laptop Computer</td>\n",
       "      <td>999.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>129.87</td>\n",
       "      <td>0</td>\n",
       "      <td>HP Laptop Computer Computer - Hardware</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>11/14/2016</td>\n",
       "      <td>Computer - Software</td>\n",
       "      <td>3</td>\n",
       "      <td>Microsoft Office</td>\n",
       "      <td>899.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>116.87</td>\n",
       "      <td>0</td>\n",
       "      <td>Microsoft Office Computer - Software</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>12/9/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>6</td>\n",
       "      <td>Coffee with Steve</td>\n",
       "      <td>300.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>39.00</td>\n",
       "      <td>0</td>\n",
       "      <td>Coffee with Steve Meals and Entertainment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>11/12/2016</td>\n",
       "      <td>Travel</td>\n",
       "      <td>4</td>\n",
       "      <td>Taxi ride</td>\n",
       "      <td>230.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>29.90</td>\n",
       "      <td>1</td>\n",
       "      <td>Taxi ride Travel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>11/21/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>7</td>\n",
       "      <td>Client dinner</td>\n",
       "      <td>200.0</td>\n",
       "      <td>NY Sales tax</td>\n",
       "      <td>17.75</td>\n",
       "      <td>0</td>\n",
       "      <td>Client dinner Meals and Entertainment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10/4/2016</td>\n",
       "      <td>Travel</td>\n",
       "      <td>6</td>\n",
       "      <td>Flight to Miami</td>\n",
       "      <td>200.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>26.00</td>\n",
       "      <td>0</td>\n",
       "      <td>Flight to Miami Travel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>10/12/2016</td>\n",
       "      <td>Computer - Hardware</td>\n",
       "      <td>7</td>\n",
       "      <td>Macbook Air Computer</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>NY Sales tax</td>\n",
       "      <td>177.41</td>\n",
       "      <td>0</td>\n",
       "      <td>Macbook Air Computer Computer - Hardware</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>12/30/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>4</td>\n",
       "      <td>Dinner with potential client</td>\n",
       "      <td>200.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>26.00</td>\n",
       "      <td>1</td>\n",
       "      <td>Dinner with potential client Meals and Enterta...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>11/6/2016</td>\n",
       "      <td>Computer - Hardware</td>\n",
       "      <td>6</td>\n",
       "      <td>iPhone</td>\n",
       "      <td>200.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>26.00</td>\n",
       "      <td>1</td>\n",
       "      <td>iPhone Computer - Hardware</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>11/7/2016</td>\n",
       "      <td>Travel</td>\n",
       "      <td>1</td>\n",
       "      <td>Airplane ticket to NY</td>\n",
       "      <td>200.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>26.00</td>\n",
       "      <td>0</td>\n",
       "      <td>Airplane ticket to NY Travel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>12/18/2016</td>\n",
       "      <td>Travel</td>\n",
       "      <td>6</td>\n",
       "      <td>Airplane ticket to NY</td>\n",
       "      <td>1500.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>195.00</td>\n",
       "      <td>1</td>\n",
       "      <td>Airplane ticket to NY Travel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>12/15/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>4</td>\n",
       "      <td>Dinner with client</td>\n",
       "      <td>200.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>26.00</td>\n",
       "      <td>0</td>\n",
       "      <td>Dinner with client Meals and Entertainment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>12/1/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>4</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>210.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>27.30</td>\n",
       "      <td>0</td>\n",
       "      <td>Dinner Meals and Entertainment</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          date                 category  employee id  \\\n",
       "0    11/1/2016                   Travel            7   \n",
       "1   11/15/2016  Meals and Entertainment            1   \n",
       "2   11/30/2016      Computer - Hardware            3   \n",
       "3   11/14/2016      Computer - Software            3   \n",
       "6    12/9/2016  Meals and Entertainment            6   \n",
       "7   11/12/2016                   Travel            4   \n",
       "8   11/21/2016  Meals and Entertainment            7   \n",
       "9    10/4/2016                   Travel            6   \n",
       "10  10/12/2016      Computer - Hardware            7   \n",
       "14  12/30/2016  Meals and Entertainment            4   \n",
       "15   11/6/2016      Computer - Hardware            6   \n",
       "16   11/7/2016                   Travel            1   \n",
       "18  12/18/2016                   Travel            6   \n",
       "19  12/15/2016  Meals and Entertainment            4   \n",
       "20   12/1/2016  Meals and Entertainment            4   \n",
       "\n",
       "             expense description  pre-tax amount      tax name  tax amount  \\\n",
       "0                      Taxi ride            40.0  NY Sales tax        3.55   \n",
       "1                     Team lunch           235.0  CA Sales tax       30.55   \n",
       "2             HP Laptop Computer           999.0  CA Sales tax      129.87   \n",
       "3               Microsoft Office           899.0  CA Sales tax      116.87   \n",
       "6              Coffee with Steve           300.0  CA Sales tax       39.00   \n",
       "7                      Taxi ride           230.0  CA Sales tax       29.90   \n",
       "8                  Client dinner           200.0  NY Sales tax       17.75   \n",
       "9                Flight to Miami           200.0  CA Sales tax       26.00   \n",
       "10          Macbook Air Computer          1999.0  NY Sales tax      177.41   \n",
       "14  Dinner with potential client           200.0  CA Sales tax       26.00   \n",
       "15                        iPhone           200.0  CA Sales tax       26.00   \n",
       "16         Airplane ticket to NY           200.0  CA Sales tax       26.00   \n",
       "18         Airplane ticket to NY          1500.0  CA Sales tax      195.00   \n",
       "19            Dinner with client           200.0  CA Sales tax       26.00   \n",
       "20                        Dinner           210.0  CA Sales tax       27.30   \n",
       "\n",
       "   holiday                                          text_data  \n",
       "0        0                                   Taxi ride Travel  \n",
       "1        0                 Team lunch Meals and Entertainment  \n",
       "2        0             HP Laptop Computer Computer - Hardware  \n",
       "3        0               Microsoft Office Computer - Software  \n",
       "6        0          Coffee with Steve Meals and Entertainment  \n",
       "7        1                                   Taxi ride Travel  \n",
       "8        0              Client dinner Meals and Entertainment  \n",
       "9        0                             Flight to Miami Travel  \n",
       "10       0           Macbook Air Computer Computer - Hardware  \n",
       "14       1  Dinner with potential client Meals and Enterta...  \n",
       "15       1                         iPhone Computer - Hardware  \n",
       "16       0                       Airplane ticket to NY Travel  \n",
       "18       1                       Airplane ticket to NY Travel  \n",
       "19       0         Dinner with client Meals and Entertainment  \n",
       "20       0                     Dinner Meals and Entertainment  "
      ]
     },
     "execution_count": 256,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cluster2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Observations:\n",
    "\n",
    "It turns out that we got the exact same clusters. So using pre-trained embeddings hasn't really helped too much in this case, may be because there wasn't enough data for the embeddings to take effect. Nevertheless, it was a good exercise for me to go through both approaches."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Just try the performance on the validation set\n",
    "\n",
    "Although not asked in the question, I am just trying out the performace on the validation set as well using the one that uses pre-trained embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 269,
   "metadata": {},
   "outputs": [],
   "source": [
    "clusters_val = kmeans.predict(x_val_final_embedding)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 270,
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster1 = df_val[clusters_val==0]\n",
    "cluster2 = df_val[clusters_val==1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 271,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>category</th>\n",
       "      <th>employee id</th>\n",
       "      <th>expense description</th>\n",
       "      <th>pre-tax amount</th>\n",
       "      <th>tax name</th>\n",
       "      <th>tax amount</th>\n",
       "      <th>holiday</th>\n",
       "      <th>text_data</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11/12/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>1</td>\n",
       "      <td>Dinner with Family</td>\n",
       "      <td>235.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>30.55</td>\n",
       "      <td>1</td>\n",
       "      <td>Dinner with Family Meals and Entertainment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>12/2/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>4</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>180.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>23.40</td>\n",
       "      <td>0</td>\n",
       "      <td>Dinner Meals and Entertainment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2/14/2017</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>4</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>500.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>65.00</td>\n",
       "      <td>0</td>\n",
       "      <td>Dinner Meals and Entertainment</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          date                 category  employee id expense description  \\\n",
       "1   11/12/2016  Meals and Entertainment            1  Dinner with Family   \n",
       "9    12/2/2016  Meals and Entertainment            4              Dinner   \n",
       "10   2/14/2017  Meals and Entertainment            4              Dinner   \n",
       "\n",
       "    pre-tax amount      tax name  tax amount holiday  \\\n",
       "1            235.0  CA Sales tax       30.55       1   \n",
       "9            180.0  CA Sales tax       23.40       0   \n",
       "10           500.0  CA Sales tax       65.00       0   \n",
       "\n",
       "                                     text_data  \n",
       "1   Dinner with Family Meals and Entertainment  \n",
       "9               Dinner Meals and Entertainment  \n",
       "10              Dinner Meals and Entertainment  "
      ]
     },
     "execution_count": 271,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cluster1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 272,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>category</th>\n",
       "      <th>employee id</th>\n",
       "      <th>expense description</th>\n",
       "      <th>pre-tax amount</th>\n",
       "      <th>tax name</th>\n",
       "      <th>tax amount</th>\n",
       "      <th>holiday</th>\n",
       "      <th>text_data</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11/10/2016</td>\n",
       "      <td>Travel</td>\n",
       "      <td>7</td>\n",
       "      <td>Taxi ride</td>\n",
       "      <td>30.0</td>\n",
       "      <td>NY Sales tax</td>\n",
       "      <td>2.66</td>\n",
       "      <td>0</td>\n",
       "      <td>Taxi ride Travel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9/2/2016</td>\n",
       "      <td>Computer - Hardware</td>\n",
       "      <td>4</td>\n",
       "      <td>Macbook Air Computer</td>\n",
       "      <td>4000.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>520.00</td>\n",
       "      <td>0</td>\n",
       "      <td>Macbook Air Computer Computer - Hardware</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9/2/2016</td>\n",
       "      <td>Office Supplies</td>\n",
       "      <td>4</td>\n",
       "      <td>Paper</td>\n",
       "      <td>20.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>2.60</td>\n",
       "      <td>0</td>\n",
       "      <td>Paper Office Supplies</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9/2/2016</td>\n",
       "      <td>Office Supplies</td>\n",
       "      <td>4</td>\n",
       "      <td>Pens</td>\n",
       "      <td>20.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>2.60</td>\n",
       "      <td>0</td>\n",
       "      <td>Pens Office Supplies</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>11/21/2016</td>\n",
       "      <td>Travel</td>\n",
       "      <td>1</td>\n",
       "      <td>Airplane ticket to Miami</td>\n",
       "      <td>200.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>26.00</td>\n",
       "      <td>0</td>\n",
       "      <td>Airplane ticket to Miami Travel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>12/4/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>2</td>\n",
       "      <td>Starbucks coffee</td>\n",
       "      <td>12.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>1.56</td>\n",
       "      <td>1</td>\n",
       "      <td>Starbucks coffee Meals and Entertainment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1/18/2010</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>6</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>30.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>3.90</td>\n",
       "      <td>0</td>\n",
       "      <td>Dinner Meals and Entertainment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>10/5/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>4</td>\n",
       "      <td>Dinner with client</td>\n",
       "      <td>220.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>28.60</td>\n",
       "      <td>0</td>\n",
       "      <td>Dinner with client Meals and Entertainment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2/14/2016</td>\n",
       "      <td>Meals and Entertainment</td>\n",
       "      <td>4</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>600.0</td>\n",
       "      <td>CA Sales tax</td>\n",
       "      <td>78.00</td>\n",
       "      <td>1</td>\n",
       "      <td>Dinner Meals and Entertainment</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          date                 category  employee id  \\\n",
       "0   11/10/2016                   Travel            7   \n",
       "2     9/2/2016      Computer - Hardware            4   \n",
       "3     9/2/2016          Office Supplies            4   \n",
       "4     9/2/2016          Office Supplies            4   \n",
       "5   11/21/2016                   Travel            1   \n",
       "6    12/4/2016  Meals and Entertainment            2   \n",
       "7    1/18/2010  Meals and Entertainment            6   \n",
       "8    10/5/2016  Meals and Entertainment            4   \n",
       "11   2/14/2016  Meals and Entertainment            4   \n",
       "\n",
       "         expense description  pre-tax amount      tax name  tax amount  \\\n",
       "0                  Taxi ride            30.0  NY Sales tax        2.66   \n",
       "2       Macbook Air Computer          4000.0  CA Sales tax      520.00   \n",
       "3                      Paper            20.0  CA Sales tax        2.60   \n",
       "4                       Pens            20.0  CA Sales tax        2.60   \n",
       "5   Airplane ticket to Miami           200.0  CA Sales tax       26.00   \n",
       "6           Starbucks coffee            12.0  CA Sales tax        1.56   \n",
       "7                     Dinner            30.0  CA Sales tax        3.90   \n",
       "8         Dinner with client           220.0  CA Sales tax       28.60   \n",
       "11                    Dinner           600.0  CA Sales tax       78.00   \n",
       "\n",
       "   holiday                                   text_data  \n",
       "0        0                            Taxi ride Travel  \n",
       "2        0    Macbook Air Computer Computer - Hardware  \n",
       "3        0                       Paper Office Supplies  \n",
       "4        0                        Pens Office Supplies  \n",
       "5        0             Airplane ticket to Miami Travel  \n",
       "6        1    Starbucks coffee Meals and Entertainment  \n",
       "7        0              Dinner Meals and Entertainment  \n",
       "8        0  Dinner with client Meals and Entertainment  \n",
       "11       1              Dinner Meals and Entertainment  "
      ]
     },
     "execution_count": 272,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cluster2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Observations:\n",
    "\n",
    "It seems that the performance on the validation set is not bad after all. Thefirst cluster hints mostly personal, whereas the second cluster hints mostly business expenses. Note that it has now put **Office Supplies** rightly in the sond cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion:\n",
    "\n",
    "There can be a myria of other ways we can possibly try to do the clustering. I have tried as much as I can in my limited time. A bit of improvement on my approach could be that we could try mutiple iterations of the K-means fitting using mutiple initial random state, and then then assign each data point to the cluster that they belong to the most number of times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}