first-place-simple-solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Beginner's Luck\n",
    "I just learned Machine Learning this summer and this is my first competition.\n",
    "\n",
    "Thanks to Kaggle and everyone participates in this competition. I learned a lot from you guys, which is more important than the ranking. Thanks for your sharing!\n",
    "\n",
    "My solution may not be the best one compared with others. but it's the **simplest** one I think. Here's what I did:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/kaggle/input/cat-in-the-dat/train.csv\n",
      "/kaggle/input/cat-in-the-dat/sample_submission.csv\n",
      "/kaggle/input/cat-in-the-dat/test.csv\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import os\n",
    "for dirname, _, filenames in os.walk('/kaggle/input'):\n",
    "    for filename in filenames:\n",
    "        print(os.path.join(dirname, filename))\n",
    "\n",
    "from sklearn.model_selection import KFold\n",
    "from sklearn.metrics import roc_auc_score as auc\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.preprocessing import OrdinalEncoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"
   },
   "outputs": [],
   "source": [
    "train = pd.read_csv(\"/kaggle/input/cat-in-the-dat/train.csv\")\n",
    "test = pd.read_csv(\"/kaggle/input/cat-in-the-dat/test.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## feature engineering\n",
    "Codes are based on Peter Hurford's great kernel: [Why Not Logistic Regression?](https://www.kaggle.com/peterhurford/why-not-logistic-regression)\n",
    "\n",
    "I did this in August, and got a 0.80845 score on leaderboard.\n",
    "\n",
    "#### Things I did are very simple: \n",
    "- Dropping bin_0\n",
    "- Ordinal encoding for the \"ord\" columns\n",
    "- One-hot encoding for \"nom\", \"day\" and \"month\" columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = train['target']\n",
    "train_id = train['id']\n",
    "test_id = test['id']\n",
    "train.drop(['target', 'id'], axis=1, inplace=True)\n",
    "test.drop('id', axis=1, inplace=True)\n",
    "\n",
    "mapper_bin_3 = {'T': 1, 'F': 0}\n",
    "mapper_bin_4 = {'Y': 1, 'N': 0}\n",
    "\n",
    "mapper_ord_1 = {'Novice': 1, \n",
    "                'Contributor': 2,\n",
    "                'Expert': 3, \n",
    "                'Master': 4, \n",
    "                'Grandmaster': 5}\n",
    "mapper_ord_2 = {'Freezing': 1, \n",
    "                'Cold': 2, \n",
    "                'Warm': 3, \n",
    "                'Hot': 4,\n",
    "                'Boiling Hot': 5, \n",
    "                'Lava Hot': 6}\n",
    "mapper_ord_3 = {'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, \n",
    "                'i': 9, 'j': 10, 'k': 11, 'l': 12, 'm': 13, 'n': 14, 'o': 15}\n",
    "mapper_ord_4 = {'A': 1, 'B': 2, 'C': 3, 'D': 4, 'E': 5, 'F': 6, 'G': 7, 'H': 8, \n",
    "                'I': 9, 'J': 10, 'K': 11, 'L': 12, 'M': 13, 'N': 14, 'O': 15,\n",
    "                'P': 16, 'Q': 17, 'R': 18, 'S': 19, 'T': 20, 'U': 21, 'V': 22, \n",
    "                'W': 23, 'X': 24, 'Y': 25, 'Z': 26}\n",
    "\n",
    "all_data = pd.concat([train, test])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_data['day'] = all_data['day'].apply(str)\n",
    "all_data['month'] = all_data['month'].apply(str)\n",
    "\n",
    "all_data['bin_3_oe'] = all_data['bin_3'].replace(mapper_bin_3)\n",
    "all_data['bin_4_oe'] = all_data['bin_4'].replace(mapper_bin_4)\n",
    "all_data['ord_1_oe'] = all_data['ord_1'].replace(mapper_ord_1)\n",
    "all_data['ord_2_oe'] = all_data['ord_2'].replace(mapper_ord_2)\n",
    "all_data['ord_3_oe'] = all_data['ord_3'].replace(mapper_ord_3)\n",
    "all_data['ord_4_oe'] = all_data['ord_4'].replace(mapper_ord_4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "ordinal_encoder = OrdinalEncoder()   \n",
    "data_ord_encoded = ordinal_encoder.fit_transform(all_data[['ord_5']])\n",
    "data_ord_encoded_PD = pd.DataFrame(data_ord_encoded, dtype=\"int64\")\n",
    "data_ord_encoded_PD.rename(columns={0:'ord_5'}, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop bin_0\n",
    "all_data.drop([\"bin_0\", \"bin_3\", \"bin_4\", 'ord_1','ord_2','ord_3','ord_4','ord_5'], axis=1, inplace=True)\n",
    "all_data = all_data.reset_index()\n",
    "all_data = pd.concat([all_data, data_ord_encoded_PD], axis=1)\n",
    "all_data.drop(\"index\", axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(300000, 16305)\n",
      "(200000, 16305)\n",
      "CPU times: user 4min 28s, sys: 976 ms, total: 4min 29s\n",
      "Wall time: 4min 29s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# One Hot Encode\n",
    "dummies = pd.get_dummies(all_data, drop_first=False, sparse=True, \n",
    "                        columns=['nom_0', 'nom_1', 'nom_2', 'nom_3', 'nom_4', 'nom_5','nom_7', 'nom_6', 'nom_8', 'nom_9',\n",
    "                                'day', 'month'])\n",
    "train_ohe = dummies.iloc[:train.shape[0], :]\n",
    "test_ohe = dummies.iloc[train.shape[0]:, :]\n",
    "\n",
    "print(train_ohe.shape)\n",
    "print(test_ohe.shape)\n",
    "\n",
    "train_ohe = train_ohe.astype(pd.SparseDtype(\"int\", 0))\n",
    "test_ohe = test_ohe.astype(pd.SparseDtype(\"int\", 0))\n",
    "\n",
    "train_ohe = train_ohe.sparse.to_coo().tocsr()\n",
    "test_ohe = test_ohe.sparse.to_coo().tocsr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## prediction\n",
    "\n",
    "\n",
    "After the competition ends, I saw this comment: [\"It's also for people who want to try out various hyper-parameter optimization algorithms. :-)\"](https://www.kaggle.com/c/cat-in-the-dat/discussion/105713#607779), so tuning hyper-parameter seems as important as feature engineering.\n",
    "\n",
    "I used C = 0.123 at first, it works well. the magic C: 0.123456789 may be the best C value for the perfect model still unknown, but does not suit my model.\n",
    "\n",
    "I tuned C and class_weight by using [optuna](https://www.kaggle.com/cuijamm/simple-onehot-logisticregression-score-0-80801).  \n",
    "Actually I don't think tuning class_weight is meaningful. when I use C=0.12385012421930243 and don't set class_weight, I got a higher score.\n",
    "\n",
    "Codes are based on Jamm's [kernel](https://www.kaggle.com/cuijamm/simple-onehot-logisticregression-score-0-80801). The original author is [Ant](https://www.kaggle.com/superant).\n",
    "\n",
    "The **solver 'liblinear'** performs best on my kernel. I think it's the point makes me get a high score with simple FE.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>300000</td>\n",
       "      <td>0.408090</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>300001</td>\n",
       "      <td>0.749326</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>300002</td>\n",
       "      <td>0.153948</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>300003</td>\n",
       "      <td>0.505720</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>300004</td>\n",
       "      <td>0.895816</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       id    target\n",
       "0  300000  0.408090\n",
       "1  300001  0.749326\n",
       "2  300002  0.153948\n",
       "3  300003  0.505720\n",
       "4  300004  0.895816"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model=LogisticRegression(C=0.09968474250024324, class_weight={0:1, 1:1.3267279323409777},max_iter=10000, solver='liblinear')\n",
    "model.fit(train_ohe, target)\n",
    "predictions=model.predict_proba(test_ohe)[:,1]\n",
    "submission = pd.DataFrame({'id': test_id, 'target': predictions})\n",
    "submission.to_csv('submission.csv', index=False)\n",
    "submission.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}