what-makes-a-super-spender.ipynb

{"cells":[{"metadata":{"_uuid":"c50ee8266e6fba08692d8fe7b4309e3729275dd1"},"cell_type":"markdown","source":"## *What Makes a Super Spender?*\n_By Nick Brooks, September 2018_\n\nBuilding off what I learned from the SRK's [notebook](https://www.kaggle.com/sudalairajkumar/simple-exploration-baseline-ga-customer-revenue).."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"1a339e7488b968b1ab994b6217cbc1541fbde44b"},"cell_type":"code","source":"import os\nimport json\nimport numpy as np\nimport pandas as pd\nimport gc\nfrom pandas.io.json import json_normalize\n\n# Visualization\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom IPython.display import display\n%matplotlib inline\n\nfrom sklearn import preprocessing\n\n# Load Function by Julian - https://www.kaggle.com/julian3833/1-quick-start-read-csv-and-flatten-json-fields/notebook\ndef load_df(csv_path='../input/train.csv', nrows=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    df = pd.read_csv(csv_path, \n                     converters={column: json.loads for column in JSON_COLUMNS}, \n                     dtype={'fullVisitorId': 'str'}, # Important!!\n                     nrows=nrows)\n    \n    for column in JSON_COLUMNS:\n        column_as_df = json_normalize(df[column])\n        column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0e61fa07e12150e58cb904f649cfbd487c9269ad","_kg_hide-input":true},"cell_type":"code","source":"%%time\ndebug = False\nif debug is True: nrows = 5000\nelse: nrows = None\n\ntrain_df = load_df(nrows = nrows)\ntest_df = load_df(\"../input/test.csv\", nrows = nrows)\n\ndisplay(train_df.sample(4))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"93ae8ec9017fcc296bb621219cc43a0629fc6de2"},"cell_type":"markdown","source":"## Minority Super-Spenders\n\nNoticing this stark contrast in customers, I want to explore the data in terms of spenders and non-spenders. My hopes is to capture the bigger picture."},{"metadata":{"trusted":true,"_uuid":"0adda9ea6c12bcd6e6ee241ae1afd515ca183aec","_kg_hide-input":true},"cell_type":"code","source":"train_df[\"totals.transactionRevenue\"] = train_df[\"totals.transactionRevenue\"].astype('float')\ngdf = train_df.groupby(\"fullVisitorId\")[\"totals.transactionRevenue\"].sum().reset_index()\n\nplt.figure(figsize=(8,6))\nplt.scatter(range(gdf.shape[0]), np.sort(np.log1p(gdf[\"totals.transactionRevenue\"].values)))\nplt.xlabel('index', fontsize=12)\nplt.ylabel('TransactionRevenue', fontsize=12)\nplt.show()\n\n# Distinguish Groups\nspending_visitors = gdf.loc[gdf[\"totals.transactionRevenue\"] > 0, \"fullVisitorId\"]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4f4dbc51281961d84fadda411751614ff25b2c7f"},"cell_type":"markdown","source":"## Pre-Processing"},{"metadata":{"trusted":true,"_uuid":"9cf90d5c9ca66adfbf236a21d545b6b21a2f5256","_kg_hide-input":true},"cell_type":"code","source":"# Dropping columns with singular outcome (including NA)\ndrop_cols = [c for c in train_df.columns if train_df[c].nunique(dropna=False)==1 ]\nprint(\"Columns with one outcome to drop:\\nCount: \", len(drop_cols),\"\\n\",drop_cols)\ntrain_df.drop(drop_cols, axis = 1, inplace = True)\ntest_df.drop([x for x in drop_cols if x in test_df.columns], axis = 1, inplace = True)\n\n# Isolate Dependent Variable and Merge\ny_name = 'totals.transactionRevenue'\nprint(\"Dependent Variable Name: \", y_name)\ny = train_df[y_name]\ntrain_df.drop(y_name, axis= 1, inplace= True)\n\n# Create Train/Test Variable..\ntrain_df[\"istrain\"] = True\ntest_df[\"istrain\"] = False\n\n# Align Train / Test Columns\nnotintest = set(train_df.columns).difference(set(test_df.columns))\nprint(\"Variables not in test but in train : \", notintest)\ntrain_df.drop(notintest, axis=1, inplace=True)\n\n# Combine Train Test and Remove High Missing..\nprint(\"Test and Train columns Match?: \", all(train_df.columns == test_df.columns))\ndf = pd.concat([train_df, test_df], axis = 0)\ndel train_df, test_df; gc.collect();\n\n# Remove columns with high missing count\nhigh_miss_cols = [c for c in df.columns if df[c].isnull().sum() > (df.shape[0]*.5)]\nprint(\"High Missing Count Columns (50% + Miss):\\n\", high_miss_cols)\ndf.drop(high_miss_cols, axis =1, inplace= True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c3b8372d8d874b1fa5ddb683e081c10affd0bb4c"},"cell_type":"markdown","source":"## What Makes a Super Spender? Explore Categorical Variables\n\nWhat are the subcategory characteristics of super spenders? This is what these table showcase. Units are **Percentages**. Looking at the **True** columns, the **higher** the percentage, the **more** superspenders in the category.\n\nLimitations: Class representation is not balanced, therefore this may over-emphasize unusual classes."},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"9e7e799e44f4ab267cef7e05cb15691cacbd88ca","_kg_hide-input":true},"cell_type":"code","source":"# Remake Train Set now that processing is done..\ntrain = pd.concat([df.loc[df[\"istrain\"] == True, :], y], axis = 1)\ntrain[\"spender\"] = False\ntrain.loc[train[\"fullVisitorId\"].isin(spending_visitors), \"spender\"] = True\n\n# Categorical Variables to Explore\ncategorical_vars = [\"channelGrouping\", \"device.browser\", \"device.deviceCategory\",\"device.isMobile\",\n                    \"device.operatingSystem\",\"geoNetwork.city\",\"geoNetwork.continent\",\"geoNetwork.country\",\n                    \"geoNetwork.metro\",\"geoNetwork.region\",\"geoNetwork.subContinent\"]\n\n# Colored CrossTab\nfor cols in categorical_vars:\n    print(\"{}\".format(cols.title()))\n    temp = pd.crosstab(train[cols], train[\"spender\"], dropna=False, normalize = \"index\").mul(100).round(2)\n    temp = temp.sort_values(by=True,ascending= False)[:10]\n    display(temp.style.background_gradient(cmap = sns.light_palette(\"purple\", as_cmap=True)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"967b245e6da61bff90ed435ea90b500c0b9009a4"},"cell_type":"markdown","source":"## Deeper Multivariate Interactions"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"88450a0079ee81d41d9b307d82a8f42a82864da0"},"cell_type":"code","source":"train[\"spender\"] = train[\"spender\"].astype(\"int\") # Binary\nprint(\"Browser and Grouping\")\ndisplay(pd.pivot_table(train, values=\"spender\",index=\"device.browser\",columns=\"channelGrouping\").mul(100).round(2).style.background_gradient(cmap = sns.light_palette(\"red\", as_cmap=True)))\nprint(\"Device and Grouping\")\ndisplay(pd.pivot_table(train, values=\"spender\",index=\"device.deviceCategory\",columns=\"channelGrouping\").mul(100).round(2).style.background_gradient(cmap = sns.light_palette(\"blue\", as_cmap=True)))\nprint(\"Is Mobile and Grouping\")\ndisplay(pd.pivot_table(train, values=\"spender\",index=\"device.isMobile\",columns=\"channelGrouping\").mul(100).round(2).style.background_gradient(cmap = sns.light_palette(\"orange\", as_cmap=True)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2b1d3c7f187e4410003b70764611304745bb6e5e"},"cell_type":"markdown","source":"### Next\n- Explore the continuous variables"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}