p2_ab_testing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analyze A/B Test Results\n",
    "## Project 2: Practical Statistics\n",
    "\n",
    "## Table of Contents\n",
    "- [Introduction](#intro)\n",
    "- [Part I - Probability](#probability)\n",
    "- [Part II - A/B Test](#ab_test)\n",
    "- [Part III - Regression](#regression)\n",
    "- [Conclusions](#conclusions)\n",
    "\n",
    "<a id='intro'></a>\n",
    "## Introduction\n",
    "\n",
    "This project covers a span of topics related to practical statistics ranging from probability and hypothesis testing to regression. Common methods used by data analysts and data scientists in the field, which cover these topics, are A/B tests. \n",
    "\n",
    "In this project, analytical methods which leverage concepts from these topics were be applied to understanding the results of an A/B test run by an e-commerce website. The company had developed a new web page in order to increase the number of users who \"convert\" or decide to pay for the company's product. The analysis performed in this project aimed to provide the company with insights into whether the company should implement the new page, keep the old page, or run the experiment longer to make a decision.\n",
    "\n",
    "<a id='probability'></a>\n",
    "## Part I - Probability\n",
    "\n",
    "Libraries which were necessary to perform the analyses for the first part of this project were imported as shown below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import random\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "random.seed(42) # use seed = 42 to reproduce the results obtained from the analyses in this project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Data Wrangling \n",
    "#### 1.1 General Properties\n",
    "* The dataframe object `df` which includes the data from `ab_data.csv` contains five columns and 294478 rows.\n",
    "* The dataset contains data for a total of 290584 unique users.\n",
    "* Because the number of rows in the dataset is greater than the number of unique users, the `user_id` column in the dataset must contain duplicates.\n",
    "* Approximately 12.1% of users converted or decided to pay for the company's product.\n",
    "* 3893 rows in the dataset show incorrect assignment of `group` and `landing_page`. Either the `treatment` group is assigned to an `old_page` or the `control` group is assigned to a `new_page`.\n",
    "* The dataset does not contain any null values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0\n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0\n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0\n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0\n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import the dataset and display top few rows\n",
    "df = pd.read_csv('data/ab_data.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(294478, 5)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of rows and columns in the dataset\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "290584"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of unique users in the dataset\n",
    "df.user_id.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.12104245244060237"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# proportion of users converted\n",
    "df.query('converted == 1').user_id.nunique() / df.user_id.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3893"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of times the new_page in the landing_page and the treatment in the group do not line up\n",
    "df[((df['group'] == 'treatment') == (df['landing_page'] == 'new_page')) == False].shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of rows with missing values\n",
    "df.isna().sum(axis = 1).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.2 Data Cleaning\n",
    "1. Incorrect assignment of `group` and `landing_page`\n",
    "    * The 3893 rows which show incorrect assignment were removed from the dataset `df` because the incorrect assignment renders these records inappropriate for analyzing the A/B test results.\n",
    "    * The updated dataset was assigned to a new object `df2`.\n",
    "    * The total number of rows decreased by 3893, from 294478 to 290585.\n",
    "    * The new dataframe object `df2` does not contain any rows with incorrectly assigned `group` and `landing_page`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(290585, 5)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# updated dataframe with correctly assigned group and landing_page\n",
    "df2 = df.drop(df[((df['group'] == 'treatment') == (df['landing_page'] == 'new_page')) == False].index)\n",
    "df2.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# validate removal of rows with inaccurate assignment of group and landing_page\n",
    "df2[((df2['group'] == 'treatment') == (df2['landing_page'] == 'new_page')) == False].shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Duplicate value in `user_id` column in the new dataframe object `df2`\n",
    "    * A single user is duplicated once because there is one more total number of rows than the number of unique users.\n",
    "    * Because the two rows with this duplicate `user_id` of 773192 contain identical data across columns, the second row was removed from `df2`.\n",
    "    * After the dataframe `df2` was updated, the total number of rows became equal to the total number of unique users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "290584"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of unique users in the updated dataframe\n",
    "df2.user_id.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "773192\n"
     ]
    }
   ],
   "source": [
    "# duplicate user_id\n",
    "user = {}\n",
    "for userid in df2.user_id:\n",
    "    if userid not in user:\n",
    "        user[userid] = 1\n",
    "    else: # more than one instance of a user_id\n",
    "        user[userid] += 1\n",
    "        duplicate_user_id = userid\n",
    "        print(duplicate_user_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1899</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-09 05:37:58.781806</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2893</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-14 02:55:59.590927</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      user_id                   timestamp      group landing_page  converted\n",
       "1899   773192  2017-01-09 05:37:58.781806  treatment     new_page          0\n",
       "2893   773192  2017-01-14 02:55:59.590927  treatment     new_page          0"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# rows containing duplicate user_id\n",
    "df2.query('user_id == @duplicate_user_id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove duplicate from df2\n",
    "ind = df2.query('user_id == @duplicate_user_id').index[1] # second of the two rows containing duplicate user_id\n",
    "df2.drop(labels = ind, inplace = True)\n",
    "df2.reset_index(inplace = True, drop = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(290584, 5)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# dimensions of updated dataframe\n",
    "df2.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Exploratory Data Analysis\n",
    "The probability of the users in the `control` group to convert was approximately 0.2% larger than the probability of the users in the `treatment` group to convert. Because the proportion of each group was nearly the same, the difference in the probability may appear to suggest that the `old_page` is still more effective than the `new_page` in converting the users.\n",
    "\n",
    "However, in terms of implementing a new page or keeping the current one, 0.2% difference in the probability may not hold any practical significance. In fact, the statistical significance of this difference remains uncertain given the 12.0% probability of users converting regardless of the group the user is in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11959708724499628"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# probability of a user converting regardless of the user's group\n",
    "(df2['converted'] == 1).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1203863045004612"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# probability of a user in the control group converting\n",
    "obs_p_old = (df2.query('group == \"control\"')['converted'] == 1).mean()\n",
    "obs_p_old"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11880806551510564"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# probability of a user in the treatment group converting\n",
    "obs_p_new = (df2.query('group == \"treatment\"')['converted'] == 1).mean()\n",
    "obs_p_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5000619442226688"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# probability of a user receiving a new page\n",
    "(df2['landing_page'] == 'new_page').mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='ab_test'></a>\n",
    "## Part II - A/B Test\n",
    "\n",
    "Given the time stamp recorded for each user's transaction, the hypothesis test can be performed continuously as each observation is observed. However, the following questions posed challenges to this approach.\n",
    "* Should the test stop as soon as one page is considered significantly better than another or does it need to happen consistently for a certain amount of time?\n",
    "* How long should the test be run to render a decision that neither page is better than another?  \n",
    "\n",
    "For simplicity, time factor was excluded from the scope of the analysis.\n",
    "\n",
    "### 1. Hypotheses\n",
    "A/B test was performed under the assumption that the old page yields a larger convert rate than the new page unless the new page proves to definitely yield a larger convert rate at a Type I error rate of 5%. Based on this assumption, \n",
    "* for the Null Hypothesis, the probability of all users who convert from landing on the old page is greater than or equal to the probability of all users who convert from landing on the new page.\n",
    "* for the Alternative Hypothesis, the probability of all users who convert from landing on the new page is greater than or equal to the probability of all users who convert from landing on the old page.\n",
    "$$H_0: p_{new} - p_{old} \\leq 0$$\n",
    "$$H_1: p_{new} - p_{old} > 0$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Sampling Distribution of Differences in Convert Rates\n",
    "Under the null hypothesis, assume that the two probabilities $p_{new}$ and $p_{old}$ are equal to the convert rate observed in `ab_data.csv` regardless of the page. To note that these probabilities are based on the null hypothesis, `null_` was added to the names of the objects as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11959708724499628"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# convert rates for each group under the null\n",
    "null_p_new = (df2['converted'] == 1).mean() # control\n",
    "null_p_old = null_p_new # treatment\n",
    "null_p_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To simulate outcomes (`convert` value of 0 or 1) from the null hypothesis for each page, the sample size for each page equal to the sizes of each page in `ab_data.csv` was used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "145310"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# sample size of the new page\n",
    "n_new = df2.query('landing_page == \"new_page\"').shape[0]\n",
    "n_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "145274"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# sample size of the old page\n",
    "n_old = df2.query('landing_page == \"old_page\"').shape[0]\n",
    "n_old"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each of the two groups, one with `new_page` and another with `old_page`, outcomes of conversion indicated by 1's for converted and 0's for not converted were simulated and stored in the objects `new_page_converted` and `old_page_converted`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# simulation of n_new outcomes with a convert rate of null_p_new\n",
    "new_page_converted = np.random.binomial(n = 1, p = null_p_new, size = n_new)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# simulation of n_old outcomes with a convert rate of null_p_old\n",
    "old_page_converted = np.random.binomial(n = 1, p = null_p_old, size = n_old)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this simulation, the difference in the convert rates between the two groups was approximately 0.02%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0001838954631202122"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# simulation of difference in covert rates between groups\n",
    "p_diff = new_page_converted.mean() - old_page_converted.mean()\n",
    "p_diff"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This simulation of calculating an estimate from the null was conducted 10000 times to perform the sampling distribution for the difference in the convert rates between the two pages. The sampling distribution of the 10000 differences was stored in `p_diffs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sampling distribution of 10000 differences in convert rates between groups\n",
    "p_diffs = []\n",
    "\n",
    "new_converted_sim = np.random.binomial(n = n_new, p = null_p_new, size = 10000)/n_new # 10000 convert rates of control group\n",
    "old_converted_sim = np.random.binomial(n = n_old, p = null_p_old, size = 10000)/n_old # 10000 convert rates of treatment group\n",
    "p_diffs.append(new_converted_sim - old_converted_sim) # 10000 differences in convert rates between groups"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given the sufficiently large size of the sample and the fact that the difference in the convert rates is a random variable, the distribution of the differences should be approximately normal according to the Central Limit Theorem. The histogram of the differences estimated from the null hypothesis confirms this expectation. The normal distribution is centered at 0 with a standard deviation of 0.0012."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-5.904008555009425e-06, 0.0011993370375277307)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# mean and standard deviation of the sampling distribution\n",
    "p_diffs_mean = np.array(p_diffs).mean()\n",
    "p_diffs_std = np.array(p_diffs).std()\n",
    "p_diffs_mean, p_diffs_std"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdoAAAEuCAYAAAAp59OgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xe8HVW5//HPN3RUmgSIlBsEFJCrgLkUC0Z6k6ZeuDbAggVEfqJXimhE8YINRbx6ETA0QRQRpBoQsFFFeo0QJCSQAFJCiZTn98ezNplM9j5nnzI5Jznf9+u1X+fMmjUza+ozZc0aRQRmZmbWjFFDXQAzM7OFmQOtmZlZgxxozczMGuRAa2Zm1iAHWjMzswY50JqZmTXIgRaQtJukP0iaIek5SQ9I+o2k7Ye6bJ1ImihpSqV7rKSQtM8QlGVKmXZIelHSY5KulXS0pLFt8s9V9i6nMUHSlv0o18RK9z6ljGv3ZTz9KVd/5nF+krR5WUfPlGWyYYd8q0n6oaSrJT1b8o7tkHdJSd+WNL3sR1dL2qJNvlGSDi3r53lJN0t6b4dxfkLSXZJmS7pb0qc65NtN0t/K+B6Q9GVJi3SxHCZUtt2Q9LSkeyX9XNJ2bfKPL/nG1+bn+2W+X5b0m5K+rqTfS3qqDLNbb+WxhVREjOgfcCAQwEnAe4AtgU8AvwG+NdTl66HcE4Eple4lgM2A0UNQlinAJWX6mwM7AIeX9FnA7rX8awEb9XEaAXyjj8NsBKxV6d6njGftQZz3tuXqzzzO53V2B/DXsr1vBizdId944BHgIuDSMr9jO+Q9A3ii7D9bAb8GngM2rOU7CpgNfAF4N/B/wMvAjrV8nyjpR5V83yjdn67l2w54CTih5Ps88DxwTBfLYUKZp7eX5bAl8DHg4pJ+GjCqkn+Zkm+ZStp/lryfL9v/G0r6RWUf2L4Ms/xQr3f/huY35AUY6h/wD+DcDv1Gzc+y9LHccwXaIS7LFOD0NumvBv4MPAusNsBpdB1ogSU6pM+3QDucf+SdrJeAI7vJW/n/450CLfCW0m/fStqiwN3A+ZW0lUqQ/Vpt+MuBW2rDzgBOqeU7GXgUWKyS9jfgqlq+rwD/AlbpZf5agXbRNv3+X+l3cC/j+GrJN6qWfj9w6iCut7bbtX/D/zfkBRjqH3nF9eMu8o0mz7zvKYHjQeDnwKq1fK0dd13yCuCZEsz3Lf0/DNxVpnsFlSuu0n8KcDp5Nj+ZPDO/EXh3Ld9cgRYYW6a7Ty3PVPLK7o+l3PcCn2ozf1uXA9bzZbofr0+jh2UzhTaBtvRrHYC/3kPZFwW+Dvy9TP9R4E/AO0r/aPObUJvHzYG/kFdQP6iUa2JlOvuUYbcg71jMAh4DfgQsVck3vuQbX5uX1vBjuyzXlNrwY4BTy/zNBm4BPtRhGpuRV4hPAdOA44Alu1gXywDHl2Fmk4Hu/wGqjb/663Udl2F7CrRHkIFt6Vr610o5lqhs/wGsU8u3b0lfs3S/s3RvU8v37pL+7tK9eun+RC3fmtQCf4d5mkCHQFv63wg82GnbKNtYfXm2W8ZR2yfOB/5Jbq9/Bt7ZZv9uu12X/p8AbmbO/nISsEJtHEHeBTiQDPpPA1cBb2ozn7uXcswq29x1wC61ffRQ8tg1u2xf361uk/SyH4/kn5/R5ga1t6QvSnpDD/lWIDeeQ8lbQV8E1gH+LGnJNvl/CVwI7EbeojtZ0jeBTwOHkAeWN5LBuu5d5G2ow4G9yA37Yklv7PvssUyZxunArsD1wI8lvbuVQdL6payzyvQOAz5H3kYbkIi4mdwp395Dti+RweA48jbgvuQVzgql/+bl78Ty/+bAiZXhlwXOAs4kb1u3W6ZVp5MnE3sAx5IHrR93Mz81vZXrFZJeRR7kdiCX727ArcBpkvZrM8hp5AFrj1K2/cltryNJo8j1uC95EHwPeUv/e+TtV0r/d5T/Typl3r3HuezOm4D7I+LZWvrtwOLA2pV8s8nlX88HsH4lH8Bt/ckXEfeTJ5brMzAXA6tJWqND/93J9Q9ztoEryt+Z5O3jVjqSNiYD5wrkdvde8mTvMklvrY277XYt6Wjgf4HLgF3IY9H25DGi/lz6Q8BO5P68L7AGcJ6kRVsZJH2WvM0/A9gbeD9wLnny3nI68OVShp2A/yFvsZ9RydPbfjxyDXWkH+of8AbyyqJ15vkouWFv28twizDnbHr3SvqEkvaRStrywIvkDlV9ttN6PvxvlbQp5JXBGpW01wCPA6dV0ibS3RXtK2f/JW2JMo8nVNJ+Th4Ulq6kjSFPLKb0tBwqZW57RVv6Xw3c2UPZLwB+3cs0gvbPQlvzuGuHck2sdO9T8v6klu9w8lZq69naeLq4ou2iXNV5PKDDOC8jD3CL1KZRv7V6AXBPL8to5/o2UNJPJIPbiqV7USpX333YV3q6ov0dcE2b9K3LMO8s3ScAD7fJt3bJ9+HSfVjpXrKWr1X2I0r3B0r3um3GORU4qZd5mkDPV7SfLP037bRtkFeN0WH6E2tplwN3AotX0hYpab/pbbsm9/OXgK/U0t9e8u9W2zbvZe7b7O8r6W8r3cuQV7od9z/m3F34SC39gyV9w8o22uN+PFJ/I/6KNiLuIW+tvos867+JPEu9VNKXq3klfbrUkJxFBs5/lF7trjQvrkzjn+TB9JqIeKqS567yd/XasNdERGvcRMTT5JXI5vTdsxFxRWVcs8mdr3qGvhlwUVSuRiJiOnnmPRhE7pCdXA/sKOkoSe+QtHgfx/8iuZN36+xa91nkc8tN+jjdvtgCeCgirqyln04+lqhfeV1Y676VuddZp2m8TJ4o1qexOP3bfrrVaR1rAPnokLfbfPVx9ke35eh9RNJS5HHml8DLkhYtV5YiT7jqNbTbbdfbkNvqGa3hyziuJW/51scxKSJeqHTfWv62tqW3kXUpTuih6NuTJ//n1Kb5u9K/Nc2B7scLrREfaAEi4qWI+ENEfDkitgZeT26QX5W0PLxye6V1u2YP8qC8WRlFu1vH/6x1/6tDWrvhH2kzvkeAVbuYnd7KAXl1U53mGPJEoN00B8PqwPQe+n+TrFCyC/ks+TFJP5O0YpfjnxERL/WhPPX5anX3Z/l2awXaL4OHK/2rHq91zybvRvQ2jcfLyVQ30xhMj3cY//KV/q2/y0uqB8F2+WgzzhW6zAewHPMux75qnQT3tP12awXy6vUI4IXa7wByuVSPye2265XK38ltxrEM8Npa/nbbEczZ/1v5p/ZQ7pXIE7VZtem1jhmtcQx0P15oLdp7lpEnIqZJOhH4Afkc9jry2eXlEXFwK5+kNRsqwsod0h5qaHrTmbMD91aOPinvZ76ODs8uAcoZ9zHAMZJWIW+Bfg9YGtizi8n09WpjZeY862t1w5zl+3z5Wz8jrx/E+uJx2t/5WKX8fWwA465OYwVJi0fEvyrpgzmNTm4Hdpe0dMz9nHZ98oRyciXfEuTrT5Nr+SBfO2rlg3wGO73LfFe3MpV3fZeu5OuvHYF/RMSDAxwP5KtPL5OV705tlyEiXq52tsnSWofb0v4kuq/r+NHyd1XmfR5eHefz5C3kdqbBoOzHC60Rf0UrqX7btmXd8rd1NbA0eRZXtW8jhYLNquWS9BqyAsLVnQcZkGvIWz5LV6Y5hp4rMPVK0qvJg8qzZI3tXkXEwxFxInnnYINKr38BSw2kPBX/WeveizwAXle6Hyh/N6jl27HNuLot11VkpZr6Mv0AeWVwZxfj6GYao8jKLFUfJMt5zSBMo5PzgcWq0y63F/cEfle5yr6klOWDteE/BNwWWYkJclt/tEO+x8kaspRHLDd3yPcClUc4fSXp/wEbksFiwCLiGfJK7y3AjRFxQ/3XxWgmkdvqGu2Gryy/bv2FvFJtVyGv5RLyCnjZDtOc1mZeO+3HI5KvaOE2SVeQtezuJ2+/7Ah8Cji78qz0EuBLkg4jD8hbkhULmvAI8DtJE8hbPV8CXkVWnW/CN8h5uVTSd8grjiNKOV7uacCKFSVtRj5vWhbYmKxVORr4r3Y7Y4uk88iD5Y3kWfpG5HOhanC+A9hJ0iUlz7SextmLHSV9m3zGtAl5u+vU8ryeiJgu6SrgUEmPkoHwQ+RVWF235ZpI1vz8taTDyVt1HySfuX2yj7e+O7mYfJ3iJ5JGk1d7O5KVmP4nIh7taeBOJLW281at2B0kzQRmRsRVABFxk6RfAN+XtBi5L32afM3mlSAYETMkHUsu26fJdb4nuT/tWsn3gqQjgP+V9BB5wN4S+Cjw2doV+2HABZL+j3w+vRFZQ/YHEfEw3dlU0ktkQHk9uT/sAJxC1qIdLJ8H/kDuayeRV+srkvvLIhFxSE8DR8TfJR0DHF/eQriKvNpcndyWTqzWyehNRDwt6VDgh5LOIWsRP02eYDwfET+MiCslnQn8StL3yOPfy2TFrB2BL0XEPV3uxyPTUNfGGuofGVDPJ69iniffe/0b8N/MXTNwKfI1i5nkhngBc97Vm1DJN4E2tRhpUzOXOTUYt67nIw+OfycD7d+ALWvDTqTL92jbzPOVwJW1tG3IimCzgfvI2pbnAn/rYhlOYU6t7ZfInex64GgqNap7KPvB5NXWY+T7gneX5VitLfl28jWp56vLvNM8Vso1sdK9D3Peoz2PPJN/nNp7tCXvasBvydt9D5PPn+apddtLuabUxjmGfG2nm/do166lT6BNzdY289x6j3Y6eeV4D5X3aEuePtU6rqzb+q++DS1FXv09XJbHtdRqWZd8i5CB8IHKcnhfh2l/ssxDqxLfZzrk24M8yM8mKyl+hVKTu5d5m1Cbp2fIW9o/B7Zrk388A6h1XNLXIyvgzSjlnUoeg3as5JlIh+269P8wuc88U7bjO8t6X62SJ6jViKfNcaKkv6+sr+fISlXXAjtX+o8iTxRb7+4+Wf7/FnmlC13sxyP113qJ3YYJZfu4f4qIDw1xOV5NHnAujIiPDWVZzMwWZL51bABI+iH5vGYaWXnpc2RN0B8MZbnMzBZ0DrTWsiRZY3Bl8pbjdeQt7VuGtFRmZgs43zo2MzNr0Ih/vcfMzKxJDrRmZmYN6jXQSpogadDvL0vaQdIFkmZIekHSI5LOlzQYXxIZUpLGl+XW1YmMpCmSovJ7QtIkSe/ofei241uuTH/j/gzfX61tpfplkAGOr0/LcUEjaTFJn5H057LOZ0u6X9LJTa47SQdJ2qOhcUd5/3vI9XbskrRPbb/r9JswH4uNpPdJOnB+TrM/+lNOSStJOkbSbZKekfS8pMmSJvb3eLcg6OYAdiKD3Bi5pO+Sn496jmzjc6vy9wngbElvGczpDYHxZCMIfQkQl5LL+e1kjd+1gItKU3J9tVyZ/nwNtA0YT9+X4wJB+dm8y8nP2V1HNuqwLflO5pqlX1MOIt87HelaH+po/fYv6QfW0js2H9qQ95UyDHd9KmdpjvUWsjW008lPRe5Avnv9RuCPkpZtoJxDrtcrj4iYSs8NTveJpA+RraN8ISK+W+v9S0k/oH0bnsNeaRHnxX4O/mhEtJrI+4ukyWQrP3uRDT/YwuUHwKZkwwfVpjWvAk4aLnd2JC0R836kYKEQETPJBmgA0JzvSt9Z2Rc7Un77VRHR331+xCjL9jdkYxbvjIjqxw6uIFsA25ts8Kan8SyY22O3LafU0oI88z6QbGrtafIA8aYuxncHcGu3LWqQTXhdTV79PkmurDfW8lxJBqWtyea/niUbyK5+m/E/S7nf3GYaFwM3VboXJT+yfRfZcss08spjyUqesWV8nyFbR5lGNkv2A9q0otPLPE5h3lajlirD/m8tfS/g9+QBYhbZatTebcpV/+1TybMH2YLLs+RdhF9S+f5tyfOBMu5ZZbnfSjYV2Ou2Avw7ufM8S7ZQdCQwqpZ3RbKlrYfKMr4L2K/NuOZZjmXdnljJuyy5g06tTePPZDOaXa/Xkm9p8lWn+8lXne4nv1k7qpJnfCnTLmSLPI+WdXI6sFwvy2kMeUJ2fJf7gMjWne4u5ZleprlMLV+v+yVzt+LV+k2sLfMNyDsss4Dz+lGGCb3Mz2iyWb57yjbyINkS06odtqd1yKvPWWRrUl9psz1tRLYj/HzZpo4AvkYv+15tHK11unWbfkuWfl8p436gbHPrlf4rAz8ty2U2eZzbt816/ynZwtWzZAtWpwKrVPKc1Wb93FU5FgbZ7OFJ5AXJ4+TxZxT5ybury7hvpdaaXBnH1uTxclb5Xdiah0qea8gmL3cgW4trjW+nbsrZYdnuU/Ls0If10SpHtdWvT5d+y5HHj4fL9ngXcEBt+E+Vaa5SSz+abF6y3bqdULaf58hj2Jtqw+5cyvUUuX/dCRzS67x0MbMT6htrKdQUcmfchbyFcD/ZklDbDyiX4V5Xhv1mlwt6+7IxTyrT+UCZxkwqO2XZcKaTbbt+qAw3iTyYrV1ZmE8A36pNY2Wy8fGDaxvRM2XBbw18tgx7TiXP2DIvD5HBf2eyrdbVyVtNQd4G3gzYrJf5nMK8gXa9Mo6Da+mHkcF921K2I0v5P1X6L0F+TzfIZgM3K7/RtY3vZHKH3bNsLPcDryl53kGeNHy/TGNb8uD9pW62FbLpyMPLcN9l3mYqlyEP2P8g20PeGvh2WdefLXlW67QcybZn76uMb1fyYBDM+Xj7q8gd8NN9XK+LUj7xRd5i3arMy/PAdyv5xpfp3Q/8sMzrZ8kd9JRellPrY+XbdrkffLPkPx7Yjgx4s0o5q8G/1/2SDEjTyba7W9vGWm3W32Fk28Lj+1GGCb3MzxvJE9L3ks1h7kU22TmFuU9mW+W5jWzeb2vmnMjuW8m3Ihl07iS3593Ik6wHGfxA+xB5ors7GYheSzbscl/5fYxszvT75D70ico4/p28TbpHme/Wyew9lGYKgbXJY9eDlfXzlsrxsLXNHVOmc0xJ+36Z/71LvmvIE+TlKtPfg9zHflW2j93JxxYzgTGVfNeU+by1lHEH8hg7m9Kkak/l7LBsTyP3oa6bYyzleJjcHvcG3k2eBLa+v/s0+ZhtO/ITpgF8pTJ8XwPtg+SJ6a5lvv9Otve+TMm3LnlMmVimuVWZxlG9zksXMzuhvrGWQt3L3G3Rvq+kv62HcW1a8vR4ZVTJf0OZzqKVtDXJwPK9StqVJW2dStpKZaM6rJL2U/I2ePXAcBAZkMeU7neWMn6kVpYPlvQNS/fY0n0jlXZkaweIjicdtfxTyMa8FyU/zbZ+WeH3AMv3MNyoMsxPgZsr6a2yfbyW/9XkzndyLX1s2YAOKt1fIL9r2tUO0Wa+D6ml/5TcKZYr3UeQO906bfI9ypyg0HY5MudEorXTf59sK/be1rbFnIPSun1crx8u3VvU8h1eltFKpXt8yXdKLd/xZd7Uw3L6Uhn2jZ3yVPKuUMY3sZb+oTKOXfq6X9LmxK62vD83wDJM6ON2swh5ghrA7m3KU78yvJX8IlCr+6iybtaopL2qbEvRh3K01mlPgfYBKm2gV6b/DJU2sEv6aeRdk1EdprcoebUeVK70yBPCyW3yt7bp+l2u20v6uEraJiVtz9I9igwkF7VZt08AR1fSrqESVEvaamV8n++tnB3m9Qrg/g7rftHKT7VyvHLXoM02vVct/XTyhLvV9nJfA+105j7Re0OZ/uGV7f1lYIm+bN8RMaBKJpMivz/Ycmv5u8YAxvmKUllkY+AXUXkGEvkZqD8D76oNcm9E3FvJN4NstLtantPI7y5uWUn7MHBZRLS+ebk9udOeI2nR1o/80gvkmWjVb6KshQH6AHmyMJvccTYA3hMRcz2vlrSOpDPLF01aH2D+OO2/dVq3OXk1eUZt3qaSt15a83Y9+RHq0yXtLGm5Ps7L2bXus8gg3/pc1vbkGen9tXJcSl4hrE/PriI3+NZ63JK8yvh9LW16RNxVmWY363V78mD6lzb5FiPP3KsurHXfSt5VGPC3fIvNyvhOr6WfRZ4g1veDwdgvzx1gGXol6dOSbpY0q4yj9ZWsdttxfRnfxtzzszlwTcz50haRn6T7bV/L1YULY+4vB0FuM38CprbZnseQV38oHSjp1jLfL5An09Dd/ttS//Tf3eSJcfUze63tvvW5zTeRwfL0WhmfIvf3+nHt9ohofS6SyLo6T9D/47s6pF/O3B+Tr3/u8O6IqH9CcgtyX/5lLf108pHbJv0s428jovUtaiK/5nUjcyoD30ged34paQ/14YP2Awm0j9e6Ww+ol6xnrGh9PPnfuhj/8uTKmd6m38PkmVhP5WmVqVqeP5Jn8x8GkLQeGcxPq+RZibyqbO0Ird+M0r/+8e925euPi4H/IJ+zHERuML+uVNBoNfQ/ifye5SHkVdp/kLeBl+hiGq2Pu1/G3PP2Anlb67UAkZ8+ez+5k54LzJR0maQ3dzkvj3ToXrVSji3alKG14/T4gfXIihQ3A+8uG/sG5BnzFeRVCeRtpisqg3W7Xlcit8962a6r5Wtpej9obedzbWfl5PMxet8PuilPXX2b7msZeiTps+Stvtbzt02YcwLTrpzt5qmabwzzbnN0SBuodvv7SuSjg/o20zqutLaZL5B3Xy4k78pswpyTlL6sn3pl0X91SKuOt7Xvn9GmnFvT+3YN8y73vngQGFMqjFZ9kjyGdfrkaLvlvQIwI+b9tOTDlf790WkbWhUgIu4gb6MvSdYpeKS8mtfrd7vna1vHETFN0p3Ae8hnQD35J+Wyv02/VcgdvK/TD0mnAwdJ+jQZcGcx9xn8Y+Rtsnd2GE39W6ODcTULc5+RXi3pSeBn5HO/b5f0zcmD8zsj4k+tAfvw3mprme1DXjXXPd36JyJ+RX5/8tVk8DoGuETSahHR2zdqVyafV1W7IZ/7tMoxg3y+0s7dvYwfMojuSQbUx8jXBqYDK5UNfyPm/g5mt+v1MfIZWP3j8C1Tuihbb64kb0m9hzlX1J20DnirUFlnZZ2/ln7sB12ob9ODXYa9gMsj4uDKuNbsRzlbptP+DsJg3VWoare/P0Y+B/9ih2FaV5d7kbduX/nmbDnZnx9a6+hg8nu4dc+3SRtMvydvvW5N5Yo8Iu4GKFf47bRb3o8DoyWNqh2LWrGiNa+teVq8NnynE/lO21DruEVETAImlQugd5CPDS6StEZEPNlhvEPyfuI3gQ0kfb5dT0kblUI/Q37n8/2lGn2r/7+RV31X9XP6p5G3Mfcgb1OcExHPVvpfQp6xLBsRN7T5dfOx8dZVxFL9LCPkB6dvBL4oaemS1vr7yq1BSctT+WB2L9P/CxlM1+4wb/MEuIiYFREXkEFrDL1cbRb1ILUXeUJzW+m+hKxY8I8O5WgF/J6W4xXkmeYnye+iRnlccDtZ23QRcudu6Xa9XkJeyc/qkK9fH0+vKtOaCOwnqe076pJ2K/+2npftVcuyJ3mi3J/9YDZ92zYHuwxLU9mGi337OI6qq4HNJLVuk7YePb1nAOPsi0vIyov3ddhmWkGk2/nu6/rpxq3kyeR6Hcp4W28jaKMv5TyTfDzwHUn9veJsuYo5lT6rPkhWRry+dLdufbceWSFpCbISUzvvqd1BfAN5x/PqesaIeD4iLiMrey5DL7fU5/vXeyLidGWrN98tB5mzyUv+lYCdyKvMceRKOYK8zXKBpP8lA+TXyAo99Xdwu53+PZKuJR+Ir8rct42JiCslnUlezX2PvGX4MllhaEey5u099OyO8vdgSRcDL9Wen3RTzpD0FfID858m5/cv5DOVH0n6Klnh48tkpY/qi96PkGd1e0m6hayocX9EPCbpi2X40eSZ5ZNlObyLDFg/l3QkeSZ3BblzrkbWOr4p8t3D3nxC2ZrT9WTtvI+TFWSeKP2PJQ/Sf5R0LHkF+yoy+L4zIlonDj0txz+QV4VbMaehAUqZDyCD+CtX1X1Yr2eQB7/LS8MqN5NnxGuRNTV3q52Y9ddBZGWLyyX9hLyNOgt4PXnAGEc+/3+8lPdQSc+QDb2sR77G8yfmfX7ZjTuAd0ramdz3Ho2IKZ0yN1CGS4AvSTqMXA9b0vnWYTeOJWvi/07ZitNs8uryuQGMsy++RZb/T5K+Tz53fQ25jDaNiPeWfJcAn5X03+RJ9HZkDem6O4CPSPoYeafm2YhodweqaxHxkqQDyOeLSwPnkMeIVcha/fdExPF9HG3X5YyI55WtkV0M3CTpR+SF1AvkCfyeJevT7YavOY/cbk6W9Dry+LELecX81cqVZavm+bElwL5M3iHsdIH5InBp2e9fBXydPLb+EEDZCtZ/kOtxKvma2mFkrLqr3QirC6C32mITaF/r+Bu1tLElfZ/exlny70juoDPJhf1IWYDvqeWrv0d7Hh3eo20zjSnUakqW9P1LWeeqgVzpP4q8rXkzefvhyfL/t5hTo601vx9vM/wiwI/I26Mv15dfh3LOUwu09PszeWtsqdK9JflKwHNk9fMDO6yj3cgd4YX6einL/goyaD9H3vY6GVi/9N+JrMjReifwQfK9vdd1s60w55npc+SB/Ov15Uw+gz+WOe+qziCfoR/U7XIkK1S9UrO4pLVqJLdb772u15hTC3ECc963fZw8aZjAnBrR42lTQ5U57wuO7WIfWKxsi60TqNY7uydSed+b9u+w/ogO79H2tl+SJzR/ZM4rURNr62+e2vJ9LMOEXuZ7KfIdyJnkgfUC8m2CuYbtVB7ybsCUWtrGzJ/3aL/cYdjXkq+dPVCWzyPklddnKnleTdasn1nW92/Ik62gUlOfvEL6JVn5KJj3Pdp31KY9T+3fTuUlH51cTD6ae75sbz8HNqnkuYasIFqfx4eBn/RWzl6W8Srk/nZH2f6eJ48/PwPeXsvbthyl33LAT+jhPdqS7y1lu5hFHmc/S+dax0eUbW5aKdeVwAa1ZfdbMm603sM/k/IKaU8/fybPzMxGrHK7+DngiIj4RhPTWOjakDUzMxtOHGjNzMwa5FvHZmZmDfIVrZmZWYPm++s9C4oVV1wxxo4dO9TFMDNboPz1r399NCJGD3U5hhMH2g7Gjh3LDTf06dVXM7MRT9IDvecaWXzr2MzMrEEOtGZmZg1yoDUzM2uQA62ZmVmDHGjNzMwa5EBrZmbWIAdaMzOzBjnQmpmZNWhYB1pJS0q6TtLNkm6X9LWSvqakayXdK+kXkhYv6UuVnbh4AAAcfElEQVSU7sml/9jKuA4t6XdL2m5o5sjMzEaa4d4y1Gxgy4iYJWkx4E+SLgY+DxwbEWdJ+gnwMfJD0h8D/hkRa0vaCzgG2FPS+sBewJuA1wGXSXpDRLw0FDNlC4exh1w4JNOdcvROQzJdM+ufYX1FG2lW6Vys/ALYEvhVST8F2K38v2vppvTfSpJK+lkRMTsi7gcmA5vMh1kwM7MRblgHWgBJi0i6CZgBTAL+DjwRES+WLFOBVcv/qwIPApT+TwKvraa3GcbMzKwxwz7QRsRLEbEhsBp5Fbpeu2zlrzr065Q+F0n7SbpB0g0zZ87sb5HNzMxeMewDbUtEPAFcCWwGLCep9Xx5NWBa+X8qsDpA6b8s8Hg1vc0w1WmcEBHjImLc6NH+ypOZmQ3csA60kkZLWq78vxSwNXAncAXwvpJtb+C88v/5pZvS//cRESV9r1IreU1gHeC6+TMXZmY2kg33WsdjgFMkLUKeFJwdERdIugM4S9I3gL8BJ5X8JwGnSZpMXsnuBRARt0s6G7gDeBHY3zWOzcxsfhjWgTYibgE2apN+H21qDUfE88D7O4zrKOCowS6jmZlZT4b1rWMzM7MFnQOtmZlZgxxozczMGuRAa2Zm1iAHWjMzswY50JqZmTXIgdbMzKxBDrRmZmYNcqA1MzNrkAOtmZlZgxxozczMGuRAa2Zm1iAHWjMzswY50JqZmTXIgdbMzKxBDrRmZmYNcqA1MzNrkAOtmZlZgxxozczMGuRAa2Zm1iAHWjMzswY50JqZmTXIgdbMzKxBDrRmZmYNcqA1MzNr0LANtJJWl3SFpDsl3S7pcyV9gqSHJN1UfjtWhjlU0mRJd0varpK+fUmbLOmQoZgfMzMbmRYd6gL04EXg4Ii4UdJrgL9KmlT6HRsR36lmlrQ+sBfwJuB1wGWS3lB6/wjYBpgKXC/p/Ii4Y77MhZmZjWjDNtBGxHRgevn/aUl3Aqv2MMiuwFkRMRu4X9JkYJPSb3JE3Acg6ayS14HWzMwaN2xvHVdJGgtsBFxbkg6QdIukkyUtX9JWBR6sDDa1pHVKNzMza9ywD7SSXg2cAxwUEU8BPwbWAjYkr3i/28raZvDoIb3dtPaTdIOkG2bOnDngspuZmQ3rQCtpMTLInhERvwaIiEci4qWIeBn4KXNuD08FVq8MvhowrYf0eUTECRExLiLGjR49enBnxszMRqRhG2glCTgJuDMivldJH1PJtjtwW/n/fGAvSUtIWhNYB7gOuB5YR9KakhYnK0ydPz/mwczMbNhWhgLeDnwYuFXSTSXtMOC/JG1I3v6dAnwSICJul3Q2WcnpRWD/iHgJQNIBwKXAIsDJEXH7/JwRMzMbuYZtoI2IP9H++epFPQxzFHBUm/SLehrOzMysKcP21rGZmdnCwIHWzMysQQ60ZmZmDXKgNTMza5ADrZmZWYMcaM3MzBrkQGtmZtYgB1ozM7MGDdsGK8ysvbGHXDhk055y9E5DNm2zBZWvaM3MzBrkQGtmZtYgB1ozM7MGOdCamZk1yIHWzMysQQ60ZmZmDfLrPbZAG8pXXczMuuErWjMzswY50JqZmTXIgdbMzKxBDrRmZmYNcqA1MzNrkAOtmZlZgxxozczMGuRAa2Zm1iAHWjMzswYN20AraXVJV0i6U9Ltkj5X0leQNEnSveXv8iVdko6TNFnSLZI2roxr75L/Xkl7D9U8mZnZyDNsAy3wInBwRKwHbAbsL2l94BDg8ohYB7i8dAPsAKxTfvsBP4YMzMBXgU2BTYCvtoKzmZlZ04ZtoI2I6RFxY/n/aeBOYFVgV+CUku0UYLfy/67AqZGuAZaTNAbYDpgUEY9HxD+BScD283FWzMxsBBu2gbZK0lhgI+BaYOWImA4ZjIGVSrZVgQcrg00taZ3SzczMGjfsA62kVwPnAAdFxFM9ZW2TFj2kt5vWfpJukHTDzJkz+15YMzOzmmEdaCUtRgbZMyLi1yX5kXJLmPJ3RkmfCqxeGXw1YFoP6fOIiBMiYlxEjBs9evTgzYiZmY1YwzbQShJwEnBnRHyv0ut8oFVzeG/gvEr6R0rt482AJ8ut5UuBbSUtXypBbVvSzMzMGjecP/z+duDDwK2SbipphwFHA2dL+hjwD+D9pd9FwI7AZOBZYF+AiHhc0teB60u+IyPi8fkzC2ZmNtIN20AbEX+i/fNVgK3a5A9g/w7jOhk4efBKZ2Zm1p1he+vYzMxsYeBAa2Zm1iAHWjMzswY1FmglbSFpnmfAkhaVtEVT0zUzMxtOmryivQJYoU36sqWfmZnZQq/JQCvat8D0WuCZBqdrZmY2bAz66z2Szi//BnC6pNmV3osAGwB/GezpmpmZDUdNvEf7WPkr4J/Ac5V+/wL+BPy0gemamZkNO4MeaCNiXwBJU4DvRIRvE5uZ2YjVWMtQEfG1psZtZma2oGgs0EpaATiKbC5xJWoVryJimaambWZmNlw02dbxSeTH2k8gP0vX9huwZmZmC7MmA+1WwDYRcW2D0zAzMxvWmnyPdgYwq8Hxm5mZDXtNBtrDgSMlvbrBaZiZmQ1rTd46/jIwFpgh6QHghWrPiHhzg9M2MzMbFpoMtL9qcNxmZmYLBL9Ha2Zm1iB/j9bMzKxBTTZY8TQ9vDvrBivMzGwkaPIZ7QG17sXIBizeS7YYZWZmttBr8hntKe3SJd1INmbxw6ambWZmNlwMxTPaK4D3DMF0zczM5ruhCLR7AY8OwXTNzMzmuyYrQ93K3JWhBKwMrAB8uqnpmpmZDSfzs8GKl4GZwJURcVc3I5B0MrAzMCMiNihpE4BPlHEBHBYRF5V+hwIfA14CDoyIS0v69sAPgEWAEyPi6AHMl5mZWdeGe4MVE4HjgVNr6cdGxHeqCZLWJ29Lvwl4HXCZpDeU3j8CtgGmAtdLOj8i7hiE8pmZmfWoyStaACRtCaxP3ka+PSKu7HbYiPiDpLFdZt8VOCsiZgP3S5oMbFL6TY6I+0p5zip5HWjNzKxxTT6jXRU4F3gr+eF3gNdJugHYPSKmdRy4dwdI+ghwA3BwRPwTWBW4ppJnakkDeLCWvukApm1mZta1JmsdH0c+K107IlaPiNWBdUracQMY74+BtYANgenAd0u62uSNHtLnIWk/STdIumHmzJntspiZmfVJk4F2G2D/iLi/lVBu3x5Y+vVLRDwSES9FxMvAT5lze3gqsHol62rklXSn9HbjPiEixkXEuNGjR/e3iGZmZq8YivdoXx7IwJLGVDp3B24r/58P7CVpCUlrklfP1wHXA+tIWlPS4mSFqfMHUgYzM7NuNVkZ6nLgOEn/FREPAkhag3zN5vJuRiDpTGA8sKKkqcBXgfGSNiRv/04BPgkQEbdLOpus5PQieTX9UhnPAcCl5Os9J0fE7YM1k2ZmZj1pMtAeCJwH3CdpGhkYVwVuKf16FRH/1Sb5pB7yH0WbDxaU92wv6maaZmZmg6nJ92gfBDaWtA2wLlkp6Y6IuKypaZqZmQ03g/6MVtIOkqZIWhYgIiZFxA8j4jiysYgpkrYd7OmamZkNR01UhjoA+HZEPFnvUdKOAT7XwHTNzMyGnSYC7ZuBnm4P/x54SwPTNTMzG3aaCLSj6fkVngBe28B0zczMhp0mAu1U8qq2kzcDDzUwXTMzs2GniUB7IfB1SUvVe0haGjiy5DEzM1voNfF6z1HA+4B7Jf0QaH17dj2yopSAbzYwXTMzs2Fn0ANtRMyQ9Day8f9vMqdR/yBbZ/pMRDwy2NM1MzMbjhppsCIiHgB2lLQ8sDYZbO8tn7MzMzMbMRr98HsJrNc3OQ0zM7PhbCi+3mNmZjZiONCamZk1yIHWzMysQQ60ZmZmDXKgNTMza5ADrZmZWYMcaM3MzBrkQGtmZtYgB1ozM7MGOdCamZk1yIHWzMysQQ60ZmZmDXKgNTMza1CjX+8xs4XL2EMuHJLpTjl6pyGZrtlgGNZXtJJOljRD0m2VtBUkTZJ0b/m7fEmXpOMkTZZ0i6SNK8PsXfLfK2nvoZgXMzMbmYZ1oAUmAtvX0g4BLo+IdYDLSzfADsA65bcf8GPIwAx8FdgU2AT4ais4m5mZNW1YB9qI+APweC15V+CU8v8pwG6V9FMjXQMsJ2kMsB0wKSIeLx+in8S8wdvMzKwRwzrQdrByREwHKH9XKumrAg9W8k0taZ3SzczMGrcgBtpO1CYtekifdwTSfpJukHTDzJkzB7VwZmY2Mi2IgfaRckuY8ndGSZ8KrF7JtxowrYf0eUTECRExLiLGjR49etALbmZmI8+CGGjPB1o1h/cGzqukf6TUPt4MeLLcWr4U2FbS8qUS1LYlzczMrHHD+j1aSWcC44EVJU0law8fDZwt6WPAP4D3l+wXATsCk4FngX0BIuJxSV8Hri/5joyIegUrMzOzRgzrQBsR/9Wh11Zt8gawf4fxnAycPIhFMzMz68qCeOvYzMxsgeFAa2Zm1iAHWjMzswYN62e0tuAYqsbmzcyGO1/RmpmZNciB1szMrEEOtGZmZg1yoDUzM2uQA62ZmVmDHGjNzMwa5EBrZmbWIAdaMzOzBjnQmpmZNciB1szMrEEOtGZmZg1yoDUzM2uQA62ZmVmDHGjNzMwa5EBrZmbWIAdaMzOzBjnQmpmZNciB1szMrEEOtGZmZg1yoDUzM2uQA62ZmVmDFthAK2mKpFsl3STphpK2gqRJku4tf5cv6ZJ0nKTJkm6RtPHQlt7MzEaKBTbQFu+OiA0jYlzpPgS4PCLWAS4v3QA7AOuU337Aj+d7Sc3MbERa0ANt3a7AKeX/U4DdKumnRroGWE7SmKEooJmZjSwLcqAN4HeS/ippv5K2ckRMByh/VyrpqwIPVoadWtLMzMwatehQF2AA3h4R0yStBEySdFcPedUmLebJlAF7P4A11lhjcEppZmYj2gJ7RRsR08rfGcC5wCbAI61bwuXvjJJ9KrB6ZfDVgGltxnlCRIyLiHGjR49usvhmZjZCLJCBVtKrJL2m9T+wLXAbcD6wd8m2N3Be+f984COl9vFmwJOtW8xmZmZNWlBvHa8MnCsJch5+HhGXSLoeOFvSx4B/AO8v+S8CdgQmA88C+87/IpuZ2Ui0QAbaiLgPeEub9MeArdqkB7D/fCiamZnZXBbIW8dmZmYLCgdaMzOzBjnQmpmZNciB1szMrEEOtGZmZg1yoDUzM2uQA62ZmVmDHGjNzMwa5EBrZmbWIAdaMzOzBjnQmpmZNciB1szMrEEL5EcFzGxkGXvIhUMy3SlH7zQk07WFi69ozczMGuRAa2Zm1iAHWjMzswY50JqZmTXIlaEWIkNVYcTMzDrzFa2ZmVmDHGjNzMwa5EBrZmbWIAdaMzOzBjnQmpmZNciB1szMrEEOtGZmZg0aMYFW0vaS7pY0WdIhQ10eMzMbGUZEgxWSFgF+BGwDTAWul3R+RNwxtCUzs+FsKBuB8ZeDFh4jItACmwCTI+I+AElnAbsCjQRat9BkZmYtI+XW8arAg5XuqSXNzMysUSPlilZt0mKeTNJ+wH6lc5akuxss04rAow2Of0HgZeBl0OLlUFsGOmYISzIw/zbUBRhuRkqgnQqsXuleDZhWzxQRJwAnzI8CSbohIsbNj2kNV14GXgYtXg5eBguzkXLr+HpgHUlrSloc2As4f4jLZGZmI8CIuKKNiBclHQBcCiwCnBwRtw9xsczMbAQYEYEWICIuAi4a6nJUzJdb1MOcl4GXQYuXg5fBQksR89QJMjMzs0EyUp7RmpmZDQkH2kEkaQVJkyTdW/4u3yHf3iXPvZL2rqS/VdKtpZnI4ySpNtwXJIWkFZuel4FoajlI+rakuyTdIulcScvNr3nqVm9NfUpaQtIvSv9rJY2t9Du0pN8tabtuxzncDPYykLS6pCsk3Snpdkmfm39z039NbAul3yKS/ibpgubnwgZFRPg3SD/gW8Ah5f9DgGPa5FkBuK/8Xb78v3zpdx2wOfne78XADpXhVicrcz0ArDjU8zoUywHYFli0/H9Mu/EO8XwvAvwdeD2wOHAzsH4tz2eAn5T/9wJ+Uf5fv+RfAlizjGeRbsY5nH4NLYMxwMYlz2uAe4bzMmhqOVSG+zzwc+CCoZ5P/7r7+Yp2cO0KnFL+PwXYrU2e7YBJEfF4RPwTmARsL2kMsExEXB25N51aG/5Y4L9p09DGMNTIcoiI30XEi2X4a8j3oYeTV5r6jIh/Aa2mPquqy+ZXwFblin1X4KyImB0R9wOTy/i6GedwMujLICKmR8SNABHxNHAnw79ltya2BSStBuwEnDgf5sEGiQPt4Fo5IqYDlL8rtcnTqTnIVcv/9XQk7QI8FBE3N1HoBjSyHGo+Sl7tDifdNPX5Sp5y0vAk8Noehl3Qmg9tYhm8otxe3Qi4dhDL3ISmlsP3yRPulwe/yNaUEfN6z2CRdBmwSpteh3c7ijZp0Sld0tJl3Nt2Of75Yn4vh9q0DwdeBM7oclrzSzdNffZ1vtudDA/nuxpNLIMcSHo1cA5wUEQ81e8Szh+Dvhwk7QzMiIi/Sho/wPLZfORA20cRsXWnfpIekTQmIqaXW6Az2mSbCoyvdK8GXFnSV6ulTwPWIp/T3FzqBK0G3Chpk4h4eACzMiBDsBxa494b2BnYqtxaHk66aeqzlWeqpEWBZYHHexm21+ZDh5FGloGkxcgge0ZE/LqZog+qJpbDLsAuknYElgSWkXR6RHyomVmwQTPUD4kXph/wbeauBPStNnlWAO4nKwAtX/5fofS7HtiMOZWAdmwz/BSGf2WoRpYDsD35acPRQz2PHeZ7UbJS15rMqQDzplqe/Zm7AszZ5f83MXcFmPvICjW9jnM4/RpaBiKf1X9/qOdvKJdDbdjxuDLUAvMb8gIsTD/y+crlwL3lbytwjANOrOT7KFnBYTKwbyV9HHAbWcvweEqDIrVpLAiBtpHlUPI9CNxUfj8Z6nltM+87krVi/w4cXtKOBHYp/y8J/LLMy3XA6yvDHl6Gu5u5a5zPM87h/BvsZQC8g7ylektl3c9zEjrcfk1sC5X+DrQL0M8tQ5mZmTXItY7NzMwa5EBrZmbWIAdaMzOzBjnQmpmZNciB1szMrEEOtNYoSVdKOr7SvbSkX0l6snyJaGy7tKErcTPKfIakcUNdFjObvxxorc8kTSxBIyS9IGlG+YzZ/qUFn6o9gEMr3R8FtiDfjRxDvhfbLm1h8yA5bzcNdESS1pJ0kqQHJc2WNKWcqLxt4MVsXl9OOirbWUiaJelmSfv0Y5pznfCZzU8OtNZfl5GBYyzZDvNvga8Bf5T0qlamyK/zPF0Zbm3gzoi4NSIejoiXOqT1WZsgP2xExEtl3l7sPXdnJTjdSLYe9Bnyk2q7AH8FfjjggjZM0uL9GOwT5Lb2FuAXwM/q32g1G9aGusUM/xa8HzCRNq3SABsA/wK+Vkm7Eji+8n9Ufle2Syt5Fye/OTsVeIZslnG7ynjHl/w7kq3q/AvYufR7Dxl4niebdjwKWLwy7BTgy8D/AU+VaXyxNi/LAD8Gppfx3AnsWen/NuAq4FngoZJ3mR6W2dhS3nG18m9FfonmWeAGyndXO4xDZItZf6PWJF/pv1zl/38nT4aeI9vPnQgsW1+HwOdK+f8J/AxYuvT/JPAI5fu/leF+DpxX6e5mWU8ATgaeIFtCCtqs8w7zHMD7ammPAd+tdL8WOLOsx+eA25m7pbGJbaY5tvRbH7gQeJpsk/tMYJXacry8bCdPk00jvnuo90H/FqzfkBfAvwXvR4dAW/qdD9xW6b6SOYF2hXLA/Qv55Z8V2qWVvGeQ35zdgvx49gFkMH1L6d8KVLeSV9SvB0aT37l9CtiX/CDDu8lm7L5TKdOUcrA+gLya/mwZ1+alv4A/k+0qb1/GvQOwe+n/78As4GBgHWBT4GrgVz0ss7G0D7TXlTKuC1xKBvR5mt4sw2xUhvlAL+tnaTJ4/qaU9V1kU4Dn1Nbhk8BPgfXKMnwCOLT0Xx6YDWxfGeZV5EnP+0t3t8v6KfLTbmuX5fUfZT62q67zDvPySqAl2z3+z5J2dCXPqsAXgQ3LutqvbCtblf7LktvXyWV6qzDng/KPkid06wFvJu/MXAeMKsPeCpxe1s/awO6t7cQ//7r9DXkB/FvwfvQcaI8Gnq10X0kJtKX7eGpXMPW0ctB+GVijlu83wP+W/1uB6r21PH8Ajqil7UYGxlaTo1OAM2t57gW+XP7fpkx/vQ7zeCpwUi1tw1KelToMM5b2gbZ6lf72krZah3G0gsxGvayfT5BB9DWVtNb01q6swwepXLGSQfeySve5wGmV7g+V8S7Zx2X9256WRS/zEuRV6izy04hBBse1exnuLOZuV3uu7bCkHQlcXktbvkxjk9L9FLD3UOxn/i08P38mzwabGPj3Ujcu47mjfBqwZQng97W8N9S63wpsIulLlbRRwFLklcz0knZLbbhpzPlA/UbA9Ii4s0P53gqsLWnPSlqroGvR/rOAnVTL0fqM2krkbdC6dt8pbWc94JaY+9n4X8iTh/XJRuwB7oi5nxlPI6/OW04HJkpaOiKeBT5IXrU/X/p3u6zr66ivvghcQn467nvAtyOiNQ9IWoT8StSe5NXtEuSjhyt7Ge9bgS0kzWrTby3yyvZ7wInl84yXk3cF7hrQ3NiI40Brg2198rNeAzGKDNb/AbxQ6/dcrfuZNsN+jXwWWDez8n99vMGcyoG9BbRRwInAsW36PdTLsHXVcrROUDpVUryn/F2PfE7bSU8nO9X0npYB5DPcF4FdJV0ObE3eYm7pdlnX11FfPVwC62RJ7ye/x3xjJeB9gbyN/znyVu8s4JvMOXHqZBT5fPYLbfo9AhAREySdQT462A74qqRPRcTJA5wnG0EcaG3QSNqAfKb5jQGO6m9ksFglIq7o47A3AutWr3j64UZgjKT1OlzV3kh+W3Qg0+iPm8jnxl+U9Iuo1c6WtFxEPFHyfFTSaypXtW8jA0unq/R5RMRsSb8ir2RXBB4mK4C19HdZ/6v8XaSPwxERkyX9GvgWWdsa8rWw30bEaQDK2yBvIJ85V6dZn96N5O34ByKiftJRnea95KOF4yT9GPg4+bzXrCt+vcf6awlJq0h6naS3SPo8eavur8B3BjLiiLiHrAw1UdL7JL1e0jhJX5C0Ry+DHwl8QNKRkjaQtG4Zx7f6UITLyZrA50jaTtKakraRtFvpfwx5y/QnkjaStLaknSX9X59ntg8iIphT8ejPZZprSfp3Sf9N1jKGXHbPAKeWfluQNax/3Y+geDp5Jfcp4OcR8XKlX3+X9QzyzsR2klaWtGwfy/RdYGdJm5Tue4CtJL1D0rrkM/81a8NMIdfZWEkrShoF/IisKPULSZuW7WxrSSdIeo2kpST9SNL4MtymZFC/o4/ltRHOgdb6a2vyGdw/yMC0C3kbcYuIGOitQsiA8jPyyuUu8jbmFsADPQ0UEZcCO5E1YK8rv0NKObtSgskOZM3j08mrwB+Qz/2IiFtKWcaSV3g3A/9Dud3YpIi4jny2eCfwk/L3QmATshY15XnqduQrStcB55G1oj/aj0n+gbwdvj65LKpl6deyLs+FDySvDKeV8nUtIm4lTypad06+UaZ9cSnvM+TJRtV3yKvaO8jb2mtExDSyAtrL5DPg28ngO7v8XiIrR51C1qY+l1yOn+9Lec384XczM7MG+YrWzMysQQ60ZmZmDXKgNTMza5ADrZmZWYMcaM3MzBrkQGtmZtYgB1ozM7MGOdCamZk1yIHWzMysQf8faVznXAYRzfkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# histogram of 10000 differences in convert rates between groups\n",
    "plt.hist(x = p_diffs)\n",
    "plt.title('Sampling Distribution of 10000 Differences\\nin Convert Rates between Control and Treatment Groups', fontsize = 16)\n",
    "plt.ylabel('Count', fontsize = 14)\n",
    "plt.xlabel('Difference in Convert Rates', fontsize = 14);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. p-Value\n",
    "The value of `obs_p_diff` or the difference in the two convert rates observed in `ab_test.csv` is 0.158%, showing higher convert rate for the old page. According to the visualization below, the proportion of values in `p_diffs` which are larger than the observed statistic and thus are considered extreme was 90.7%, which exceeds the tolerance level of 5%.\n",
    "\n",
    "This significantly large p-value also illustrated by the relatively large portion of the histogram to the right of the red vertical line suggested a large probability in rejecting the null hypothesis when the null hypothesis is in fact true. Therefore, based on this p-value, the null hypothesis could not be rejected. The convert rate of all users in the old page was found to be equal to or larger than that of all users in the new page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.0015782389853555567"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# observed difference in convert rates between groups\n",
    "obs_p_diff = obs_p_new - obs_p_old\n",
    "obs_p_diff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9074"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# p-value: proportion of sampled differences larger than the observed difference\n",
    "p_val = (p_diffs > obs_p_diff).mean()\n",
    "p_val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdoAAAEuCAYAAAAp59OgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xe4FdXZ9/HvDSKIFRQVQV8MYoIVFFssD1awxJ7IE03UJJoYjZqosSsSNRqjxkQfjVGCLfZGYkWDLVZUFAUVoihHUFBsVBXv9497bRiGvc/ZpwznwPl9rmtf58yaNTNr6j1lzRpzd0RERKQYbZq7ACIiIkszBVoREZECKdCKiIgUSIFWRESkQAq0IiIiBVKgFRERKZACLWBm+5rZE2Y21cxmm9m7ZnaPmQ1s7rJVYmbDzGxipruHmbmZHdYMZZmYpu1m9rWZfWxmz5nZBWbWo0z+hcpe5TQGm9lODSjXsEz3YamM69VnPA0pV0PmcXEys23SOpqZlkmfCvm6m9lfzOwZM5uV8vaokLeDmV1kZlPSfvSMme1QJl8bMzs1rZ85ZvaKmR1QYZxHmNkbZjbXzN40s19UyLevmb2cxveumZ1hZm2rWA6DM9uum9kXZjbezP5hZgPK5O+f8vXPzc+f0nx/Y2b3pPTvmNm/zezzNMy+dZVHllLu3qp/wLGAA9cC3wN2Ao4A7gH+0Nzlq6Xcw4CJme72wNZAl2Yoy0TgwTT9bYDdgdNT+gxgv1z+nkDfek7DgXPrOUxfoGem+7A0nvWacN7Llqsh87iY19lY4MW0vW8NdKyQrz/wIXA/8FCa3x4V8t4EfJr2n52Bu4DZQJ9cvvOAucCJwI7AX4FvgD1y+Y5I6eelfOem7qNy+QYA84CrU77fAHOAC6tYDoPTPG2blsNOwE+BB1L6DUCbTP6VUr6VMmk/SHl/k7b/9VP6/WkfGJiG6dTc612/5vk1ewGa+we8B9xdoV+bxVmWepZ7oUDbzGWZCNxYJn0F4D/ALKB7I6dRdaAF2ldIX2yBtiX/iDtZ84Ah1eTN/P+zSoEW2DT1OzyTtgzwJjA8k7Z6CrLn5IZ/FHg1N+xU4LpcvqHAR0C7TNrLwOO5fGcBXwJr1jF/pUC7TJl+v079TqhjHGenfG1y6e8A1zfheiu7XevX8n/NXoDm/hFXXFdWka8Lceb9Vgock4B/AN1y+Uo77neIK4CZKZgfnvr/CHgjTXckmSuu1H8icCNxNj+BODN/Cdgxl2+hQAv0SNM9LJenhriyezKVezzwizLzt0s6YM1J0/1Zfhq1LJuJlAm0qV/pAPy7Wsq+DPA74L9p+h8BTwHbpf5e5jc4N4/bAE8TV1CXZco1LDOdw9KwOxB3LGYAHwNXAMtl8vVP+frn5qU0fI8qyzUxN3xX4Po0f3OBV4FDKkxja+IK8XNgMvBnoEMV62Il4PI0zFwi0P0asNz4s78613EatrZAeyYR2Drm0s9J5Wif2f4d6JXLd3hKXzd1b5+6d83l2zGl75i6107dR+TyrUsu8FeYp8FUCLSp/0vApErbRtrG8suz3DL23D4xHPiE2F7/A2xfZv8uu12n/kcAr7Bgf7kW6JwbhxN3AY4lgv4XwOPAhmXmc79Ujhlpm3se2Du3j55KHLvmpu3r4uw2SR37cWv+6RltbFCHmtlJZrZ+Lfk6ExvPqcStoJOAXsB/zKxDmfy3A/cB+xK36Iaa2fnAUcApxIHl20Swzvsf4jbU6cAgYsN+wMy+Xf/ZY6U0jRuBfYAXgCvNbMdSBjPbIJV1RpreacBxxG20RnH3V4idcttasp1MBIM/E7cBDyeucDqn/tukv8PS/9sA12SGXxm4BbiZuG1dbplm3UicTOwPXEoctK6sZn5y6irXfGa2PHGQ251YvvsCY4AbzOzIMoPcQByw9k9lO5rY9ioyszbEejycOAh+j7ilfwlx+5XUf7v0/7WpzPvVOpfV2RB4x91n5dJfB5YF1svkm0ss/3w+gA0y+QBea0g+d3+HOLHcgMZ5AOhuZutU6L8fsf5hwTYwMv2dRtw+LqVjZpsRgbMzsd0dQJzsPWJmm+fGXXa7NrMLgP8DHgH2Jo5FA4ljRP659CHAnsT+fDiwDnCvmS1TymBmvyJu808FDgW+D9xNnLyX3AickcqwJ/B74hb7TZk8de3HrVdzR/rm/gHrE1cWpTPPj4gNe7c6hmvLgrPp/TLpg1PajzNpnYCviR0q+2yn9Hz4/2XSJhJXButk0lYEpgM3ZNKGUd0V7fyz/5TWPs3j1Zm0fxAHhY6ZtK7EicXE2pZDpsxlr2hT/2eAcbWU/V/AXXVMwyn/LLQ0j/tUKNewTPdhKe9VuXynE7dSS8/W+lPFFW0V5crO4zEVxvkIcYBrm5tG/tbqv4C36lhGe+W3gZR+DRHcVkvdy5C5+q7HvlLbFe3DwLNl0ndJw2yfuq8GPiiTb72U70ep+7TU3SGXr1T2M1P3D1P3d8qMswa4to55GkztV7Q/T/23qrRtEFeNXmH6w3JpjwLjgGUzaW1T2j11bdfEfj4POCuXvm3Kv29u2xzPwrfZD0zp303dKxFXuhX3PxbcXfhxLv3glN4ns43Wuh+31l+rv6J197eIW6v/Q5z1jybOUh8yszOyec3sqFRDcgYRON9LvcpdaT6QmcYnxMH0WXf/PJPnjfR37dywz7p7ady4+xfElcg21N8sdx+ZGddcYufLnqFvDdzvmasRd59CnHk3BSN2yEpeAPYws/PMbDszW7ae4/+a2MmrdVuu+xbiueWW9ZxufewAvO/uj+XSbyQeS+SvvO7LdY9h4XVWaRrfECeK+WksS8O2n2pVWsfWiHxUyFttvvw4G6LactQ9IrPliOPM7cA3ZrZMurI04oQrX0O73Ha9K7Gt3lQaPo3jOeKWb34cI9z9q0z3mPS3tC19l6hLcXUtRR9InPzfmZvmw6l/aZqN3Y+XWq0+0AK4+zx3f8Ldz3D3XYBvERvk2WbWCebfXindrtmfOChvnUZR7tbxJ7nuLyuklRv+wzLj+xDoVsXs1FUOiKub7DS7EicC5abZFNYGptTS/3yiQsnexLPkj83s72a2WpXjn+ru8+pRnvx8lbobsnyr1Znyy+CDTP+s6bnuucTdiLqmMT2dTFUzjaY0vcL4O2X6l/52MrN8ECyXjzLj7FxlPoBVWHQ51lfpJLi27bdanYmr1zOBr3K/Y4jlkj0ml9uuV09/J5QZx0rAqrn85bYjWLD/l/LX1FLu1YkTtRm56ZWOGaVxNHY/XmotU3eW1sfdJ5vZNcBlxHPY54lnl4+6+wmlfGa2bkFFWKNC2vsFTW8KC3bguspRL+n9zLWo8OwSIJ1xXwhcaGZrErdALwE6AgdVMZn6Xm2swYJnfaVuWLB856S/+TPy/EGsPqZT/s7Hmunvx40Yd3Yanc1sWXf/MpPelNOo5HVgPzPr6As/p92AOKGckMnXnnj9aUIuH8RrR6V8EM9gp1SZ75lSpvSub8dMvobaA3jP3Sc1cjwQrz59Q1S+u75cBnf/JttZJktpHe5G+ZPo+q7jj9Lfbiz6PDw7zjnELeRyJkOT7MdLrVZ/RWtm+du2Jd9Jf0tXAx2Js7iswwspFGydLZeZrUhUQHim8iCN8ixxy6djZppdqb0CU53MbAXioDKLqLFdJ3f/wN2vIe4cbJTp9SWwXGPKk/GDXPcg4gD4fOp+N/3dKJdvjzLjqrZcjxOVavLL9IfElcG4KsZRzTTaEJVZsg4myvlsE0yjkuFAu+y00+3Fg4CHM1fZD6ayHJwb/hDgNY9KTBDb+kcV8k0nasiSHrG8UiHfV2Qe4dSXmf0a6EMEi0Zz95nEld6mwEvuPir/q2I0I4htdZ1yw2eWX7WeJq5Uy1XIK3mQuAJeucI0J5eZ10r7caukK1p4zcxGErXs3iFuv+wB/AK4LfOs9EHgZDM7jTgg70RULCjCh8DDZjaYuNVzMrA8UXW+COcS8/KQmf2RuOI4M5Xjm9oGzFjNzLYmnjetDGxG1KrsAvxvuZ2xxMzuJQ6WLxFn6X2J50LZ4DwW2NPMHkx5Jtc2zjrsYWYXEc+YtiRud12fntfj7lPM7HHgVDP7iAiEhxBXYXnVlmsYUfPzLjM7nbhVdzDxzO3n9bz1XckDxOsUV5lZF+Jqbw+iEtPv3f2j2gauxMxK23mpVuzuZjYNmObujwO4+2gzuxX4k5m1I/alo4jXbOYHQXefamaXEsv2C2KdH0TsT/tk8n1lZmcC/2dm7xMH7J2AnwC/yl2xnwb8y8z+Sjyf7kvUkL3M3T+gOluZ2TwioHyL2B92B64jatE2ld8ATxD72rXE1fpqxP7S1t1PqW1gd/+vmV0IXJ7eQnicuNpcm9iWrsnWyaiLu39hZqcCfzGzO4laxF8QJxhz3P0v7v6Ymd0M3GFmlxDHv2+Iill7ACe7+1tV7setU3PXxmruHxFQhxNXMXOI915fBn7LwjUDlyNes5hGbIj/YsG7eoMz+QZTphYjZWrmsqAG4y75fMTB8b9EoH0Z2Ck37DCqfI+2zDw/BjyWS9uVqAg2F3ibqG15N/ByFctwIgtqbc8jdrIXgAvI1KiupewnEFdbHxPvC76ZlmO2tuS2xGtSc7LLvNI8Zso1LNN9GAveo72XOJOfTu492pS3O/BP4nbfB8Tzp0Vq3dZRrom5cXYlXtup5j3a9XLpgylTs7XMPJfeo51CXDm+ReY92pSnXrWOM+s2/8tvQ8sRV38fpOXxHLla1ilfWyIQvptZDgdWmPbP0zyUKvH9skK+/YmD/FyikuJZpJrcdczb4Nw8zSRuaf8DGFAmf38aUes4pfcmKuBNTeWtIY5Be2TyDKPCdp36/4jYZ2am7XhcWu/dM3mcXI14yhwnUvqBaX3NJipVPQfslenfhjhRLL27+1n6/w/ElS5UsR+31l/pJXZpISzax33K3Q9p5nKsQBxw7nP3nzZnWURElmS6dSwAmNlfiOc1k4nKS8cRNUEva85yiYgs6RRopaQDUWNwDeKW4/PELe1Xm7VUIiJLON06FhERKVCrf71HRESkSAq0IiIiBaoz0JrZYDNr8vvLZra7mf3LzKaa2Vdm9qGZDTezpviSSLMys/5puVV1ImNmE83MM79PzWyEmW1X99Blx7dKmv5mDRm+oUrbSvbLII0cX72W45LGzNqZ2S/N7D9pnc81s3fMbGiR687Mjjez/Qsat6f3v5tdXccuMzsst99V+g1ejMXGzA40s2MX5zQboiHlNLPVzexCM3vNzGaa2Rwzm2Bmwxp6vFsSVHMAu4YmbozczC4mPh81m2jjc+f091PgNjPbtCmn1wz6E40g1CdAPEQs522JGr89gftTU3L1tUqa/mINtAXoT/2X4xLB4rN5jxKfs3ueaNRhN+KdzHVTv6IcT7x32tqVPtRR+h2d0o/NpVdsPrQgB6YytHT1KmdqjvVVojW0G4lPRe5OvHv9beBJM1u5gHI2uzqvPNy9htobnK4XMzuEaB3lRHe/ONf7djO7jPJteLZ4qUWcrxs4+EfuXmoi72kzm0C08jOIaPhBli6XAVsRDR9km9Z8HLi2pdzZMbP2vuhHCpYK7j6NaIAGAFvwXelxmX2xIotvv5q7N3SfbzXSsr2HaMxie3fPfuxgJNEC2KFEgze1jWfJ3B6rbTkll+bEmfexRFNrXxAHiA2rGN9YYEy1LWoQTXg9Q1z9fkasrG/n8jxGBKVdiOa/ZhENZGe/zfiDVO5NykzjAWB0pnsZ4iPbbxAtt0wmrjw6ZPL0SOP7JdE6ymSiWbLLKNOKTh3zOJFFW41aLg37f7n0QcC/iQPEDKLVqEPLlCv/OyyTZ3+iBZdZxF2E28l8/zbl+WEa94y03McQTQXWua0AGxM7zyyihaIhQJtc3tWIlrbeT8v4DeDIMuNaZDmmdXtNJu/KxA5ak5vGf4hmNKterylfR+JVp3eIV53eIb5Z2yaTp38q095EizwfpXVyI7BKHcupK3FCdnmV+4ARrTu9mcozJU1zpVy+OvdLFm7Fq/QbllvmGxF3WGYA9zagDIPrmJ8uRLN8b6VtZBLRElO3CttTL+LqcwbRmtRZZbanvkQ7wnPSNnUmcA517Hu5cZTW6S5l+nVI/c5K4343bXO9U/81gL+l5TKXOM4dXma9/41o4WoW0YLV9cCamTy3lFk/b2SOhU40e3gtcUEynTj+tCE+efdMGvcYcq3JpXHsQhwvZ6TffaV5yOR5lmjycneitbjS+PasppwVlu1hKc/u9VgfpXJkW/06KvVbhTh+fJC2xzeAY3LD/yJNc81c+gVE85Ll1u3gtP3MJo5hG+aG3SuV63Ni/xoHnFLnvFQxs4PzG2sq1ERiZ9ybuIXwDtGSUNkPKKfh1krDnl/lgh6YNuYRaTo/TNOYRmanTBvOFKJt10PScCOIg9l6mYX5KfCH3DTWIBofPyG3Ec1MC34X4Fdp2DszeXqkeXmfCP57EW21rk3canLiNvDWwNZ1zOdEFg20vdM4Tsiln0YE991S2Yak8v8i9W9PfE/XiWYDt06/LrmNbyixwx6UNpZ3gBVTnu2Ik4Y/pWnsRhy8T65mWyGajjw9DXcxizZTuRJxwH6PaA95F+CitK5/lfJ0r7QcibZn386Mbx/iYOAs+Hj78sQOeFQ91+sypE98EbdYd07zMge4OJOvf5reO8Bf0rz+ithBr6tjOZU+Vr5blfvB+Sn/5cAAIuDNSOXMBv8690siIE0h2u4ubRs9y6y/04i2hfs3oAyD65ifbxMnpAcQzWEOIprsnMjCJ7Ol8rxGNO+3CwtOZA/P5FuNCDrjiO15X+IkaxJNH2jfJ0509yMC0apEwy5vp99PieZM/0TsQ0dkxrExcZt0/zTfpZPZt0jNFALrEceuSZn1s2nmeFja5i5M07kwpf0pzf+hKd+zxAnyKpnp70/sY3ek7WM/4rHFNKBrJt+zaT7HpDLuThxj55KaVK2tnBWW7Q3EPlR1c4ypHB8Q2+OhwI7ESWDp+7tfEI/ZBhCfMHXgrMzw9Q20k4gT033SfP+XaO99pZTvO8QxZVia5s5pGufVOS9VzOzg/MaaCjWehduiPTClf7eWcW2V8tR6ZZTJPypNZ5lM2rpEYLkkk/ZYSuuVSVs9bVSnZdL+RtwGzx4YjicCctfUvX0q449zZTk4pfdJ3T1S90tk2pHNHSAqnnTk8k8kGvNehvg02wZphb8FdKpluDZpmL8Br2TSS2X7WS7/CsTONzSX3iNtQMen7hOJ75pWtUOUme9Tcul/I3aKVVL3mcRO16tMvo9YEBTKLkcWnEiUdvo/EW3Fji9tWyw4KH2nnuv1R6l7h1y+09MyWj1190/5rsvluzzNm9WynE5Ow367Up5M3s5pfMNy6Yekcexd3/2SMid2ueV9XCPLMLie201b4gTVgf3KlCd/ZTiG+CJQqfu8tG7WyaQtn7Ylr0c5Suu0tkD7Lpk20DPTn0mmDeyUfgNx16RNhektQ1ytO5krPeKEcEKZ/KVtOn+X6/WU3i+TtmVKOyh1tyECyf1l1u2nwAWZtGfJBNWU1j2N7zd1lbPCvI4E3qmw7pfJ/CxXjvl3Dcps04Ny6TcSJ9yltpfrG2insPCJ3vpp+qdntvdvgPb12b7dvVGVTEZ4fH+wZEz6u04jxjlfqiyyGXCrZ56BeHwG6j/A/+QGGe/u4zP5phKNdmfLcwPx3cWdMmk/Ah5x99I3LwcSO+2dZrZM6Ud86QXiTDTrHk9roZF+SJwszCV2nI2A77n7Qs+rzayXmd2cvmhS+gDzzyj/rdO8bYiryZty81ZD3HopzdsLxEeobzSzvcxslXrOy2257luIIF/6XNZA4oz0nVw5HiKuEDagdo8TG3xpPe5EXGX8O5c2xd3fyEyzmvU6kDiYPl0mXzvizD3rvlz3GOKuQqO/5ZtsncZ3Yy79FuIEMb8fNMV+eXcjy1AnMzvKzF4xsxlpHKWvZJXbjvPL+DUWnp9tgGd9wZe28Pgk3T/rW64q3OcLfzkIYpt5Cqgpsz13Ja7+sHCsmY1J8/0VcTIN1e2/JflP/71JnBhnP7NX2u5Ln9vckAiWN+bK+Dmxv+ePa6+7e+lzkXjU1fmUhh/frUL6oyz8Mfn85w7fdPf8JyR3IPbl23PpNxKP3LZsYBn/6e6lb1Hj8TWvl1hQGfgl4rhzu5ntb/X4oH1jAu30XHfpAXWHfMaM0seT/18V4+9ErJwpZfp9QJyJ1VaeUpmy5XmSOJv/EYCZ9SaC+Q2ZPKsTV5WlHaH0m5r65z/+Xa58DfEAsAXxnOV4YoO5K1NBo9TQ/wjie5anEFdpWxC3gdtXMY3Sx90fYeF5+4q4rbUqgMenz75P7KR3A9PM7BEz26TKefmwQne3TDl2KFOG0o5T6wfWPSpSvALsmDb2jYgz5pHEVQnEbaaRmcGqXa+rE9tnvmzP5/KVFL0flLbzhbazdPL5MXXvB9WUJy+/Tde3DLUys18Rt/pKz9+2ZMEJTLlylpunbL6uLLrNUSGtscrt76sTjw7y20zpuFLaZk4k7r7cR9yV2ZIFJyn1WT/5yqJfVkjLjre0799Uppy7UPd2DYsu9/qYBHRNFUazfk4cwyp9crTc8u4MTPVFPy35QaZ/Q1TahroBuPtY4jZ6B6JOwYfp1bw6v9u9WNs6dvfJZjYO+B7xDKg2n5Au+8v0W5PYwes7fTezG4HjzewoIuDOYOEz+I+J22TbVxhN/lujTXE1CwufkT5jZp8Bfyee+12U0rchDs7bu/tTpQHr8d5qaZkdRlw1531R+sfd7yC+P7kCEbwuBB40s+7uXtc3atcgnldluyGe+5TKMZV4vlLOm3WMHyKIHkQE1I+J1wamAKunDb8vC38Hs9r1+jHxDCz/cfiSiVWUrS6PEbekvseCK+pKSge8Ncmss7TOV6UB+0EV8tt0U5dhEPCou5+QGde6DShnyRTK30FoqrsKWeX294+J5+AnVRimdHU5iLh1O/+bs+lkf3EoraMTiO/h5s0pk9aU/k3cet2FzBW5u78JkK7wyym3vKcDXcysTe5YVIoVpXktzdOyueErnchX2oZKxy3cfQQwIl0AbUc8NrjfzNZx988qjLdZ3k88H9jIzH5TrqeZ9U2Fnkl85/P7qRp9qf//I676Hm/g9G8gbmPuT9ymuNPdZ2X6P0icsazs7qPK/Kr52HjpKmK5BpYR4oPTLwEnmVnHlFb6O//WoJl1IvPB7Dqm/zQRTNerMG+LBDh3n+Hu/yKCVlfquNpM8kFqEHFC81rqfpCoWPBehXKUAn5ty3Ekcab5c+K7qJ4eF7xO1DZtS+zcJdWu1weJK/kZFfI16OPpWWlaw4AjzazsO+pmtm/6t/S8bFAuy0HEiXJD9oO51G/bbOoydCSzDSeH13McWc8AW5tZ6TZp6dHT9xoxzvp4kKi8+HaFbaYURKqd7/qun2qMIU4me1co42t1jaCM+pTzZuLxwB/NrKFXnCWPs6DSZ9bBRGXEF1J36dZ36ZEVZtaeqMRUzvdydxDXJ+54PpPP6O5z3P0RorLnStRxS32xf73H3W+0aPXm4nSQuY245F8d2JO4yuxHrJQzidss/zKz/yMC5DlEhZ78O7jVTv8tM3uOeCDejYVvG+Puj5nZzcTV3CXELcNviApDexA1b9+idmPT3xPM7AFgXu75STXldDM7i/jA/FHE/D5NPFO5wszOJip8nEFU+si+6P0hcVY3yMxeJSpqvOPuH5vZSWn4LsSZ5WdpOfwPEbD+YWZDiDO5kcTO2Z2odTza493Duhxh0ZrTC0TtvJ8RFWQ+Tf0vJQ7ST5rZpcQV7PJE8N3e3UsnDrUtxyeIq8KdWdDQAKnMxxBBfP5VdT3W603Ewe/R1LDKK8QZcU+ipua+uROzhjqeqGzxqJldRdxGnQF8izhg9COe/09P5T3VzGYSDb30Jl7jeYpFn19WYyywvZntRex7H7n7xEqZCyjDg8DJZnYasR52ovKtw2pcStTEf9iiFae5xNXl7EaMsz7+QJT/KTP7E/HcdUViGW3l7gekfA8CvzKz3xIn0QOIGtJ5Y4Efm9lPiTs1s9y93B2oqrn7PDM7hni+2BG4kzhGrEnU6n/L3S+v52irLqe7z7FojewBYLSZXUFcSH1FnMAflLJ+UW74nHuJ7Waoma1FHD/2Jq6Yz85cWZZqnl+aAuw3xB3CSheYXwMPpf1+eeB3xLH1LwAWrWBtQazHGuI1tdOIWPVGuRFmF0BdtcUGU77W8bm5tB4p/bC6xpny70HsoNOIhf1hWoDfy+XLv0d7LxXeoy0zjYnkakqm9KNTWReqgZzp34a4rfkKcfvhs/T/H1hQo600vz8rM3xb4Ari9ug3+eVXoZyL1AJN/f5D3BpbLnXvRLwSMJuofn5shXW0L7EjfJVfL2nZjySC9mzittdQYIPUf0+iIkfpncBJxHt7a1WzrbDgmels4kD+u/xyJp7BX8qCd1WnEs/Qj692ORIVqubXLE5ppRrJ5dZ7nevVF9RCHMyC922nEycNg1lQI7o/ZWqosuB9wR5V7APt0rZYOoEqvbN7DZn3vSn/DusVVHiPtq79kjiheZIFr0QNy62/RWrL17MMg+uY7+WIdyCnEQfWfxFvEyw0bKXyEHcDJubSNmPxvEd7RoVhVyVeO3s3LZ8PiSuvX2byrEDUrJ+W1vc9xMmWk6mpT1wh3U5UPnIWfY92u9y0F6n9W6m8xKOTB4hHc3PS9vYPYMtMnmeJCqL5efwAuKquctaxjNck9rexafubQxx//g5sm8tbthyp3yrAVdTyHm3Kt2naLmYQx9lfUbnW8Zlpm5ucyvUYsFFu2f2TiBul9/BvJr1CWttPn8kTEZFWK90ung2c6e7nFjGNpa4NWRERkZZEgVZERKRAunUsIiJSIF3RioiIFGixv96zpFhttdW8R48ezV0MyXszver77fq0WCcii8uLL774kbt3ae5ytCQKtBX06NGDUaPq9eqrLA79+8ffxx5rzlKISAVm9m7duVoX3ToWEREpkAKtiIhIgRRoRURECqRntCJSp6+++oqamhrmzCn6Iy+ypOjQoQPdu3enXbv8l+8kT4FWROoXpiJyAAAgAElEQVRUU1PDiiuuSI8ePTCr9A1vaS3cnY8//piamhrWXbcxXzhsHXTrWETqNGfOHFZddVUFWQHAzFh11VV1h6NKCrQiUhUFWcnS9lA9BVoREZECtehAa2YdzOx5M3vFzF43s3NS+rpm9pyZjTezW81s2ZTePnVPSP17ZMZ1akp/08wGNM8ciUhD1dTUsM8++9CrVy969uzJcccdx5dffgnAsGHDOOaYYxo9jWHDhtGlSxf69u1Lr169GDBgAE8//fT8/meddRaPPPIIAE8++SQbbrghffr0Yfbs2Zx00klsuOGGnHTSSY0uB8B1111Hr1696NWrF9ddd13ZPLfffjsbbrghbdq0WaiBnYkTJ7LccsvRp08f+vTpwy9+8Yv5/W6++WY23nhjNtlkEwYOHMhHH30EwMknn8wmm2zCj3/84/l5b7jhBi677LImmZ/WrKVXhpoL7OTuM8ysHfCUmT0A/Aa41N1vMbOrgJ8SH5L+KfCJu69nZoOAC4GDzGwDYBCwIbAW8IiZre/u85pjpmTp0OOU+5pluhMv2LNZptuc3J3999+fo446invvvZd58+Zx5JFHcvrpp3PRRRc16bQOOuggLr/8cgBGjhzJ/vvvz8iRI+nduzdDhgyZn++mm27ixBNP5PDDDwfgr3/9K9OmTaN9+/aNLsP06dM555xzGDVqFGbG5ptvzt57702nTp0WyrfRRhtx11138fOf/3yRcfTs2ZPRo0cvlPb1119z3HHHMXbsWFZbbTV++9vfcvnll/PrX/+ap59+mldffZWDDz6YMWPGsN566zFs2DAefPDBRs9Pa9eir2g9zEid7dLPgZ2AO1L6dcC+6f99Ujep/84WDxL2AW5x97nu/g4wAdhyMcyCiDSBf//733To0GF+UGvbti2XXnopQ4cOZdasWQBMmjSJgQMH8u1vf5tzzjkHgJkzZ7Lnnnuy6aabstFGG3HrrbfWa7o77rgjRx55JFdffTUAhx12GHfccQfXXHMNt912G0OGDOHggw9m7733ZubMmWy11Vb1nkY5Dz30ELvuuiudO3emU6dO7LrrrmUDXu/evfl2Pdr9dnfcnZkzZ+LufP7556y11lq0adOGL7/8Endn9uzZtGvXjosuuohjjz1Wr+80gZZ+RYuZtQVeBNYDrgD+C3zq7l+nLDVAt/R/N2ASgLt/bWafAaum9Gczo80OIyL1cfzxkLtSarQ+feBPf6rY+/XXX2fzzTdfKG2llVZinXXWYcKECQA8//zzvPbaa3Ts2JEtttiCPffck3fffZe11lqL++6Luw+fffZZvYu22Wab8de//nWhtJ/97Gc89dRT7LXXXhx44IEArLDCCotcQZbTp0+fOvO9//77rL322vO7u3fvzvvvv1+vcr/zzjv07duXlVZaiXPPPZftt9+edu3aceWVV7Lxxhuz/PLL06tXL6644gratm3LAQccQN++fdl5551ZeeWVeeGFFzjrrLPqNU0pr0Vf0QK4+zx37wN0J65Ce5fLlv6WqwbntaQvxMyONLNRZjZq2rRpDS2yiDQxdy9byzWbvuuuu7Lqqquy3HLLsf/++/PUU0+x8cYb88gjj3DyySfz5JNPsvLKKzdo2k2pmmBcbpr1qeXbtWtX3nvvPV5++WUuueQSfvjDH/L555/z1VdfceWVV/Lyyy8zefJkNtlkE37/+98D8Nvf/pbRo0dz8cUXc+aZZzJkyBCuueYafvCDH3DuuedWP4OyiBZ/RVvi7p+a2WPA1sAqZrZMuqrtDkxO2WqAtYEaM1sGWBmYnkkvyQ6TncbVwNUA/fr1a9q9S2RpUcuVZ1E23HBD7rzzzoXSPv/8cyZNmkTPnj158cUXFwlEZsb666/Piy++yP3338+pp57KbrvtVu+rtJdffpnevcud3zed5557bv5z1iFDhtC9e3cey3yhqqamhv6lL1dVoX379vOfFW+++eb07NmTt956a34A79mzJwA/+MEPuOCCCxYa9uWXXwZg/fXX57jjjuOJJ55g0KBBjB8/nl69ejV0Flu1Fn1Fa2ZdzGyV9P9ywC7AOGAkcGDKdihwb/p/eOom9f+3x5Y1HBiUaiWvC/QCnl88cyEijbXzzjsza9Ysrr/+egDmzZvHCSecwGGHHUbHjh0BGDFiBNOnT2f27Nncc889bLvttkyePJmOHTtyyCGHcOKJJ/LSSy/Va7qPP/44V199NUcccUSTz1PWVlttxejRoxk9ejR77703AwYM4OGHH+aTTz7hk08+4eGHH2bAgOpflpg2bRrz5kVdz7fffpvx48fzrW99i27dujF27FhKd+xGjBixyElE6Wr2q6++mj+ONm3azH8WLvXXogMt0BUYaWavAi8AI9z9X8DJwG/MbALxDPbalP9aYNWU/hvgFAB3fx24DRgLPAgcrRrHIksOM+Puu+/m9ttvp1evXqy//vp06NCB888/f36e7bbbjh/96Ef06dOHAw44gH79+jFmzBi23HJL+vTpw3nnnccZZ5wBxGs6w4cPLzutW2+9lT59+rD++utz/vnnc+edd9b7inb48OEVr5z79OlT5/CdO3fmzDPPZIsttmCLLbbgrLPOonPnzkA8Hy69ynP33XfTvXt3nnnmGfbcc8/5wfiJJ55gk002YdNNN+XAAw/kqquuonPnzqy11lqcffbZ7LDDDmyyySaMHj2a0047bf5077nnHrbYYgvWWmstVlllFbbZZhs23nhjzIxNN920XstAFrCmfv6wtOjXr5/rw+8tUAv68Htrer1n3Lhxhd8+lSVPue3CzF50937NVKQWqaVf0YqIiCzRFGhFREQKpEArIlXRYybJ0vZQPQVaEalThw4d+Pjjj3VwFWDB92g7dOjQ3EVZIiwx79GKSPPp3r07NTU1qCEXKenQoQPdu3dv7mIsERRoRaRO7dq1Y911123uYogskXTrWEREpEAKtCIiIgVSoBURESmQAq2IiEiBFGhFREQKpEArIiJSIAVaERGRAinQioiIFEiBVkREpEAKtCIiIgVSoBURESmQAq2IiEiBFGhFREQKpEArIiJSIAVaERGRAinQioiIFEiBVkREpEAKtCIiIgVSoBURESlQiw20Zra2mY00s3Fm9rqZHZfSB5vZ+2Y2Ov32yAxzqplNMLM3zWxAJn1gSptgZqc0x/yIiEjrtExzF6AWXwMnuPtLZrYi8KKZjUj9LnX3P2Yzm9kGwCBgQ2At4BEzWz/1vgLYFagBXjCz4e4+drHMhYiItGotNtC6+xRgSvr/CzMbB3SrZZB9gFvcfS7wjplNALZM/Sa4+9sAZnZLyqtAKyIihWuxt46zzKwH0Bd4LiUdY2avmtlQM+uU0roBkzKD1aS0SukiIiKFa/GB1sxWAO4Ejnf3z4ErgZ5AH+KK9+JS1jKDey3p5aZ1pJmNMrNR06ZNa3TZRUREWnSgNbN2RJC9yd3vAnD3D919nrt/A/yNBbeHa4C1M4N3BybXkr4Id7/a3fu5e78uXbo07cyIiEir1GIDrZkZcC0wzt0vyaR3zWTbD3gt/T8cGGRm7c1sXaAX8DzwAtDLzNY1s2WJClPDF8c8iIiItNjKUMC2wI+AMWY2OqWdBvyvmfUhbv9OBH4O4O6vm9ltRCWnr4Gj3X0egJkdAzwEtAWGuvvri3NGRESk9Wqxgdbdn6L889X7axnmPOC8Mun31zaciIhIUVrsrWMREZGlgQKtiIhIgRRoRURECqRAKyIiUiAFWhERkQIp0IqIiBRIgVZERKRACrQiIiIFarENVohIeT1Oua/Zpj3xgj2bbdoiSypd0YqIiBRIgVZERKRACrQiIiIFUqAVEREpkAKtiIhIgRRoRURECqTXe2SJ1pyvuoiIVENXtCIiIgVSoBURESmQAq2IiEiBFGhFREQKpEArIiJSIAVaERGRAinQioiIFEiBVkREpEAKtCIiIgVqsYHWzNY2s5FmNs7MXjez41J6ZzMbYWbj099OKd3M7M9mNsHMXjWzzTLjOjTlH29mhzbXPImISOvTYgMt8DVwgrv3BrYGjjazDYBTgEfdvRfwaOoG2B3olX5HAldCBGbgbGArYEvg7FJwFhERKVqLDbTuPsXdX0r/fwGMA7oB+wDXpWzXAfum//cBrvfwLLCKmXUFBgAj3H26u38CjAAGLsZZERGRVqzFBtosM+sB9AWeA9Zw9ykQwRhYPWXrBkzKDFaT0iqli4iIFK7FB1ozWwG4Ezje3T+vLWuZNK8lvdy0jjSzUWY2atq0afUvrIiISE6LDrRm1o4Isje5+10p+cN0S5j0d2pKrwHWzgzeHZhcS/oi3P1qd+/n7v26dOnSdDMiIiKtVosNtGZmwLXAOHe/JNNrOFCqOXwocG8m/cep9vHWwGfp1vJDwG5m1ilVgtotpYmIiBSuJX/4fVvgR8AYMxud0k4DLgBuM7OfAu8B30/97gf2ACYAs4DDAdx9upn9Dngh5Rvi7tMXzyyIiEhr12IDrbs/RfnnqwA7l8nvwNEVxjUUGNp0pRMREalOi711LCIisjRQoBURESmQAq2IiEiBCgu0ZraDmS3yDNjMljGzHYqaroiISEtS5BXtSKBzmfSVUz8REZGlXpGB1ijfAtOqwMwCpysiItJiNPnrPWY2PP3rwI1mNjfTuy2wEfB0U09XRESkJSriPdqP018DPgFmZ/p9CTwF/K2A6YqIiLQ4TR5o3f1wADObCPzR3XWbWEREWq3CWoZy93OKGreIiMiSorBAa2adgfOI5hJXJ1fxyt1XKmraIiIiLUWRbR1fS3ys/Wris3RlvwErIiKyNCsy0O4M7OruzxU4DRERkRatyPdopwIzChy/iIhIi1dkoD0dGGJmKxQ4DRERkRatyFvHZwA9gKlm9i7wVbanu29S4LRFRERahCID7R0FjltERGSJoPdoRURECqTv0YqIiBSoyAYrvqCWd2fVYIWIiLQGRT6jPSbX3Y5owOIAosUoERGRpV6Rz2ivK5duZi8RjVn8pahpi4iItBTN8Yx2JPC9ZpiuiIjIYtccgXYQ8FEzTFdERGSxK7Iy1BgWrgxlwBpAZ+CooqYrIiLSkizOBiu+AaYBj7n7G9WMwMyGAnsBU919o5Q2GDgijQvgNHe/P/U7FfgpMA841t0fSukDgcuAtsA17n5BI+ZLRESkai29wYphwOXA9bn0S939j9kEM9uAuC29IbAW8IiZrZ96XwHsCtQAL5jZcHcf2wTlExERqVWRV7QAmNlOwAbEbeTX3f2xaod19yfMrEeV2fcBbnH3ucA7ZjYB2DL1m+Dub6fy3JLyKtCKiEjhinxG2w24G9ic+PA7wFpmNgrYz90nVxy4bseY2Y+BUcAJ7v4J0A14NpOnJqUBTMqlb9WIaYuIiFStyFrHfyaela7n7mu7+9pAr5T250aM90qgJ9AHmAJcnNKtTF6vJX0RZnakmY0ys1HTpk0rl0VERKReigy0uwJHu/s7pYR0+/bY1K9B3P1Dd5/n7t8Af2PB7eEaYO1M1u7ElXSl9HLjvtrd+7l7vy5dujS0iCIiIvM1x3u03zRmYDPrmuncD3gt/T8cGGRm7c1sXeLq+XngBaCXma1rZssSFaaGN6YMIiIi1SqyMtSjwJ/N7H/dfRKAma1DvGbzaDUjMLObgf7AamZWA5wN9DezPsTt34nAzwHc/XUzu42o5PQ1cTU9L43nGOAh4vWeoe7+elPNpIiISG2KDLTHAvcCb5vZZCIwdgNeTf3q5O7/Wyb52lryn0eZDxak92zvr2aaIiIiTanI92gnAZuZ2a7Ad4hKSWPd/ZGipikiItLSNPkzWjPb3cwmmtnKAO4+wt3/4u5/JhqLmGhmuzX1dEVERFqiIipDHQNc5O6f5XuktAuB4wqYroiISItTRKDdBKjt9vC/gU0LmK6IiEiLU0Sg7ULtr/A4sGoB0xUREWlxigi0NcRVbSWbAO8XMF0REZEWp4hAex/wOzNbLt/DzDoCQ1IeERGRpV4Rr/ecBxwIjDezvwClb8/2JipKGXB+AdMVERFpcZo80Lr7VDP7LtH4//ksaNTfidaZfunuHzb1dEVERFqiQhqscPd3gT3MrBOwHhFsx6fP2YmIiLQahX74PQXWF4qchoiISEvWHF/vERERaTUUaEVERAqkQCsiIlIgBVoREZECKdCKiIgUSIFWRESkQAq0IiIiBVKgFRERKZACrYiISIEUaEVERAqkQCsiIlIgBVoREZECKdCKiIgUqNCv94jI0qXHKfc1y3QnXrBns0xXpCm06CtaMxtqZlPN7LVMWmczG2Fm49PfTindzOzPZjbBzF41s80ywxya8o83s0ObY15ERKR1atGBFhgGDMylnQI86u69gEdTN8DuQK/0OxK4EiIwA2cDWwFbAmeXgrOIiEjRWnSgdfcngOm55H2A69L/1wH7ZtKv9/AssIqZdQUGACPcfXr6EP0IFg3eIiIihWjRgbaCNdx9CkD6u3pK7wZMyuSrSWmV0kVERAq3JAbaSqxMmteSvugIzI40s1FmNmratGlNWjgREWmdlsRA+2G6JUz6OzWl1wBrZ/J1BybXkr4Id7/a3fu5e78uXbo0ecFFRKT1WRID7XCgVHP4UODeTPqPU+3jrYHP0q3lh4DdzKxTqgS1W0oTEREpXIt+j9bMbgb6A6uZWQ1Re/gC4DYz+ynwHvD9lP1+YA9gAjALOBzA3aeb2e+AF1K+Ie6er2AlIiJSiBYdaN39fyv02rlMXgeOrjCeocDQJiyaiIhIVZbEW8ciIiJLDAVaERGRAinQioiIFKhFP6OVJcfiamz+lrc/BmBQMzVuLyJSX7qiFRERKZACrYiISIEUaEVERAqkQCsiIlIgBVoREZECKdCKiIgUSIFWRESkQAq0IiIiBVKgFRERKZACrYiISIEUaEVERAqkQCsiIlIgBVoREZECKdCKiIgUSIFWRESkQAq0IiIiBVKgFRERKZACrYiISIEUaEVERAqkQCsiIlIgBVoREZECLbGB1swmmtkYMxttZqNSWmczG2Fm49PfTindzOzPZjbBzF41s82at/QiItJaLLGBNtnR3fu4e7/UfQrwqLv3Ah5N3QC7A73S70jgysVeUhERaZWW9ECbtw9wXfr/OmDfTPr1Hp4FVjGzrs1RQBERaV2W5EDrwMNm9qKZHZnS1nD3KQDp7+opvRswKTNsTUoTEREp1DLNXYBG2NbdJ5vZ6sAIM3ujlrxWJs0XyRQB+0iAddZZp2lKKSIirdoSe0Xr7pPT36nA3cCWwIelW8Lp79SUvQZYOzN4d2BymXFe7e793L1fly5diiy+iIi0EktkoDWz5c1sxdL/wG7Aa8Bw4NCU7VDg3vT/cODHqfbx1sBnpVvMIiIiRVpSbx2vAdxtZhDz8A93f9DMXgBuM7OfAu8B30/57wf2ACYAs4DDF3+RRUSkNVoiA627vw1sWib9Y2DnMukOHL0YiiYiIrKQJfLWsYiIyJJCgVZERKRACrQiIiIFUqAVEREpkAKtiIhIgRRoRURECqRAKyIiUiAFWhERkQIp0IqIiBRIgVZERKRACrQiIiIFUqAVEREp0BL5UQERaV16nHJfs0x34gV7Nst0ZemiK1oREZECKdCKiIgUSIFWRESkQAq0IiIiBVJlqKVIc1UYERGRynRFKyIiUiAFWhERkQIp0IqIiBRIgVZERKRACrQiIiIFUqAVEREpkAKtiIhIgVpNoDWzgWb2pplNMLNTmrs8IiLSOrSKBivMrC1wBbArUAO8YGbD3X1s85ZMRFqy5mwERl8OWnq0ikALbAlMcPe3AczsFmAfoJBAqxaaRESkpLXcOu4GTMp016Q0ERGRQrWWK1ork+aLZDI7Ejgydc4wszcLLNNqwEcFjn9JUO9lsE3pnwv3avLCNBNtB0HLIbcM7MJmLEnj/L/mLkBL01oCbQ2wdqa7OzA5n8ndrwauXhwFMrNR7t5vcUyrpdIy0DIo0XLQMliatZZbxy8AvcxsXTNbFhgEDG/mMomISCvQKq5o3f1rMzsGeAhoCwx199ebuVgiItIKtIpAC+Du9wP3N3c5MhbLLeoWTstAy6BEy0HLYKll7ovUCRIREZEm0lqe0YqIiDQLBdomZGadzWyEmY1PfztVyHdoyjPezA7NpG9uZmNSM5F/NjPLDXeimbmZrVb0vDRGUcvBzC4yszfM7FUzu9vMVllc81Stupr6NLP2ZnZr6v+cmfXI9Ds1pb9pZgOqHWdL09TLwMzWNrORZjbOzF43s+MW39w0XBHbQurX1sxeNrN/FT8X0iTcXb8m+gF/AE5J/58CXFgmT2fg7fS3U/q/U+r3PPGqqAEPALtnhlubqMz1LrBac89rcywHYDdgmfT/heXG28zz3Rb4L/AtYFngFWCDXJ5fAlel/wcBt6b/N0j52wPrpvG0rWacLelX0DLoCmyW8qwIvNWSl0FRyyEz3G+AfwD/au751K+6n65om9Y+wHXp/+uAfcvkGQCMcPfp7v4JMAIYaGZdgZXc/RmPven63PCXAr+lTEMbLVAhy8HdH3b3r9PwzxLvQ7ck85v6dPcvgVJTn1nZZXMHsHO6Yt8HuMXd57r7O8CENL5qxtmSNPkycPcp7v4SgLt/AYyj5bfsVsS2gJl1B/YErlkM8yBNRIG2aa3h7lMA0t/Vy+Sp1Bxkt/R/Ph0z2xt4391fKaLQBShkOeT8hLjabUmqaepzfp500vAZsGotwy5pzYcWsQzmS7dX+wLPNWGZi1DUcvgTccL9TdMXWYrSal7vaSpm9giwZplep1c7ijJpXindzDqmce9W5fgXi8W9HHLTPh34GripymktLtU09Vnf+S53MtyS72oUsQxiILMVgDuB49398waXcPFo8uVgZnsBU939RTPr38jyyWKkQFtP7r5LpX5m9qGZdXX3KekW6NQy2WqA/pnu7sBjKb17Ln0y0JN4TvNKqhPUHXjJzLZ09w8aMSuN0gzLoTTuQ4G9gJ3TreWWpJqmPkt5asxsGWBlYHodw9bZfGgLUsgyMLN2RJC9yd3vKqboTaqI5bA3sLeZ7QF0AFYysxvd/ZBiZkGaTHM/JF6afsBFLFwJ6A9l8nQG3iEqAHVK/3dO/V4AtmZBJaA9ygw/kZZfGaqQ5QAMJD5t2KW557HCfC9DVOpalwUVYDbM5TmahSvA3Jb+35CFK8C8TVSoqXOcLelX0DIw4ln9n5p7/ppzOeSG7Y8qQy0xv2YvwNL0I56vPAqMT39LgaMfcE0m30+ICg4TgMMz6f2A14hahpeTGhTJTWNJCLSFLIeUbxIwOv2uau55LTPvexC1Yv8LnJ7ShgB7p/87ALeneXke+FZm2NPTcG+ycI3zRcbZkn9NvQyA7Yhbqq9m1v0iJ6Et7VfEtpDpr0C7BP3UMpSIiEiBVOtYRESkQAq0IiIiBVKgFRERKZACrYiISIEUaEVERAqkQCuFMrPHzOzyTHdHM7vDzD5LXyLqUS6t+UpcjDSfbmb9mrssIrJ4KdBKvZnZsBQ03My+MrOp6TNmR6cWfLL2B07NdP8E2IF4N7Ir8V5subSlzSRi3kY3dkRm1tPMrjWzSWY218wmphOV7za+mMWrz0lHZjtzM5thZq+Y2WENmOZCJ3wii5MCrTTUI0Tg6EG0w/xP4BzgSTNbvpTJ4+s8X2SGWw8Y5+5j3P0Dd59XIa3eygT5FsPd56V5+7ru3JWl4PQS0XrQL4lPqu0NvAj8pdEFLZiZLduAwY4gtrVNgVuBv+e/0SrSojV3ixn6LXk/YBhlWqUBNgK+BM7JpD0GXJ753zO/x8qlpbzLEt+crQFmEs0yDsiMt3/KvwfRqs6XwF6p3/eIwDOHaNrxPGDZzLATgTOAvwKfp2mclJuXlYArgSlpPOOAgzL9vws8DswC3k95V6plmfVI5e2XK//OxJdoZgGjSN9drTAOI1rMeplck3yp/yqZ/zcmToZmE+3nDgNWzq9D4LhU/k+AvwMdU/+fAx+Svv+bGe4fwL2Z7mqW9WBgKPAp0RKSU2adV5hnBw7MpX0MXJzpXhW4Oa3H2cDrLNzS2LAy0+yR+m0A3Ad8QbTJfTOwZm45Ppq2ky+IphF3bO59UL8l69fsBdBvyftRIdCmfsOB1zLdj7Eg0HZOB9yniS//dC6XlvLeRHxzdgfi49nHEMF009S/FKjGEFfU3wK6EN+5/Rw4nPggw45EM3Z/zJRpYjpYH0NcTf8qjWub1N+A/xDtKg9M494d2C/13xiYAZwA9AK2Ap4B7qhlmfWgfKB9PpXxO8BDREBfpOnNNEzfNMwP61g/HYngeU8q6/8QTQHemVuHnwF/A3qnZfgpcGrq3wmYCwzMDLM8cdLz/dRd7bL+nPi023ppeW2R5mNAdp1XmJf5gZZo9/gHKe2CTJ5uwElAn7Sujkzbys6p/8rE9jU0TW9NFnxQ/iPihK43sAlxZ+Z5oE0adgxwY1o/6wH7lbYT/fSr9tfsBdBvyftRe6C9AJiV6X6MFGhT9+XkrmDyaemg/Q2wTi7fPcD/pf9LgeqAXJ4ngDNzafsSgbHU5OhE4OZcnvHAGen/XdP0e1eYx+uBa3NpfVJ5Vq8wTA/KB9rsVfq2Ka17hXGUgkzfOtbPEUQQXTGTVpreepl1OInMFSsRdB/JdN8N3JDpPiSNt0M9l/U/a1sWdcyLE1epM4hPIzoRHNerY7hbWLhd7YW2w5Q2BHg0l9YpTWPL1P05cGhz7Gf6LT0/fSZPmprR+O+lbpbGMzZ9GrCkPfDvXN5Rue7NgS3N7ORMWhtgOeJKZkpKezU33GQWfKC+LzDF3cdVKN/mwHpmdlAmrVTQnpT/LGAl2XKUPqO2OnEbNK/cd0rL6Q286gs/G3+aOHnYgGjEHmCsL/zMeDJxdV5yIzDMzDq6+yzgYOKqfU7qX+2yzq+j+joJeJD4dNwlwEXuXpoHzKwt8SDcXYwAAAN3SURBVJWog4ir2/bEo4fH6hjv5sAOZjajTL+exJXtJcA16fOMjxJ3Bd5o1NxIq6NAK01tA+KzXo3RhgjWWwBf5frNznXPLDPsOcSzwLxpmf/z43UWVA6sK6C1Aa4BLi3T7/06hs3LlqN0glKpkuJb6W9v4jltJbWd7GTTa1sGEM9wvwb2MbNHgV2IW8wl1S7r/Dqqrw9SYJ1gZt8nvsf8UibgnUjcxj+OuNU7AzifBSdOlbQhns+eWKbfhwDuPtjMbiIeHQwAzjazX7j70EbOk7QiCrTSZMxsI+KZ5rmNHNXLRLBY091H1nPYl4DvZK94GuAloKuZ9a5wVfsS8W3RxkyjIUYTz41PMrNbPVc728xWcfdPU56fmNmKmava7xKBpdJV+iLcfa6Z3UFcya4GfEBUACtp6LL+Mv1tW8/hcPcJZnYX8AeitjXEa2H/dPcbACxug6xPPHPOTjM/vZeI2/Hvunv+pCM7zfHEo4U/m9mVwM+I570iVdHrPdJQ7c1sTTNby8w2NbPfELfqXgT+2JgRu/tbRGWoYWZ2oJl9y8z6mdmJZrZ/HYMPAX5oZkPMbCMz+04axx/qUYRHiZrAd5rZADNb18x2NbN9U/8LiVumV5lZXzNbz8z2MrO/1ntm68HdnQUVj/6TptnTzDY2s98StYwhlt1M4PrUbweihvVdDQiKNxJXcr8A/uHu32T6NXRZTyXuTAwwszXMbOV6luliYC8z2zJ1vwXsbGbbmdl3iGf+6+aGmUissx5mtpqZtQGuICpK3WpmW6XtbBczu9rMVjSz5czsCjPrn4bbigjqY+tZXmnlFGiloXYhnsG9RwSmvYnbiDu4e2NvFUIElL8TVy5vELcxdwDerW0gd38I2JOoAft8+p2SylmVFEx2J2oe30hcBV5GPPfD3V9NZelBXOG9AvyedLuxSO7+PPFscRxwVfp7H7AlUYua9Dx1APGK0vPAvUSt6J80YJJPELfDNyCWRbYsDVrW6bnwscSV4eRUvqq5+xjipKJ05+TcNO0HUnlnEicbWX8krmrHEre113H3yUQFtG+IZ8CvE8F3bvrNIypHXUfUpr6bWI6/qU95RfThdxERkQLpilZERKRACrQiIiIFUqAVEREpkAKtiIhIgRRoRURECqRAKyIiUiAFWhERkQIp0IqIiBRIgVZERKRA/x8YTj5tIrRM/gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# histogram of 10000 differences in convert rates between groups\n",
    "plt.hist(x = p_diffs)\n",
    "plt.axvline(x = obs_p_diff, color = 'r', label = 'Obs. Diff. : {}%'.format(round(obs_p_diff*100, 3)))\n",
    "plt.title('Sampling Distribution of 10000 Differences\\nin Convert Rates between Control and Treatment Groups', fontsize = 16)\n",
    "plt.ylabel('Count', fontsize = 14)\n",
    "plt.xlabel('Difference in Convert Rates', fontsize = 14)\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Two-sample Z-test\n",
    "In an attempt to reproduce the results from the previous analysis, a two-sample Z-test was performed with the method `statsmodels.stats.proportion.proportions_ztest()`, the documentation for which is available [here](https://www.statsmodels.org/dev/generated/statsmodels.stats.proportion.proportions_ztest.html). This method requires for each of the two samples, the size of the sample or the number of users who received each page and the number of successes or the number of conversions for each page.\n",
    "* The size of each sample was computed earlier in __Sampling Distribution of Differences in Convert Rates__. `n_old` and `n_new` refer to the number of rows associated with the old page and new pages, respectively.\n",
    "* The number of conversions for the old page and the new page are computed below and stored in `convert_old` and `convert_new`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "from statsmodels.stats import proportion\n",
    "\n",
    "# number of conversions for each page\n",
    "convert_old = (df2.query('landing_page == \"old_page\"')['converted'] == 1).sum() # old page\n",
    "convert_new = (df2.query('landing_page == \"new_page\"')['converted'] == 1).sum() # new page"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the alternative hypothesis was defined as $p_{new} - p_{old} > 0$, the first elements specified in the `count` and `nobs` arguments correspond to the users with the new page whereas the second elements in these two arguments correspond to the users with the old page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-1.3109241984234394, 0.9050583127590245)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# z-score and p-value from z-test\n",
    "zscore, pval = proportion.proportions_ztest(count = [convert_new, convert_old] # number of conversions\n",
    "                                            , nobs = [n_new, n_old] # number of users in each page\n",
    "                                            , alternative = 'larger')\n",
    "zscore, pval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-1.3110034357328786, 0.9074)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# z-score and p-value from sampling distribution\n",
    "z_score = (obs_p_diff - p_diffs_mean) / p_diffs_std\n",
    "z_score, p_val"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The p-value obtained from the two-sample Z-test and the p-value obtained from the sampling distribution were both approximately 90%. Also, the z-score obtained from the Z-test and the z-score obtained from the sampling distribution were both approximately -1.3. Findings from both approaches indicated that the observed statistic was 1.3 standard deviation below the mean and suggest that the null hypothesis could not be rejected: the conversion rate of the old page must be equal to or larger than the conversion rate of the new page."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='regression'></a>\n",
    "## Part III - Regression Approach\n",
    "\n",
    "### 1. Conversion vs. Landing Page\n",
    "Because the analysis deals with a binary outcome, whether or not the `landing_page` resulted in a conversion, a logistic regression was also used to analyze the A/B test results. To begin performing this regression, a new column for the `intercept` was added to the dataframe `df2`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp    group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739  control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739  control     old_page          0   \n",
       "\n",
       "   intercept  \n",
       "0          1  \n",
       "1          1  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add intercept column\n",
    "df2['intercept'] = 1\n",
    "df2.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the categorical variable `landing_page` which will serve as a predictor variable along with the `intercept` column to predict conversions, a dummy variable called `ab_page` was created. Values 0 and 1 encode the old page and the new page, respectively. This dummy variable was added to the dataframe `df2` to create a new dataframe `df3` for this logistic regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create dummy variable for the landing page\n",
    "dummy = pd.get_dummies(data = df2['landing_page'])['new_page'] # column with 1 for new page and 0 for old page"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "\n",
       "   intercept  ab_page  \n",
       "0          1        0  \n",
       "1          1        0  \n",
       "2          1        1  \n",
       "3          1        1  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add the dummy column to df2 and create a new dataframe\n",
    "df3 = df2.join(dummy)\n",
    "df3.rename(columns = {'new_page':'ab_page'}, inplace = True)\n",
    "df3.head(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the `statsmodels`, the model was instantiated and fit with the two columns `intercept` and `ab_page` to predict the value for `converted`. The summary of this logistic regression is shown and discussed below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366118\n",
      "         Iterations 6\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Logit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>     <td>converted</td>    <th>  No. Observations:  </th>   <td>290584</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>               <td>Logit</td>      <th>  Df Residuals:      </th>   <td>290582</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>               <td>MLE</td>       <th>  Df Model:          </th>   <td>     1</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>          <td>Fri, 22 Mar 2019</td> <th>  Pseudo R-squ.:     </th>  <td>8.077e-06</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>              <td>19:38:07</td>     <th>  Log-Likelihood:    </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>           <td>True</td>       <th>  LL-Null:           </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th> </th>                      <td> </td>        <th>  LLR p-value:       </th>   <td>0.1899</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>   -1.9888</td> <td>    0.008</td> <td> -246.669</td> <td> 0.000</td> <td>   -2.005</td> <td>   -1.973</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>   -0.0150</td> <td>    0.011</td> <td>   -1.311</td> <td> 0.190</td> <td>   -0.037</td> <td>    0.007</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                           Logit Regression Results                           \n",
       "==============================================================================\n",
       "Dep. Variable:              converted   No. Observations:               290584\n",
       "Model:                          Logit   Df Residuals:                   290582\n",
       "Method:                           MLE   Df Model:                            1\n",
       "Date:                Fri, 22 Mar 2019   Pseudo R-squ.:               8.077e-06\n",
       "Time:                        19:38:07   Log-Likelihood:            -1.0639e+05\n",
       "converged:                       True   LL-Null:                   -1.0639e+05\n",
       "                                        LLR p-value:                    0.1899\n",
       "==============================================================================\n",
       "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "intercept     -1.9888      0.008   -246.669      0.000      -2.005      -1.973\n",
       "ab_page       -0.0150      0.011     -1.311      0.190      -0.037       0.007\n",
       "==============================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# logistic regression for conversion vs. page\n",
    "import statsmodels.api as sm\n",
    "\n",
    "log_mod = sm.Logit(df3['converted'], df3[['intercept', 'ab_page']])\n",
    "results = log_mod.fit()\n",
    "results.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, the p-value associated with `ab_page` is different from the p-values previously obtained from the sampling distribution and the two-sample Z-test because of the discrepancy in the null hypothesis between the linear regression and the other two approaches from __Part II__. This regression aims to provide estimates for the two constants, $\\beta_0$ and $\\beta_1$, in the model shown below in which $p_i$ and $ab\\_page_i$ denote the convert rate and the landing page for the $i$-th observation, respectively.\n",
    "$$log\\frac{p_i}{1-p_i} = \\beta_0 + \\beta_1 * ab\\_page_i$$\n",
    "Based on this model, the null hypothesis for the logistic regression is that whether a conversion occurs is independent of the landing page and can be quantitatively expressed as the following.\n",
    "$$\\beta_1 = 0$$\n",
    "Although the p-value associated with `ab_page` is relatively small, this null hypothesis could not be rejected because this p-value of 19% is larger than the tolerance level of 5%. In fact, the coefficient associated with `ab_page` indicates that holding all other variables constant, the convert rate for the old page is approximately 1.02 times the convert rate for the new page. In other words, the results from the logistic regression and those from the sampling distribution and the two-sample Z-test all indicate that the company should not implement the new page but keep the current one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.015113064615719"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# multiplicative increase in convert rate for old page\n",
    "beta1 = -0.0150\n",
    "1/np.exp(beta1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This recommendation for the company can also be verified by converting the p-values between two-tail and one-tail tests.\n",
    "* In case of the two-tail test, the proportion of extreme values below the expected values from the null is half the p-value from the logistic regression or `0.190/2=0.095`.\n",
    "* In case of the one-tail test, the proportion of extreme values that were larger than the observed statistic was approximately 91% according to the p-value obatined from sampling values from the null hypothesis.\n",
    "\n",
    "The p-value from the two-tail test can be converted to the p-value from the one-tail test by excluding the proportion of extreme values below the expected values from the null, `1−0.095 = 0.905`, which is approximately 91%. Again, because the p-value from either tests is larger than the tolerance level of 5%, the null hypothesis could not be rejected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to the landing page, there may be other factors which influence whether or not an individual converts. Adding these additional factors to the model may be beneficial in that the model may not only be fit more accurately but be generalized to predict outcomes from a new dataset. A potential disadvantage accompanied by this addition is the possibility of any correlation among the predictor variables added to the model to predict the outcome. In such case, interactive terms for the predictor variables which are dependent of each other need to be added, resulting in a more complex model which renders analysis of estimates less intuitive or more challenging."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Conversion vs. Landing Page and Country\n",
    "To explore the impact of the country where the user lives on the convert rate, data available in `countries.csv` was added to the previous dataframe `df3`. No cleaning of the resulting dataframe `df_new` was necessary besides resetting the index to perform the `join()` because `countries.csv` and `df3` not only have the same number of rows but have the same number of unique users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>834778</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>928468</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>822059</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>711597</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>710616</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id country\n",
       "0   834778      UK\n",
       "1   928468      US\n",
       "2   822059      UK\n",
       "3   711597      UK\n",
       "4   710616      UK"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# countries dataset\n",
    "countries_df = pd.read_csv('data/countries.csv')\n",
    "countries_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of rows in countries_df and df3 is the same\n",
    "countries_df.shape[0] == df3.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of unique users in countries_df and df3 is the same\n",
    "countries_df.user_id.nunique() == df3.user_id.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>country</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>834778</td>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-14 23:08:43.304998</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>928468</td>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-23 14:44:16.387854</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>822059</td>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-16 14:04:14.719771</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>711597</td>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-22 03:14:24.763511</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>710616</td>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-16 13:14:44.000513</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id country                   timestamp      group landing_page  \\\n",
       "0   834778      UK  2017-01-14 23:08:43.304998    control     old_page   \n",
       "1   928468      US  2017-01-23 14:44:16.387854  treatment     new_page   \n",
       "2   822059      UK  2017-01-16 14:04:14.719771  treatment     new_page   \n",
       "3   711597      UK  2017-01-22 03:14:24.763511    control     old_page   \n",
       "4   710616      UK  2017-01-16 13:14:44.000513  treatment     new_page   \n",
       "\n",
       "   converted  intercept  ab_page  \n",
       "0          0          1        0  \n",
       "1          0          1        1  \n",
       "2          1          1        1  \n",
       "3          0          1        0  \n",
       "4          0          1        1  "
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# join countries_df and df3\n",
    "df_new = countries_df.set_index('user_id').join(df3.set_index('user_id'), how='inner').reset_index()\n",
    "df_new.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To include country to the logistic regression, a dummy variable was added to `df_new` for each unique value of `country`. `US` was selected as the baseline among the three dummy variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "US    203619\n",
       "UK     72466\n",
       "CA     14499\n",
       "Name: country, dtype: int64"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# unique values for country\n",
    "df_new.country.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>country</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>CA</th>\n",
       "      <th>UK</th>\n",
       "      <th>US</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>834778</td>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-14 23:08:43.304998</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>928468</td>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-23 14:44:16.387854</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>822059</td>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-16 14:04:14.719771</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>711597</td>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-22 03:14:24.763511</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>710616</td>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-16 13:14:44.000513</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id country                   timestamp      group landing_page  \\\n",
       "0   834778      UK  2017-01-14 23:08:43.304998    control     old_page   \n",
       "1   928468      US  2017-01-23 14:44:16.387854  treatment     new_page   \n",
       "2   822059      UK  2017-01-16 14:04:14.719771  treatment     new_page   \n",
       "3   711597      UK  2017-01-22 03:14:24.763511    control     old_page   \n",
       "4   710616      UK  2017-01-16 13:14:44.000513  treatment     new_page   \n",
       "\n",
       "   converted  intercept  ab_page  CA  UK  US  \n",
       "0          0          1        0   0   1   0  \n",
       "1          0          1        1   0   0   1  \n",
       "2          1          1        1   0   1   0  \n",
       "3          0          1        0   0   1   0  \n",
       "4          0          1        1   0   1   0  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create dummy variables\n",
    "df_new[['CA', 'UK', 'US']] = pd.get_dummies(df_new['country'])\n",
    "df_new.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As shown below in the summary of the logistic regression, the pseudo R-squared increased by an order of magnitude with the addition of `country` to the predictor variables. This change suggests an increase in the capability of this new model to predict the conversion from the predictor variables. Also, the coefficients associated with the three countries show that holding all other variables constant, convert rate increases in the order of `CA`, `US`, and `UK`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366113\n",
      "         Iterations 6\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Logit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>     <td>converted</td>    <th>  No. Observations:  </th>   <td>290584</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>               <td>Logit</td>      <th>  Df Residuals:      </th>   <td>290580</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>               <td>MLE</td>       <th>  Df Model:          </th>   <td>     3</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>          <td>Fri, 22 Mar 2019</td> <th>  Pseudo R-squ.:     </th>  <td>2.323e-05</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>              <td>19:38:11</td>     <th>  Log-Likelihood:    </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>           <td>True</td>       <th>  LL-Null:           </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th> </th>                      <td> </td>        <th>  LLR p-value:       </th>   <td>0.1760</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>   -1.9893</td> <td>    0.009</td> <td> -223.763</td> <td> 0.000</td> <td>   -2.007</td> <td>   -1.972</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>   -0.0149</td> <td>    0.011</td> <td>   -1.307</td> <td> 0.191</td> <td>   -0.037</td> <td>    0.007</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>CA</th>        <td>   -0.0408</td> <td>    0.027</td> <td>   -1.516</td> <td> 0.130</td> <td>   -0.093</td> <td>    0.012</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>UK</th>        <td>    0.0099</td> <td>    0.013</td> <td>    0.743</td> <td> 0.457</td> <td>   -0.016</td> <td>    0.036</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                           Logit Regression Results                           \n",
       "==============================================================================\n",
       "Dep. Variable:              converted   No. Observations:               290584\n",
       "Model:                          Logit   Df Residuals:                   290580\n",
       "Method:                           MLE   Df Model:                            3\n",
       "Date:                Fri, 22 Mar 2019   Pseudo R-squ.:               2.323e-05\n",
       "Time:                        19:38:11   Log-Likelihood:            -1.0639e+05\n",
       "converged:                       True   LL-Null:                   -1.0639e+05\n",
       "                                        LLR p-value:                    0.1760\n",
       "==============================================================================\n",
       "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "intercept     -1.9893      0.009   -223.763      0.000      -2.007      -1.972\n",
       "ab_page       -0.0149      0.011     -1.307      0.191      -0.037       0.007\n",
       "CA            -0.0408      0.027     -1.516      0.130      -0.093       0.012\n",
       "UK             0.0099      0.013      0.743      0.457      -0.016       0.036\n",
       "==============================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# logistic regression for conversion vs. page, country\n",
    "log_mod = sm.Logit(df_new['converted'], df_new[['intercept', 'ab_page', 'CA', 'UK']])\n",
    "results = log_mod.fit()\n",
    "results.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.0416437559600236, 1.0099491671175422)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1/(np.exp(-0.0408)), np.exp(0.0099)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Interaction Terms\n",
    "To explore the impact of the interaction between page and country on the conversion, two new columns `ab_page_CA` and `ab_page_UK`, which represent the interaction terms, were added to `df_new`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>country</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>CA</th>\n",
       "      <th>UK</th>\n",
       "      <th>US</th>\n",
       "      <th>ab_page_CA</th>\n",
       "      <th>ab_page_UK</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>834778</td>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-14 23:08:43.304998</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>928468</td>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-23 14:44:16.387854</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>822059</td>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-16 14:04:14.719771</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>711597</td>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-22 03:14:24.763511</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>710616</td>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-16 13:14:44.000513</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id country                   timestamp      group landing_page  \\\n",
       "0   834778      UK  2017-01-14 23:08:43.304998    control     old_page   \n",
       "1   928468      US  2017-01-23 14:44:16.387854  treatment     new_page   \n",
       "2   822059      UK  2017-01-16 14:04:14.719771  treatment     new_page   \n",
       "3   711597      UK  2017-01-22 03:14:24.763511    control     old_page   \n",
       "4   710616      UK  2017-01-16 13:14:44.000513  treatment     new_page   \n",
       "\n",
       "   converted  intercept  ab_page  CA  UK  US  ab_page_CA  ab_page_UK  \n",
       "0          0          1        0   0   1   0           0           0  \n",
       "1          0          1        1   0   0   1           0           0  \n",
       "2          1          1        1   0   1   0           0           1  \n",
       "3          0          1        0   0   1   0           0           0  \n",
       "4          0          1        1   0   1   0           0           1  "
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add interaction terms to the dataframe\n",
    "df_new['ab_page_CA'] = df_new['ab_page'] * df_new['CA']\n",
    "df_new['ab_page_UK'] = df_new['ab_page'] * df_new['UK']\n",
    "df_new.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the addition of the two interaction terms, the pseudo R-square increased from the previous model. However, the magnitude of the increase is less than the increase which resulted from the addition of the country to the predictor variables. In other words, the capability of the model to predict conversion increased with the addition of the two interaction terms but not to a degree that was accomplished by the addition of the country to the logistic regression model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366109\n",
      "         Iterations 6\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Logit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>     <td>converted</td>    <th>  No. Observations:  </th>   <td>290584</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>               <td>Logit</td>      <th>  Df Residuals:      </th>   <td>290578</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>               <td>MLE</td>       <th>  Df Model:          </th>   <td>     5</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>          <td>Fri, 22 Mar 2019</td> <th>  Pseudo R-squ.:     </th>  <td>3.482e-05</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>              <td>19:38:14</td>     <th>  Log-Likelihood:    </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>           <td>True</td>       <th>  LL-Null:           </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th> </th>                      <td> </td>        <th>  LLR p-value:       </th>   <td>0.1920</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "       <td></td>         <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th>  <td>   -1.9865</td> <td>    0.010</td> <td> -206.344</td> <td> 0.000</td> <td>   -2.005</td> <td>   -1.968</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>    <td>   -0.0206</td> <td>    0.014</td> <td>   -1.505</td> <td> 0.132</td> <td>   -0.047</td> <td>    0.006</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>CA</th>         <td>   -0.0175</td> <td>    0.038</td> <td>   -0.465</td> <td> 0.642</td> <td>   -0.091</td> <td>    0.056</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>UK</th>         <td>   -0.0057</td> <td>    0.019</td> <td>   -0.306</td> <td> 0.760</td> <td>   -0.043</td> <td>    0.031</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page_CA</th> <td>   -0.0469</td> <td>    0.054</td> <td>   -0.872</td> <td> 0.383</td> <td>   -0.152</td> <td>    0.059</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page_UK</th> <td>    0.0314</td> <td>    0.027</td> <td>    1.181</td> <td> 0.238</td> <td>   -0.021</td> <td>    0.084</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                           Logit Regression Results                           \n",
       "==============================================================================\n",
       "Dep. Variable:              converted   No. Observations:               290584\n",
       "Model:                          Logit   Df Residuals:                   290578\n",
       "Method:                           MLE   Df Model:                            5\n",
       "Date:                Fri, 22 Mar 2019   Pseudo R-squ.:               3.482e-05\n",
       "Time:                        19:38:14   Log-Likelihood:            -1.0639e+05\n",
       "converged:                       True   LL-Null:                   -1.0639e+05\n",
       "                                        LLR p-value:                    0.1920\n",
       "==============================================================================\n",
       "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "intercept     -1.9865      0.010   -206.344      0.000      -2.005      -1.968\n",
       "ab_page       -0.0206      0.014     -1.505      0.132      -0.047       0.006\n",
       "CA            -0.0175      0.038     -0.465      0.642      -0.091       0.056\n",
       "UK            -0.0057      0.019     -0.306      0.760      -0.043       0.031\n",
       "ab_page_CA    -0.0469      0.054     -0.872      0.383      -0.152       0.059\n",
       "ab_page_UK     0.0314      0.027      1.181      0.238      -0.021       0.084\n",
       "==============================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# logistic regression for conversion vs. page, country, including interaction terms\n",
    "log_mod = sm.Logit(df_new['converted'], df_new[['intercept', 'ab_page', 'CA', 'UK', 'ab_page_CA', 'ab_page_UK']])\n",
    "results = log_mod.fit()\n",
    "results.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='conclusions'></a>\n",
    "## Conclusions\n",
    "\n",
    "In this project, sampling distribution, two-sample Z-test, and logistic regressions were used to analyze the A/B tests conducted by the e-commerce company.\n",
    "\n",
    "The relatively small p-values obtained from both the sampling distribution of the differences in the convert rates between the treatment and the control groups and the two-sample Z-test did not provide enough evidence for rejecting the null hypothesis. In other words, according to the A/B tests, the company was not recommended to implement the new page because the convert rate for the old page was determined to be equal to or larger than the convert rate for the new page. Additional data concerned with the context in which the timestamp of each transaction was recorded or the interpretation of these timestamps would introduce potential improvements to the A/B testing in that outcomes of transactions can be monitored continuously. Findings from a more comprehensive analysis would provide the company with insights into the optimal duration of the A/B testing before the company makes a decision on implementing the new page or keeping the old page.\n",
    "\n",
    "The logistic regressions did not provide similar p-values as the sampling distribution and the two-sample Z-test because the null and the alternative hypotheses for the logistic regressions were concerned with whether the outcome can be predicted with certain predictor variables. However, conversion of the p-value from the two-tail test to the one-tail test showed that the findings from the logistic regression also supported the null hypothesis. The first model aimed to predict the conversion with the landing page. The second model added country where the users live to the predictor variables of the first model. The third model included interaction terms among the predictor variables from the second model. From the first to the second and the third models, increase in the pseudo R-square suggested an increase in the capability of the model to account for the observed outcome and predict conversion."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}