Supervised_Learning_Models.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Supervised Learning: Models\n",
        "**Supervised learning** is an area of machine learning where the chosen algorithm tries to fit a target using the given input. A set of training data that contains labels is supplied to the algorithm. Based on a massive set of data, the algorithm will learn a rule that it uses to predict the labels for new observations. In other words, supervised learning algorithms are provided with historical data and asked to find the relationship that has the best predictive power.\n",
        "\n",
        "There are two varieties of supervised learning algorithms: regression and classification algorithms.\n",
        "\n",
        "**a. Regression-based supervised learning** methods try to predict outputs based on input variables. Classification-based supervised learning methods identify which category a set of data items belongs to.\n",
        "\n",
        "**b. Classification algorithms** are probability-based, meaning the outcome is the category for which the algorithm finds the highest probability that the dataset belongs to it. Regression algorithms, in contrast, estimate the outcome of problems that have an infinite number of solutions (continuous set of possible outcomes).\n",
        "\n"
      ],
      "metadata": {
        "id": "TCQsj8wMCkDI"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Supervised Learning Models: An Overview\n",
        "\n",
        "**Classification predictive modelling** problems are different from regression predictive modelling problems, as classification is the task of predicting a discrete class label and regression is the task of predicting a continuous quantity. However, both share the same concept of utilizing known variables to make predictions, and there is a significant overlap between the two models. Hence, the models for classification and regression are presented together in this chapter.  [Figure 1](https://www.oreilly.com/library/view/machine-learning-and/9781492073048/ch04.html)  summarizes the list of the models commonly used for classification and regression.\n",
        "\n",
        "Some models can be used for both classification and regression with small modifications. These are  K-nearest neighbors, decision trees, support vector, ensemble bagging/boosting methods, and ANNs (including deep neural networks), as shown in  [Figure 1](https://www.oreilly.com/library/view/machine-learning-and/9781492073048/ch04.html) . However, some models, such as linear regression and logistic regression, cannot (or cannot easily) be used for both problem types.\n",
        "\n",
        "#Figure 1. Models for regression and classification\n",
        "This section contains the following details about the models:\n",
        "\n",
        "1. Theory of the models.\n",
        "2. Implementation in Scikit-learn or Keras.\n",
        "3. Grid search for different models.\n",
        "4. Pros and cons of the models."
      ],
      "metadata": {
        "id": "Wa5cwC78C6E-"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Linear Regression (Ordinary Least Squares)\n",
        "**Linear regression (Ordinary Least Squares Regression or OLS Regression)** is perhaps one of the most well-known and best-understood algorithms in statistics and machine learning. Linear regression is a linear model, e.g., a model that assumes a linear relationship between the input variables ( x) and the single output variable ( y). The goal of linear regression is to train a linear model to predict a new  y given a previously unseen  x with as little error as possible.\n",
        "\n",
        "Our model will be a function that predicts  y given x 1 , x 2 . . . x i :\n",
        "\n",
        "\n",
        "y = β 0 + β 1 x 1 + . . . + β i x i\n",
        "\n",
        "\n",
        "where, β 0 is called intercept and β 1 . . . β i are the coefficient of the regression.\n",
        "# Implementation in Python\n",
        "**from sklearn.linear_model import LinearRegression**\n",
        "\n",
        "model = LinearRegression()\n",
        "\n",
        "\n",
        "model.fit(X, Y)\n",
        "\n",
        "\n",
        "In the following section, we cover the training of a linear regression model and grid search of the model. However, the overall concepts and related approaches are applicable to all other supervised learning models.\n",
        "\n",
        "Mathematically, we look at the difference between each real data point ( y) and our model’s prediction ( ŷ). Square these differences to avoid negative numbers and penalize larger differences, and then add them up and take the average. This is a measure of how well our data fits the line.\n",
        "\n",
        "Image label\n",
        "\n",
        "Figure 2. Linear regression\n",
        "#Grid search\n",
        "The overall idea of the grid search is to create a grid of all possible hyperparameter combinations and train the model using each one of them. Hyperparameters are the external characteristic of the model, can be considered the model’s settings, and are not estimated based on data-like model parameters. These hyperparameters are tuned during grid search to achieve better model performance.\n",
        "\n",
        "Due to its exhaustive search, a grid search is guaranteed to find the optimal parameter within the grid. The drawback is that the size of the grid grows exponentially with the addition of more parameters or more considered values.\n",
        "\n",
        "\n",
        "**The GridSearchCV** class in the model_selection module of the sklearn package facilitates the systematic evaluation of all combinations of the hyperparameter values that we would like to test.\n",
        "\n",
        "\n",
        "The first step is to create a model object. We then define a dictionary where the keywords name the hyperparameters and the values list the parameter settings to be tested. For linear regression, the hyperparameter is fit_intercept, which is a boolean variable that determines whether or not to calculate the  intercept for this model. If set to False, no intercept will be used in calculations:\n",
        "\n",
        "\n",
        "model = LinearRegression()\n",
        "\n",
        "\n",
        "param_grid = {'fit_intercept': [ True, False]}\n",
        "\n",
        "\n",
        "}\n",
        "\n",
        "\n",
        "The second step is to instantiate the GridSearchCV object and provide the estimator object and parameter grid, as well as a scoring method and cross validation choice, to the initialization method. Cross validation is a resampling procedure used to evaluate machine learning models, and scoring parameter is the evaluation metrics of the model: 1\n",
        "\n",
        "\n",
        "With all settings in place, we can fit GridSearchCV:\n",
        "\n",
        "\n",
        "grid = GridSearchCV(estimator=model, param_grid=param_grid, scoring= 'r2', \\\n",
        "\n",
        "\n",
        "cv=kfold)\n",
        "\n",
        "\n",
        "grid_result = grid.fit(X, Y)\n",
        "\n",
        "\n",
        "#Advantages and disadvantages\n",
        "In terms of advantages, linear regression is easy to understand and interpret. However, it may not work well when there is a nonlinear relationship between predicted and predictor variables. Linear regression is prone to  overfitting (which we will discuss in the next section) and when a large number of features are present, it may not handle irrelevant features well. Linear regression also requires the data to follow certain  assumptions , such as the absence of multicollinearity. If the assumptions fail, then we cannot trust the results obtained.\n"
      ],
      "metadata": {
        "id": "wdCgTpdDDeE4"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Regularized Regression\n",
        "When a linear regression model contains many independent variables, their coefficients will be poorly determined, and the model will have a tendency to fit extremely well to the training data (data used to build the model) but fit poorly to testing data (data used to test how good the model is). This is known as overfitting or high variance.\n",
        "\n",
        "One popular technique to control overfitting is  regularization, which involves the addition of a  penalty term to the error or loss function to discourage the coefficients from reaching large values. Regularization, in simple terms, is a penalty mechanism that applies shrinkage to model parameters (driving them closer to zero) in order to build a model with higher prediction accuracy and interpretation. Regularized regression has two advantages over linear regression:\n",
        "\n",
        "**Prediction accuracy**\n",
        "The performance of the model working better on the testing data suggests that the model is trying to generalize from training data. A model with too many parameters might try to fit noise specific to the training data. By shrinking or setting some coefficients to zero, we trade off the ability to fit complex models (higher bias) for a more generalizable model (lower variance).\n",
        "\n",
        "**Interpretation**\n",
        "A large number of predictors may complicate the interpretation or communication of the big picture of the results. It may be preferable to sacrifice some detail to limit the model to a smaller subset of parameters with the strongest effects.\n",
        "\n",
        "The common ways to regularize a linear regression model are as follows:\n",
        "\n",
        "\n",
        "a. L1 regularization or Lasso regression\n",
        "Lasso regression performs  L1 regularization by adding a factor of the sum of the absolute value of coefficients in the cost function (RSS) for linear regression, as mentioned in  [Equation 4-1](https://www.oreilly.com/library/view/machine-learning-and/9781492073048/ch04.html) . The equation for lasso regularization can be represented as follows:\n",
        "\n",
        "C o s t F u n c t i o n = R S S + λ * ∑ j=1 p β j\n",
        "\n",
        "\n",
        "L1 regularization can lead to zero coefficients (i.e., some of the features are completely neglected for the evaluation of output). The larger the value of λ , the more features are shrunk to zero. This can eliminate some features entirely and give us a subset of predictors, reducing model complexity. So Lasso regression not only helps in reducing overfitting, but also can help in feature selection. Predictors not shrunk toward zero signify that they are important, and thus L1 regularization allows for feature selection (sparse selection). The regularization parameter ( λ ) can be controlled, and a lambda value of zero produces the basic linear regression equation.\n",
        "\n",
        "\n",
        "A lasso regression model can be constructed using the Lasso class of the sklearn package of Python, as shown in the code snippet that follows:\n",
        "\n",
        "\n",
        "**from sklearn.linear_model import** Lasso\n",
        "\n",
        "\n",
        "model = Lasso()\n",
        "\n",
        "\n",
        "model.fit(X, Y)\n",
        "\n",
        "\n",
        "**b. L2 regularization or Ridge regression**\n",
        "Ridge regression performs  L2 regularization by adding a factor of the sum of the square of coefficients in the cost function (RSS) for linear regression, as mentioned in  [Equation 1](https://www.oreilly.com/library/view/machine-learning-and/9781492073048/ch04.html) . The equation for ridge regularization can be represented as follows:\n",
        "\n",
        "C o s t F u n c t i o n = R S S + λ * ∑ j=1 p β j 2\n",
        "\n",
        "\n",
        "Ridge regression puts constraint on the coefficients. The penalty term ( λ ) regularizes the coefficients such that if the coefficients take large values, the optimization function is penalized. So ridge regression shrinks the coefficients and helps to reduce the model complexity. Shrinking the coefficients leads to a lower variance and a lower error value. Therefore, ridge regression decreases the complexity of a model but does not reduce the number of variables; it just shrinks their effect. When λ is closer to zero, the cost function becomes similar to the linear regression cost function. So the lower the constraint (low λ ) on the features, the more the model will resemble the linear regression model.\n",
        "\n",
        "\n",
        "A ridge regression model can be constructed using the Ridge class of the sklearn package of Python, as shown in the code snippet that follows:\n",
        "\n",
        "\n",
        "**from sklearn.linear_model import** Ridge\n",
        "\n",
        "\n",
        "model = Ridge()\n",
        "\n",
        "\n",
        "model.fit(X, Y)\n",
        "\n",
        "\n",
        "**c. Elastic net**\n",
        "Elastic nets add regularization terms to the model, which are a combination of both L1 and L2 regularization, as shown in the following equation:\n",
        "\n",
        "C o s t F u n c t i o n = R S S + λ * (1–α) / 2 * ∑ j=1 p β j 2 + α * ∑ j=1 p β j\n",
        "\n",
        "\n",
        "In addition to setting and choosing a λ value, an elastic net also allows us to tune the alpha parameter, where α =  0 corresponds to ridge and α =  1 to lasso. Therefore, we can choose an α value between  0 and  1 to optimize the elastic net. Effectively, this will shrink some coefficients and set some to  0 for sparse selection.\n",
        "\n",
        "\n",
        "An elastic net regression model can be constructed using the ElasticNet class of the sklearn package of Python, as shown in the following code snippet:\n",
        "\n",
        "\n",
        "**from sklearn.linear_model import** ElasticNet\n",
        "\n",
        "\n",
        "model = ElasticNet()\n",
        "\n",
        "\n",
        "model.fit(X, Y)\n",
        "\n",
        "\n",
        "For all the regularized regression, λ is the key parameter to tune during grid search in Python. In an elastic net, α can be an additional parameter to tune.\n"
      ],
      "metadata": {
        "id": "7kzxIwCnEDAJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Logistic Regression\n",
        "Logistic regression is one of the most widely used algorithms for classification. The logistic regression model arises from the desire to model the probabilities of the output classes given a function that is linear in  x, at the same time ensuring that output probabilities sum up to one and remain between zero and one as we would expect from probabilities.\n",
        "\n",
        "If we train a linear regression model on several examples where  Y = 0 or  1, we might end up predicting some probabilities that are less than zero or greater than one, which doesn’t make sense. Instead, we use a logistic regression model (or  logit model), which is a modification of linear regression that makes sure to output a probability between zero and one by applying the sigmoid function. [2](https://www.oreilly.com/library/view/machine-learning-and/9781492073048/ch04.html)\n",
        "\n",
        "[Equation 4-2](https://www.oreilly.com/library/view/machine-learning-and/9781492073048/ch04.html)  shows the equation for a logistic regression model. Similar to linear regression, input values ( x) are combined linearly using weights or coefficient values to predict an output value ( y). The output coming from  [Equation 4-2](https://www.oreilly.com/library/view/machine-learning-and/9781492073048/ch04.html)  is a probability that is transformed into a binary value ( 0 or  1) to get the model prediction.\n",
        "\n",
        "\n",
        "Equation 4-2. Logistic regression equation\n",
        "\n",
        "\n",
        "y = exp(β 0 +β 1 x 1 +....+β i x 1 ) 1+exp(β 0 +β 1 x 1 +....+β i x 1 )\n",
        "\n",
        "\n",
        "Where  y is the predicted output, β 0 is the bias or intercept term and B1 is the coefficient for the single input value ( x). Each column in the input data has an associated β coefficient (a constant real value) that must be learned from the training data.\n",
        "\n",
        "\n",
        "In logistic regression, the cost function is basically a measure of how often we predicted one when the true answer was zero, or vice versa. Training the logistic regression coefficients is done using techniques such as maximum likelihood estimation (MLE) to predict values close to  1 for the default class and close to  0 for the other class. [3](https://www.oreilly.com/library/view/machine-learning-and/9781492073048/ch04.html)\n",
        "\n",
        "\n",
        "A logistic regression model can be constructed using the LogisticRegression class of the sklearn package of Python, as shown in the following code snippet:\n",
        "\n",
        "\n",
        "**from sklearn.linear_model import** LogisticRegression\n",
        "\n",
        "\n",
        "model = LogisticRegression()\n",
        "\n",
        "\n",
        "model.fit(X, Y)\n",
        "#Hyperparameters\n",
        "Regularization (penalty in sklearn)\n",
        "\n",
        "Similar to linear regression, logistic regression can have regularization, which can be L1, L2, or elasticnet. The values in the sklearn library are [l1, l2, elasticnet].\n",
        "\n",
        "\n",
        "Regularization strength (C in sklearn)\n",
        "\n",
        "\n",
        "This parameter controls the regularization strength. Good values of the penalty parameters can be [100, 10, 1.0, 0.1, 0.01].\n",
        "\n",
        "#Advantages and disadvantages\n",
        "\n",
        "In terms of the advantages, the logistic regression model is easy to implement, has good interpretability, and performs very well on linearly separable classes. The output of the model is a probability, which provides more insight and can be used for ranking. The model has small number of hyperparameters. Although there may be risk of overfitting, this may be addressed using  L1/L2 regularization, similar to the way we addressed overfitting for the linear regression models.\n",
        "\n",
        "In terms of disadvantages, the model may overfit when provided with large numbers of features. Logistic regression can only learn linear functions and is less suitable to complex relationships between features and the target variable. Also, it may not handle irrelevant features well, especially if the features are strongly correlated.\n"
      ],
      "metadata": {
        "id": "G7tBP2hzE7p7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Support Vector Machine\n",
        "The objective of the  support vector machine (SVM) algorithm is to maximize the margin (shown as shaded area in  [Figure 3](https://www.oreilly.com/library/view/machine-learning-and/9781492073048/ch04.html) ), which is defined as the distance between the separating hyperplane (or decision boundary) and the training samples that are closest to this hyperplane, the so-called support vectors. The margin is calculated as the perpendicular distance from the line to only the closest points, as shown in  [Figure 4-3](https://www.oreilly.com/library/view/machine-learning-and/9781492073048/ch04.html) . Hence, SVM calculates a maximum-margin boundary that leads to a homogeneous partition of all data points.\n",
        "\n",
        "\n",
        "Image label\n",
        "\n",
        "**Figure 3. Support vector machine**\n",
        "\n",
        "In practice, the data is messy and cannot be separated perfectly with a hyperplane. The constraint of maximizing the margin of the line that separates the classes must be relaxed. This change allows some points in the training data to violate the separating line. An additional set of coefficients is introduced that give the margin wiggle room in each dimension. A tuning parameter is introduced, simply called  C, that defines the magnitude of the wiggle allowed across all dimensions. The larger the value of  C, the more violations of the hyperplane are permitted.\n",
        "\n",
        "\n",
        "In some cases, it is not possible to find a hyperplane or a linear decision boundary, and kernels are used. A kernel is just a transformation of the input data that allows the SVM algorithm to treat/process the data more easily. Using kernels, the original data is projected into a higher dimension to classify the data better.\n",
        "\n",
        "\n",
        "SVM is used for both classification and regression. We achieve this by converting the original optimization problem into a dual problem. For regression, the trick is to reverse the objective. Instead of trying to fit the largest possible street between two classes while limiting margin violations, SVM regression tries to fit as many instances as possible on the street (shaded area in  [Figure 4-3](https://www.oreilly.com/library/view/machine-learning-and/9781492073048/ch04.html) ) while limiting margin violations. The width of the street is controlled by a hyperparameter.\n",
        "\n",
        "\n",
        "The SVM regression and classification models can be constructed using the sklearn package of Python, as shown in the following code snippets:\n",
        "\n",
        "\n",
        "Regression\n",
        "\n",
        "\n",
        "**from sklearn.svm import** SVR\n",
        "\n",
        "\n",
        "model = SVR()\n",
        "\n",
        "\n",
        "model.fit(X, Y)\n",
        "\n",
        "\n",
        "Classification\n",
        "\n",
        "\n",
        "**from sklearn.svm import** SVC\n",
        "\n",
        "\n",
        "model = SVC()\n",
        "\n",
        "\n",
        "model.fit(X, Y)\n",
        "\n",
        "\n",
        "**Hyperparameters**\n",
        "The following key parameters are present in the sklearn implementation of SVM and can be tweaked while performing the grid search:\n",
        "\n",
        "Kernels (kernel in sklearn)\n",
        "\n",
        "\n",
        "The choice of kernel controls the manner in which the input variables will be projected. There are many kernels to choose from, but  linear and  RBF  are the most common.\n",
        "\n",
        "\n",
        "**Penalty (C in sklearn)**\n",
        "\n",
        "\n",
        "The penalty parameter tells the SVM optimization how much you want to avoid misclassifying each training example. For large values of the penalty parameter, the optimization will choose a smaller-margin hyperplane. Good values might be a log scale from 10 to 1,000.\n",
        "\n",
        "\n",
        "#Advantages and disadvantages\n",
        "In terms of advantages, SVM is fairly robust against overfitting, especially in higher dimensional space. It handles the nonlinear relationships quite well, with many kernels to choose from. Also, there is no distributional requirement for the data.\n",
        "\n",
        "In terms of disadvantages, SVM can be inefficient to train and memory-intensive to run and tune. It doesn’t perform well with large datasets. It requires the feature scaling of the data. There are also many hyperparameters, and their meanings are often not intuitive.\n",
        "\n"
      ],
      "metadata": {
        "id": "LSrdA8GoFzar"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# K-Nearest Neighbors\n",
        "K-nearest neighbors (KNN) is considered a “lazy learner,” as there is no learning required in the model. For a new data point, predictions are made by searching through the entire training set for the  K most similar instances (the neighbors) and summarizing the output variable for those  K instances.\n",
        "\n",
        "To determine which of the  K instances in the training dataset are most similar to a new input, a distance measure is used. The most popular distance measure is  Euclidean distance, which is calculated as the square root of the sum of the squared differences between a point  a and a point  b across all input attributes  i, and which is represented as d ( a , b ) = ∑ i=1 n (a i –b i ) 2 . Euclidean distance is a good distance measure to use if the input variables are similar in type.\n",
        "\n",
        "\n",
        "Another distance metric is  Manhattan distance, in which the distance between point  a and point  b is represented as d ( a , b ) = ∑ i=1 n | a i – b i | . Manhattan distance is a good measure to use if the input variables are not similar in type.\n",
        "\n",
        "\n",
        "The steps of KNN can be summarized as follows:\n",
        "\n",
        "\n",
        "Choose the number of  K and a distance metric.\n",
        "\n",
        "\n",
        "Find the  K-nearest neighbors of the sample that we want to classify.\n",
        "\n",
        "\n",
        "Assign the class label by majority vote.\n",
        "\n",
        "\n",
        "KNN regression and classification models can be constructed using the sklearn package of Python, as shown in the following code:\n",
        "\n",
        "**Classification**\n",
        "\n",
        "\n",
        "**from sklearn.neighbors import** KNeighborsClassifier\n",
        "\n",
        "\n",
        "model = KNeighborsClassifier()\n",
        "\n",
        "\n",
        "model.fit(X, Y)\n",
        "\n",
        "\n",
        "**Regression**\n",
        "\n",
        "\n",
        "**from sklearn.neighbors import** KNeighborsRegressor\n",
        "\n",
        "\n",
        "model = KNeighborsRegressor()\n",
        "\n",
        "\n",
        "model.fit(X, Y)\n",
        "\n",
        "\n",
        "**Hyperparameters**\n",
        "\n",
        "The following key parameters are present in the sklearn implementation of KNN and can be tweaked while performing the grid search:\n",
        "\n",
        "Number of neighbors (n_neighbors in sklearn)\n",
        "\n",
        "\n",
        "The most important hyperparameter for KNN is the number of neighbors (n_neighbors). Good values are between 1 and 20.\n",
        "\n",
        "\n",
        "**Distance metric (metric in sklearn)**\n",
        "\n",
        "\n",
        "It may also be interesting to test different distance metrics for choosing the composition of the neighborhood. Good values are  euclidean and  manhattan.\n",
        "\n",
        "\n",
        "#Advantages and disadvantages\n",
        "In terms of advantages, no training is involved and hence there is no learning phase. Since the algorithm requires no training before making predictions, new data can be added seamlessly without impacting the accuracy of the algorithm. It is intuitive and easy to understand. The model naturally handles multiclass classification and can learn complex decision boundaries. KNN is effective if the training data is large. It is also robust to noisy data, and there is no need to filter the outliers.\n",
        "\n",
        "In terms of the disadvantages, the distance metric to choose is not obvious and difficult to justify in many cases. KNN performs poorly on high dimensional datasets. It is expensive and slow to predict new instances because the distance to all neighbors must be recalculated. KNN is sensitive to noise in the dataset. We need to manually input missing values and remove outliers. Also, feature scaling (standardization and normalization) is required before applying the KNN algorithm to any dataset; otherwise, KNN may generate wrong predictions."
      ],
      "metadata": {
        "id": "5twHVnieG_Ai"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Linear Discriminant Analysis\n",
        "The objective of the  linear discriminant analysis (LDA) algorithm is to project the data onto a lower-dimensional space in a way that the class separability is maximized and the variance within a class is minimized. 4\n",
        "\n",
        "During the training of the LDA model, the statistical properties (i.e., mean and covariance matrix) of each class are computed. The statistical properties are estimated on the basis of the following assumptions about the data:\n",
        "\n",
        "\n",
        "Data is  [normally distributed](https://www.oreilly.com/library/view/machine-learning-and/9781492073048/ch04.html) , so that each variable is shaped like a bell curve when plotted.\n",
        "\n",
        "\n",
        "Each attribute has the same variance, and the values of each variable vary around the mean by the same amount on average.\n",
        "\n",
        "\n",
        "To make a prediction, LDA estimates the probability that a new set of inputs belongs to every class. The output class is the one that has the highest probability.\n",
        "\n",
        "\n",
        "Implementation in Python and hyperparameters\n",
        "\n",
        "\n",
        "The LDA classification model can be constructed using the sklearn package of Python, as shown in the following code snippet:\n",
        "\n",
        "\n",
        "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis\n",
        "\n",
        "\n",
        "model = LinearDiscriminantAnalysis()\n",
        "\n",
        "\n",
        "model.fit(X, Y)\n",
        "\n",
        "\n",
        "The key hyperparameter for the LDA model is number of components for dimensionality reduction, which is represented by n_components in sklearn.\n",
        "\n",
        "\n",
        "Advantages and disadvantages\n",
        "In terms of advantages, LDA is a relatively simple model with fast implementation and is easy to implement. In terms of disadvantages, it requires feature scaling and involves complex matrix operations."
      ],
      "metadata": {
        "id": "Veh7PCJmHeWA"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Classification and Regression Trees\n",
        "\n",
        "In the most general terms, the purpose of an analysis via tree-building algorithms is to determine a set of  if–then logical (split) conditions that permit accurate prediction or classification of cases.  Classification and regression trees (or  CART or  decision tree classifiers) are attractive models if we care about interpretability. We can think of this model as breaking down our data and making a decision based on asking a series of questions. This algorithm is the foundation of ensemble methods such as random forest and gradient boosting method.\n",
        "\n",
        "Representation\n",
        "The model can be represented by a  binary tree (or  decision tree), where each node is an input variable  x with a split point and each leaf contains an output variable  y for prediction.\n",
        "\n",
        "[Figure 4](https://www.oreilly.com/library/view/machine-learning-and/9781492073048/ch04.html) shows an example of a simple classification tree to predict whether a person is a male or a female based on two inputs of height (in centimeters) and weight (in kilograms).\n",
        "\n",
        "\n",
        "\n",
        "Image label\n",
        "\n",
        "**Figure 4. Classification and regression tree example**\n",
        "\n",
        "Learning a CART model\n",
        "Creating a binary tree is actually a process of dividing up the input space. A  greedy approach called  recursive binary splitting is used to divide the space. This is a numerical procedure in which all the values are lined up and different split points are tried and tested using a cost (loss) function. The split with the best cost (lowest cost, because we minimize cost) is selected. All input variables and all possible split points are evaluated and chosen in a greedy manner (e.g., the very best split point is chosen each time).\n",
        "\n",
        "For regression predictive modeling problems, the cost function that is minimized to choose split points is the  sum of squared errors across all training samples that fall within the rectangle:\n",
        "\n",
        "\n",
        "∑ i=1 n (y i –prediction i ) 2\n",
        "\n",
        "\n",
        "where y i is the output for the training sample and prediction is the predicted output for the rectangle. For classification, the  Gini cost function is used; it provides an indication of how pure the leaf nodes are (i.e., how mixed the training data assigned to each node is) and is defined as:\n",
        "\n",
        "\n",
        "G = ∑ i=1 n p k * ( 1 – p k )\n",
        "\n",
        "\n",
        "where  G is the Gini cost over all classes and p k is the number of training instances with class  k in the rectangle of interest. A node that has all classes of the same type (perfect class purity) will have  G = 0, while a node that has a  50–50 split of classes for a binary classification problem (worst purity) will have  G = 0.5.\n",
        "\n",
        "\n",
        "Stopping criterion\n",
        "\n",
        "\n",
        "The recursive binary splitting procedure described in the preceding section needs to know when to stop splitting as it works its way down the tree with the training data. The most common stopping procedure is to use a minimum count on the number of training instances assigned to each leaf node. If the count is less than some minimum, then the split is not accepted and the node is taken as a final leaf node.\n",
        "\n",
        "\n",
        "Pruning the tree\n",
        "\n",
        "\n",
        "The stopping criterion is important as it strongly influences the performance of the tree. Pruning can be used after learning the tree to further lift performance. The complexity of a decision tree is defined as the number of splits in the tree. Simpler trees are preferred as they are faster to run and easy to understand, consume less memory during processing and storage, and are less likely to overfit the data. The fastest and simplest pruning method is to work through each leaf node in the tree and evaluate the effect of removing it using a test set. A leaf node is removed only if doing so results in a drop in the overall cost function on the entire test set. The removal of nodes can be stopped when no further improvements can be made.\n",
        "\n",
        "\n",
        "Implementation in Python\n",
        "\n",
        "\n",
        "CART regression and classification models can be constructed using the sklearn package of Python, as shown in the following code snippet:\n",
        "\n",
        "\n",
        "Classification\n",
        "\n",
        "\n",
        "from sklearn.tree import DecisionTreeClassifier\n",
        "\n",
        "\n",
        "model = DecisionTreeClassifier()\n",
        "\n",
        "\n",
        "model.fit(X, Y)\n",
        "\n",
        "\n",
        "Regression\n",
        "\n",
        "\n",
        "from sklearn.tree import DecisionTreeRegressor\n",
        "\n",
        "\n",
        "model = DecisionTreeRegressor ()\n",
        "\n",
        "\n",
        "model.fit(X, Y)\n",
        "\n",
        "\n",
        "Hyperparameters\n",
        "CART has many hyperparameters. However, the key hyperparameter is the maximum depth of the tree model, which is the number of components for dimensionality reduction, and which is represented by max_depth in the sklearn package. Good values can range from  2 to  30 depending on the number of features in the data.\n",
        "\n",
        "Advantages and disadvantages\n",
        "In terms of advantages, CART is easy to interpret and can adapt to learn complex relationships. It requires little data preparation, and data typically does not need to be scaled. Feature importance is built in due to the way decision nodes are built. It performs well on large datasets. It works for both regression and classification problems.\n",
        "\n",
        "In terms of disadvantages, CART is prone to overfitting unless pruning is used. It can be very nonrobust, meaning that small changes in the training dataset can lead to quite major differences in the hypothesis function that gets learned. CART generally has worse performance than ensemble models, which are covered next."
      ],
      "metadata": {
        "id": "djug3UA9Hhqw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "The goal of  ensemble models is to combine different classifiers into a meta-classifier that has better generalization performance than each individual classifier alone. For example, assuming that we collected predictions from 10 experts, ensemble methods would allow us to strategically combine their predictions to come up with a prediction that is more accurate and robust than the experts’ individual predictions.\n",
        "\n",
        "The two most popular ensemble methods are bagging and boosting.  Bagging (or  bootstrap aggregation) is an ensemble technique of training several individual models in a parallel way. Each model is trained by a random subset of the data.  Boosting, on the other hand, is an ensemble technique of training several individual models in a sequential way. This is done by building a model from the training data and then creating a second model that attempts to correct the errors of the first model. Models are added until the training set is predicted perfectly or a maximum number of models is added. Each individual model learns from mistakes made by the previous model. Just like the decision trees themselves, bagging and boosting can be used for classification and regression problems.\n",
        "\n",
        "\n",
        "By combining individual models, the ensemble model tends to be more flexible (less bias) and less data-sensitive (less variance). 5  Ensemble methods combine multiple, simpler algorithms to obtain better performance.\n",
        "\n",
        "\n",
        "In this section we will cover random forest, AdaBoost, the gradient boosting method, and extra trees, along with their implementation using sklearn package.# Ensemble Models"
      ],
      "metadata": {
        "id": "gbUqJJxmHoHK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Random forest\n",
        "Random forest is a tweaked version of bagged decision trees. In order to understand a random forest algorithm, let us first understand the  bagging algorithm. Assuming we have a dataset of one thousand instances, the steps of bagging are:\n",
        "\n",
        "Create many (e.g., one hundred) random subsamples of our dataset.\n",
        "\n",
        "\n",
        "Train a CART model on each sample.\n",
        "\n",
        "\n",
        "Given a new dataset, calculate the average prediction from each model and aggregate the prediction by each tree to assign the final label by majority vote.\n",
        "\n",
        "\n",
        "A problem with decision trees like CART is that they are greedy. They choose the variable to split by using a greedy algorithm that minimizes error. Even after bagging, the decision trees can have a lot of structural similarities and result in high correlation in their predictions. Combining predictions from multiple models in ensembles works better if the predictions from the submodels are uncorrelated, or at best are weakly correlated. Random forest changes the learning algorithm in such a way that the resulting predictions from all of the subtrees have less correlation.\n",
        "\n",
        "\n",
        "In CART, when selecting a split point, the learning algorithm is allowed to look through all variables and all variable values in order to select the most optimal split point. The random forest algorithm changes this procedure such that each subtree can access only a random sample of features when selecting the split points. The number of features that can be searched at each split point ( m) must be specified as a parameter to the algorithm.\n",
        "\n",
        "\n",
        "As the bagged decision trees are constructed, we can calculate how much the error function drops for a variable at each split point. In regression problems, this may be the drop in sum squared error, and in classification, this might be the Gini cost. The bagged method can provide feature importance by calculating and averaging the error function drop for individual variables.\n",
        "\n",
        "\n",
        "Implementation in Python\n",
        "\n",
        "\n",
        "Random forest regression and classification models can be constructed using the sklearn package of Python, as shown in the following code:\n",
        "\n",
        "\n",
        "Classification\n",
        "\n",
        "\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "\n",
        "\n",
        "model = RandomForestClassifier()\n",
        "\n",
        "\n",
        "model.fit(X, Y)\n",
        "\n",
        "\n",
        "Regression\n",
        "\n",
        "\n",
        "from sklearn.ensemble import RandomForestRegressor\n",
        "\n",
        "\n",
        "model = RandomForestRegressor()\n",
        "\n",
        "\n",
        "model.fit(X, Y)\n",
        "\n",
        "\n",
        "Hyperparameters\n",
        "Some of the main hyperparameters that are present in the sklearn implementation of random forest and that can be tweaked while performing the grid search are:\n",
        "\n",
        "Maximum number of features (max_features in sklearn)\n",
        "\n",
        "\n",
        "This is the most important parameter. It is the number of random features to sample at each split point. You could try a range of integer values, such as 1 to 20, or 1 to half the number of input features.\n",
        "\n",
        "\n",
        "Number of estimators (n_estimators in sklearn)\n",
        "\n",
        "\n",
        "This parameter represents the number of trees. Ideally, this should be increased until no further improvement is seen in the model. Good values might be a log scale from 10 to 1,000.\n",
        "\n",
        "\n",
        "Advantages and disadvantages\n",
        "The random forest algorithm (or model) has gained huge popularity in ML applications during the last decade due to its good performance, scalability, and ease of use. It is flexible and naturally assigns feature importance scores, so it can handle redundant feature columns. It scales to large datasets and is generally robust to overfitting. The algorithm doesn’t need the data to be scaled and can model a nonlinear relationship.\n",
        "\n",
        "In terms of disadvantages, random forest can feel like a black box approach, as we have very little control over what the model does, and the results may be difficult to interpret. Although random forest does a good job at classification, it may not be good for regression problems, as it does not give a precise continuous nature prediction. In the case of regression, it doesn’t predict beyond the range in the training data and may overfit datasets that are particularly noisy.\n",
        "\n"
      ],
      "metadata": {
        "id": "IP2PrVl2HskR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Extra trees\n",
        "Extra trees, otherwise known as  extremely randomized trees, is a variant of a random forest; it builds multiple trees and splits nodes using random subsets of features similar to random forest. However, unlike random forest, where observations are drawn with replacement, the observations are drawn without replacement in extra trees. So there is no repetition of observations.\n",
        "\n",
        "Additionally, random forest selects the best split to convert the parent into the two most homogeneous child nodes. 6  However, extra trees selects a random split to divide the parent node into two random child nodes. In extra trees, randomness doesn’t come from bootstrapping the data; it comes from the random splits of all observations.\n",
        "\n",
        "\n",
        "In real-world cases, performance is comparable to an ordinary random forest, sometimes a bit better. The advantages and disadvantages of extra trees are similar to those of random forest.\n",
        "\n",
        "\n",
        "Implementation in Python\n",
        "\n",
        "\n",
        "Extra trees regression and classification models can be constructed using the sklearn package of Python, as shown in the following code snippet. The hyperparameters of extra trees are similar to random forest, as shown in the previous section:\n",
        "\n",
        "\n",
        "Classification\n",
        "\n",
        "\n",
        "from sklearn.ensemble import ExtraTreesClassifier\n",
        "\n",
        "\n",
        "model = ExtraTreesClassifier()\n",
        "\n",
        "\n",
        "model.fit(X, Y)\n",
        "\n",
        "\n",
        "Regression\n",
        "\n",
        "\n",
        "from sklearn.ensemble import ExtraTreesRegressor\n",
        "\n",
        "\n",
        "model = ExtraTreesRegressor()\n",
        "\n",
        "\n",
        "model.fit(X, Y)\n",
        "\n"
      ],
      "metadata": {
        "id": "wQyiDcjSHxBg"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Adaptive Boosting (AdaBoost)\n",
        "Adaptive Boosting or  AdaBoost is a boosting technique in which the basic idea is to try predictors sequentially, and each subsequent model attempts to fix the errors of its predecessor. At each iteration, the AdaBoost algorithm changes the sample distribution by modifying the weights attached to each of the instances. It increases the weights of the wrongly predicted instances and decreases the ones of the correctly predicted instances.\n",
        "\n",
        "The steps of the AdaBoost algorithm are:\n",
        "\n",
        "\n",
        "Initially, all observations are given equal weights.\n",
        "\n",
        "\n",
        "A model is built on a subset of data, and using this model, predictions are made on the whole dataset. Errors are calculated by comparing the predictions and actual values.\n",
        "\n",
        "\n",
        "While creating the next model, higher weights are given to the data points that were predicted incorrectly. Weights can be determined using the error value. For instance, the higher the error, the more weight is assigned to the observation.\n",
        "\n",
        "\n",
        "This process is repeated until the error function does not change, or until the maximum limit of the number of estimators is reached.\n",
        "\n",
        "\n",
        "Implementation in Python\n",
        "\n",
        "\n",
        "AdaBoost regression and classification models can be constructed using the sklearn package of Python, as shown in the following code snippet:\n",
        "\n",
        "\n",
        "Classification\n",
        "\n",
        "\n",
        "from sklearn.ensemble import AdaBoostClassifier\n",
        "\n",
        "\n",
        "model = AdaBoostClassifier()\n",
        "\n",
        "\n",
        "model.fit(X, Y)\n",
        "\n",
        "\n",
        "Regression\n",
        "\n",
        "\n",
        "from sklearn.ensemble import AdaBoostRegressor\n",
        "\n",
        "\n",
        "model = AdaBoostRegressor()\n",
        "\n",
        "\n",
        "model.fit(X, Y)\n",
        "\n",
        "\n",
        "Hyperparameters\n",
        "Some of the main hyperparameters that are present in the sklearn implementation of AdaBoost and that can be tweaked while performing the grid search are as follows:\n",
        "\n",
        "Learning rate (learning_rate in sklearn)\n",
        "\n",
        "\n",
        "Learning rate shrinks the contribution of each classifier/regressor. It can be considered on a log scale. The sample values for grid search can be 0.001, 0.01, and 0.1.\n",
        "\n",
        "\n",
        "Number of estimators (n_estimators in sklearn)\n",
        "\n",
        "\n",
        "This parameter represents the number of trees. Ideally, this should be increased until no further improvement is seen in the model. Good values might be a log scale from 10 to 1,000.\n",
        "\n",
        "\n",
        "Advantages and disadvantages\n",
        "In terms of advantages, AdaBoost has a high degree of precision. AdaBoost can achieve similar results to other models with much less tweaking of parameters or settings. The algorithm doesn’t need the data to be scaled and can model a nonlinear relationship.\n",
        "\n",
        "In terms of disadvantages, the training of AdaBoost is time consuming. AdaBoost can be sensitive to noisy data and outliers, and data imbalance leads to a decrease in classification accuracy\n",
        "\n"
      ],
      "metadata": {
        "id": "qRotnSjaH1MI"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Gradient boosting method\n",
        "Gradient boosting method (GBM) is another boosting technique similar to AdaBoost, where the general idea is to try predictors sequentially. Gradient boosting works by sequentially adding the previous underfitted predictions to the ensemble, ensuring the errors made previously are corrected.\n",
        "\n",
        "The following are the steps of the gradient boosting algorithm:\n",
        "\n",
        "\n",
        "A model (which can be referred to as the first weak learner) is built on a subset of data. Using this model, predictions are made on the whole dataset.\n",
        "\n",
        "\n",
        "Errors are calculated by comparing the predictions and actual values, and the loss is calculated using the loss function.\n",
        "\n",
        "\n",
        "A new model is created using the errors of the previous step as the target variable. The objective is to find the best split in the data to minimize the error. The predictions made by this new model are combined with the predictions of the previous. New errors are calculated using this predicted value and actual value.\n",
        "\n",
        "\n",
        "This process is repeated until the error function does not change or until the maximum limit of the number of estimators is reached.\n",
        "\n",
        "\n",
        "Contrary to AdaBoost, which tweaks the instance weights at every interaction, this method tries to fit the new predictor to the residual errors made by the previous predictor.\n",
        "\n",
        "\n",
        "Implementation in Python and hyperparameters\n",
        "\n",
        "\n",
        "Gradient boosting method regression and classification models can be constructed using the sklearn package of Python, as shown in the following code snippet. The hyperparameters of gradient boosting method are similar to AdaBoost, as shown in the previous section:\n",
        "\n",
        "\n",
        "Classification\n",
        "\n",
        "\n",
        "from sklearn.ensemble import GradientBoostingClassifier\n",
        "\n",
        "\n",
        "model = GradientBoostingClassifier()\n",
        "\n",
        "\n",
        "model.fit(X, Y)\n",
        "\n",
        "\n",
        "Regression\n",
        "\n",
        "\n",
        "from sklearn.ensemble import GradientBoostingRegressor\n",
        "\n",
        "\n",
        "model = GradientBoostingRegressor()\n",
        "\n",
        "\n",
        "model.fit(X, Y)\n",
        "\n",
        "\n",
        "Advantages and disadvantages\n",
        "In terms of advantages, gradient boosting method is robust to missing data, highly correlated features, and irrelevant features in the same way as random forest. It naturally assigns feature importance scores, with slightly better performance than random forest. The algorithm doesn’t need the data to be scaled and can model a nonlinear relationship.\n",
        "\n",
        "In terms of disadvantages, it may be more prone to overfitting than random forest, as the main purpose of the boosting approach is to reduce bias and not variance. It has many hyperparameters to tune, so model development may not be as fast. Also, feature importance may not be robust to variation in the training dataset.\n",
        "\n"
      ],
      "metadata": {
        "id": "osi5ziNeH5OZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#ANN-Based Models\n",
        "The details provided in that chapter are applicable across all areas of machine learning, including supervised learning. However, there are a few additional details from the supervised learning perspective, which we will cover in this section.\n",
        "\n",
        "Neural networks are reducible to a classification or regression model with the activation function of the node in the output layer. In the case of a regression problem, the output node has linear activation function (or no activation function). A linear function produces a continuous output ranging from -inf to +inf. Hence, the output layer will be the linear function of the nodes in the layer before the output layer, and it will be a regression-based model.\n",
        "\n",
        "\n",
        "In the case of a classification problem, the output node has a sigmoid or softmax activation function. A sigmoid or softmax function produces an output ranging from zero to one to represent the probability of target value. Softmax function can also be used for multiple groups for classification.\n",
        "\n",
        "\n",
        "ANN using sklearn\n",
        "\n",
        "\n",
        "ANN regression and classification models can be constructed using the sklearn package of Python, as shown in the following code snippet:\n",
        "\n",
        "\n",
        "Classification\n",
        "\n",
        "\n",
        "from sklearn.neural_network import MLPClassifier\n",
        "\n",
        "\n",
        "model = MLPClassifier()\n",
        "\n",
        "\n",
        "model.fit(X, Y)\n",
        "\n",
        "\n",
        "Regression\n",
        "\n",
        "\n",
        "from sklearn.neural_network import MLPRegressor\n",
        "\n",
        "\n",
        "model = MLPRegressor()\n",
        "\n",
        "\n",
        "model.fit(X, Y)\n",
        "\n",
        "\n",
        "Hyperparameters\n",
        "ANN has many hyperparameters. Some of the hyperparameters that are present in the sklearn implementation of ANN and can be tweaked while performing the grid search are:\n",
        "\n",
        "Hidden Layers (hidden_layer_sizes in sklearn)\n",
        "\n",
        "\n",
        "It represents the number of layers and nodes in the ANN architecture. In sklearn implementation of ANN, the ith element represents the number of neurons in the ith hidden layer. A sample value for grid search in the sklearn implementation can be [( 20,), ( 50,), ( 20,  20), ( 20,  30,  20)].\n",
        "\n",
        "\n",
        "Activation Function (activation in sklearn)\n",
        "\n",
        "\n",
        "It represents the activation function of a hidden layer. Some of the activation are such as sigmoid, relu, or tanh, can be used.\n",
        "\n"
      ],
      "metadata": {
        "id": "y0TndcHPH9rK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Deep neural network\n",
        "ANNs with more than a single hidden layer are often called deep networks. We prefer using the library Keras to implement such networks, given the flexibility of the library. The detailed implementation of a deep neural network in Keras. Similar to MLPClassifier and MLPRegressor in sklearn for classification and regression, Keras has modules called KerasClassifier and KerasRegressor that can be used for creating classification and regression models with deep network.\n",
        "\n",
        "A popular problem in finance is time series prediction, which is predicting the next value of a time series based on a historical overview. Some of the deep neural networks, such as recurrent neural network (RNN), can be directly used for time series prediction.\n",
        "\n",
        "\n",
        "Advantages and disadvantages\n",
        "The main advantage of an ANN is that it captures the nonlinear relationship between the variables quite well. ANN can more easily learn rich representations and is good with a large number of input features with a large dataset. ANN is flexible in how it can be used. This is evident from its use across a wide variety of areas in machine learning and AI, including reinforcement learning and NLP.\n",
        "\n",
        "The main disadvantage of ANN is the interpretability of the model, which is a drawback that often cannot be ignored and is sometimes the determining factor when choosing a model. ANN is not good with small datasets and requires a lot of tweaking and guesswork. Choosing the right topology/algorithms to solve a problem is difficult. Also, ANN is computationally expensive and can take a lot of time to train."
      ],
      "metadata": {
        "id": "BYUoVtFgICNK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Model Performance\n",
        "In the previous section, we discussed grid search as a way to find the right hyperparameter to achieve better performance. In this section, we will expand on that process by discussing the key components of evaluating the model performance, which are overfitting, cross validation, and evaluation metrics.\n",
        "\n",
        "Overfitting and Underfitting\n",
        "A common problem in machine learning is  overfitting, which is defined by learning a function that perfectly explains the training data that the model learned from but doesn’t generalize well to unseen test data. Overfitting happens when a model overlearns from the training data to the point that it starts picking up idiosyncrasies that aren’t representative of patterns in the real world. This becomes especially problematic as we make our models increasingly more complex.  Underfitting is a related issue in which the model is not complex enough to capture the underlying trend in the data.  [Figure 5](https://www.oreilly.com/library/view/machine-learning-and/9781492073048/ch04.html)  illustrates overfitting and underfitting. The left-hand panel of  [Figure 5](https://www.oreilly.com/library/view/machine-learning-and/9781492073048/ch04.html)  shows a linear regression model; a straight line clearly underfits the true function. The middle panel shows that a high degree polynomial approximates the true relationship reasonably well. On the other hand, a polynomial of a very high degree fits the small sample almost perfectly, and performs best on the training data, but this doesn’t generalize, and it would do a horrible job at explaining a new data point.\n",
        "\n",
        "The concepts of overfitting and underfitting are closely linked to  bias-variance trade-off.  Bias refers to the error due to overly simplistic assumptions or faulty assumptions in the learning algorithm. Bias results in underfitting of the data, as shown in the left-hand panel of  [Figure 5](https://www.oreilly.com/library/view/machine-learning-and/9781492073048/ch04.html) . A high bias means our learning algorithm is missing important trends among the features.  Variance refers to the error due to an overly complex model that tries to fit the training data as closely as possible. In high variance cases, the model’s predicted values are extremely close to the actual values from the training set. High variance gives rise to overfitting, as shown in the right-hand panel of  [Figure 5](https://www.oreilly.com/library/view/machine-learning-and/9781492073048/ch04.html) . Ultimately, in order to have a good model, we need low bias and low variance.\n",
        "\n",
        "\n",
        "\n",
        "Image label\n",
        "\n",
        "**Figure 5. Overfitting and underfitting**\n",
        "\n",
        "There can be two ways to combat overfitting:\n",
        "\n",
        "\n",
        "Using more training data\n",
        "\n",
        "\n",
        "The more training data we have, the harder it is to overfit the data by learning too much from any single training example.\n",
        "\n",
        "\n",
        "Using regularization\n",
        "\n",
        "\n",
        "Adding a penalty in the loss function for building a model that assigns too much explanatory power to any one feature, or allows too many features to be taken into account.\n",
        "\n",
        "\n",
        "The concept of overfitting and the ways to combat it are applicable across all the supervised learning models. For example, regularized regressions address overfitting in linear regression, as discussed earlier in this chapter.\n",
        "\n",
        "\n",
        "#Cross Validation\n",
        "One of the challenges of machine learning is training models that are able to generalize well to unseen data (overfitting versus underfitting or a bias-variance trade-off). The main idea behind  cross validation is to split the data one time or several times so that each split is used once as a validation set and the remainder is used as a training set: part of the data (the training sample) is used to train the algorithm, and the remaining part (the validation sample) is used for estimating the risk of the algorithm. Cross validation allows us to obtain reliable estimates of the model’s generalization error. It is easiest to understand it with an example. When doing  k-fold cross validation, we randomly split the training data into  k folds. Then we train the model using  k-1 folds and evaluate the performance on the  kth fold. We repeat this process  k times and average the resulting scores.\n",
        "\n",
        "[Figure 6](https://www.oreilly.com/library/view/machine-learning-and/9781492073048/ch04.html)  shows an example of cross validation, where the data is split into five sets and in each round one of the sets is used for validation.\n",
        "\n",
        "\n",
        "\n",
        "Image label\n",
        "\n",
        "**Figure 6. Cross validation**\n",
        "\n",
        "A potential drawback of cross validation is the computational cost, especially when paired with a grid search for hyperparameter tuning. Cross validation can be performed in a couple of lines using the sklearn package; we will perform cross validation in the supervised learning case studies.\n",
        "\n",
        "\n",
        "In the next section, we cover the evaluation metrics for the supervised learning models that are used to measure and compare the models’ performance.\n",
        "\n",
        "\n",
        "#Evaluation Metrics\n",
        "The metrics used to evaluate the machine learning algorithms are very important. The choice of metrics to use influences how the performance of machine learning algorithms is measured and compared. The metrics influence both how you weight the importance of different characteristics in the results and your ultimate choice of algorithm.\n",
        "\n",
        "The main evaluation metrics for regression and classification are illustrated in  [Figure 7](https://www.oreilly.com/library/view/machine-learning-and/9781492073048/ch04.html) .\n",
        "\n",
        "\n",
        "\n",
        "Image label\n",
        "\n",
        "**Figure 7. Evaluation metrics for regression and classification**\n",
        "\n",
        "Let us first look at the evaluation metrics for supervised regression.\n",
        "\n",
        "\n",
        "Mean absolute error\n",
        "\n",
        "\n",
        "The  mean absolute error (MAE) is the sum of the absolute differences between predictions and actual values. The MAE is a linear score, which means that all the individual differences are weighted equally in the average. It gives an idea of how wrong the predictions were. The measure gives an idea of the magnitude of the error, but no idea of the direction (e.g., over- or underpredicting).\n",
        "\n",
        "\n",
        "Mean squared error\n",
        "\n",
        "\n",
        "The  mean squared error (MSE) represents the sample standard deviation of the differences between predicted values and observed values (called residuals). This is much like the mean absolute error in that it provides a gross idea of the magnitude of the error. Taking the square root of the mean squared error converts the units back to the original units of the output variable and can be meaningful for description and presentation. This is called the  root mean squared error (RMSE).\n",
        "\n",
        "\n",
        "R² metric\n",
        "\n",
        "\n",
        "The  R² metric provides an indication of the “goodness of fit” of the predictions to actual value. In statistical literature this measure is called the coefficient of determination. This is a value between zero and one, for no-fit and perfect fit, respectively.\n",
        "\n",
        "\n",
        "Adjusted R² metric\n",
        "\n",
        "\n",
        "Just like  R²,  adjusted R² also shows how well terms fit a curve or line but adjusts for the number of terms in a model. It is given in the following formula:\n",
        "\n",
        "\n",
        "R adj 2 = 1 – (1–R 2 )(n–1)) n–k–1\n",
        "\n",
        "\n",
        "where  n is the total number of observations and  k is the number of predictors. Adjusted  R² will always be less than or equal to  R².\n",
        "\n",
        "\n",
        "Selecting an evaluation metric for supervised regression\n",
        "\n",
        "\n",
        "In terms of a preference among these evaluation metrics, if the main goal is predictive accuracy, then RMSE is best. It is computationally simple and is easily differentiable. The loss is symmetric, but larger errors weigh more in the calculation. The MAEs are symmetric but do not weigh larger errors more.  R² and adjusted  R² are often used for explanatory purposes by indicating how well the selected independent variable(s) explains the variability in the dependent variable(s).\n",
        "\n",
        "\n",
        "Let us first look at the evaluation metrics for supervised classification.\n",
        "\n",
        "\n",
        "Classification\n",
        "\n",
        "\n",
        "For simplicity, we will mostly discuss things in terms of a binary classification problem (i.e., only two outcomes, such as true or false); some common terms are:\n",
        "\n",
        "\n",
        "True positives (TP)\n",
        "\n",
        "\n",
        "Predicted positive and are actually positive.\n",
        "\n",
        "\n",
        "False positives (FP)\n",
        "\n",
        "\n",
        "Predicted positive and are actually negative.\n",
        "\n",
        "\n",
        "True negatives (TN)\n",
        "\n",
        "\n",
        "Predicted negative and are actually negative.\n",
        "\n",
        "\n",
        "False negatives (FN)\n",
        "\n",
        "\n",
        "Predicted negative and are actually positive.\n",
        "\n",
        "\n",
        "The difference between three commonly used evaluation metrics for classification, accuracy, precision, and recall, is illustrated in  Figure 8 .\n",
        "\n",
        "\n",
        "\n",
        "Image label\n",
        "\n",
        "**Figure 8. Accuracy, precision, and recall**\n",
        "\n",
        "Accuracy\n",
        "\n",
        "\n",
        "As shown in  [Figure 8](https://www.oreilly.com/library/view/machine-learning-and/9781492073048/ch04.html) ,  accuracy is the number of correct predictions made as a ratio of all predictions made. This is the most common evaluation metric for classification problems and is also the most misused. It is most suitable when there are an equal number of observations in each class (which is rarely the case) and when all predictions and the related prediction errors are equally important, which is often not the case.\n",
        "\n",
        "\n",
        "Precision\n",
        "\n",
        "\n",
        "Precision is the percentage of positive instances out of the total predicted positive instances. Here, the denominator is the model prediction done as positive from the whole given dataset. Precision is a good measure to determine when the cost of false positives is high (e.g., email spam detection).\n",
        "\n",
        "\n",
        "Recall\n",
        "\n",
        "\n",
        "Recall (or  sensitivity or  true positive rate) is the percentage of positive instances out of the total actual positive instances. Therefore, the denominator (true positive + false negative) is the actual number of positive instances present in the dataset. Recall is a good measure when there is a high cost associated with false negatives (e.g., fraud detection).\n",
        "\n",
        "\n",
        "In addition to accuracy, precision, and recall, some of the other commonly used evaluation metrics for classification are discussed in the following sections.\n",
        "\n",
        "\n",
        "Area under ROC curve\n",
        "\n",
        "\n",
        "Area under ROC curve (AUC) is an evaluation metric for binary classification problems. ROC is a probability curve, and AUC represents degree or measure of separability. It tells how much the model is capable of distinguishing between classes. The higher the AUC, the better the model is at predicting zeros as zeros and ones as ones. An AUC of  0.5 means that the model has no class separation capacity whatsoever. The probabilistic interpretation of the AUC score is that if you randomly choose a positive case and a negative case, the probability that the positive case outranks the negative case according to the classifier is given by the AUC.\n",
        "\n",
        "\n",
        "**Confusion matrix**\n",
        "\n",
        "\n",
        "A confusion matrix lays out the performance of a learning algorithm. The confusion matrix is simply a square matrix that reports the counts of the true positive (TP), true negative (TN), false positive (FP), and false negative (FN) predictions of a classifier, as shown in  [Figure 9](https://www.oreilly.com/library/view/machine-learning-and/9781492073048/ch04.html) .\n",
        "\n",
        "\n",
        "\n",
        "Image label\n",
        "Figure 9. Confusion matrix\n",
        "\n",
        "The confusion matrix is a handy presentation of the accuracy of a model with two or more classes. The table presents predictions on the  x-axis and accuracy outcomes on the  y-axis. The cells of the table are the number of predictions made by the model. For example, a model can predict zero or one, and each prediction may actually have been a zero or a one. Predictions for zero that were actually zero appear in the cell for prediction = 0 and actual = 0, whereas predictions for zero that were actually one appear in the cell for prediction = 0 and actual = 1.\n",
        "\n",
        "\n",
        "Selecting an evaluation metric for supervised classification\n",
        "\n",
        "\n",
        "The evaluation metric for classification depends heavily on the task at hand. For example, recall is a good measure when there is a high cost associated with false negatives such as fraud detection. We will further examine these evaluation metrics in the case studies.\n",
        "\n"
      ],
      "metadata": {
        "id": "DamKGsB8IGg8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Model Selection\n",
        "Selecting the perfect machine learning model is both an art and a science. Looking at machine learning models, there is no one solution or approach that fits all. There are several factors that can affect your choice of a machine learning model. The main criteria in most of the cases is the model performance that we discussed in the previous section. However, there are many other factors to consider while performing model selection. In the following section, we will go over all such factors, followed by a discussion of model trade-offs.\n",
        "\n",
        "Factors for Model Selection\n",
        "\n",
        "\n",
        "The factors considered for the model selection process are as follows:\n",
        "\n",
        "\n",
        "Simplicity\n",
        "\n",
        "\n",
        "The degree of simplicity of the model. Simplicity usually results in quicker, more scalable, and easier to understand models and results.\n",
        "\n",
        "\n",
        "Training time\n",
        "\n",
        "\n",
        "Speed, performance, memory usage and overall time taken for model training.\n",
        "\n",
        "\n",
        "Handle nonlinearity in the data\n",
        "\n",
        "\n",
        "The ability of the model to handle the nonlinear relationship between the variables.\n",
        "\n",
        "\n",
        "Robustness to overfitting\n",
        "\n",
        "\n",
        "The ability of the model to handle overfitting.\n",
        "\n",
        "\n",
        "Size of the dataset\n",
        "\n",
        "\n",
        "The ability of the model to handle large number of training examples in the dataset.\n",
        "\n",
        "\n",
        "Number of features\n",
        "\n",
        "\n",
        "The ability of the model to handle high dimensionality of the feature space.\n",
        "\n",
        "\n",
        "Model interpretation\n",
        "\n",
        "\n",
        "How explainable is the model? Model interpretability is important because it allows us to take concrete actions to solve the underlying problem.\n",
        "\n",
        "\n",
        "Feature scaling\n",
        "\n",
        "\n",
        "Does the model require variables to be scaled or normally distributed?\n",
        "\n",
        "\n",
        "[Figure 10](https://www.oreilly.com/library/view/machine-learning-and/9781492073048/ch04.html)  compares the supervised learning models on the factors mentioned previously and outlines a general rule-of-thumb to narrow down the search for the best machine learning algorithm 7  for a given problem. The table is based on the advantages and disadvantages of different models discussed in the individual model section in this chapter.\n",
        "\n",
        "\n",
        "\n",
        "Image label\n",
        "\n",
        "**Figure 10. Model selection**\n",
        "\n",
        "We can see from the table that relatively simple models include linear and logistic regression and as we move towards the ensemble and ANN, the complexity increases. In terms of the training time, the linear models and CART are relatively faster to train as compared to ensemble methods and ANN.\n",
        "\n",
        "\n",
        "Linear and logistic regression can’t handle nonlinear relationships, while all other models can. SVM can handle the nonlinear relationship between dependent and independent variables with nonlinear kernels.\n",
        "\n",
        "\n",
        "SVM and random forest tend to overfit less as compared to the linear regression, logistic regression, gradient boosting, and ANN. The degree of overfitting also depends on other parameters, such as size of the data and model tuning, and can be checked by looking at the results of the test set for each model. Also, the boosting methods such as gradient boosting have higher overfitting risk compared to the bagging methods, such as random forest. Recall the focus of gradient boosting is to minimize the bias and not variance.\n",
        "\n",
        "\n",
        "Linear and logistic regressions are not able to handle large datasets and large number of features well. However, CART, ensemble methods, and ANN are capable of handling large datasets and many features quite well. The linear and logistic regression generally perform better than other models in case the size of the dataset is small. Application of variable reduction techniques  enables the linear models to handle large datasets. The performance of ANN increases with an increase in the size of the dataset.\n",
        "\n",
        "\n",
        "Given linear regression, logistic regression, and CART are relatively simpler models, they have better model interpretation as compared to the ensemble models and ANN.\n",
        "\n",
        "\n",
        "Model Trade-off\n",
        "\n",
        "\n",
        "Often, it’s a trade-off between different factors when selecting a model. ANN, SVM, and some ensemble methods can be used to create very accurate predictive models, but they may lack simplicity and interpretability and may take a significant amount of resources to train.\n",
        "\n",
        "\n",
        "In terms of selecting the final model, models with lower interpretability may be preferred when predictive performance is the most important goal, and it’s not necessary to explain how the model works and makes predictions. In some cases, however, model interpretability is mandatory.\n",
        "\n",
        "\n",
        "Interpretability-driven examples are often seen in the financial industry. In many cases, choosing a machine learning algorithm has less to do with the optimization or the technical aspects of the algorithm and more to do with business decisions. Suppose a machine learning algorithm is used to accept or reject an individual’s credit card application. If the applicant is rejected and decides to file a complaint or take legal action, the financial institution will need to explain how that decision was made. While that can be nearly impossible for ANN, it’s relatively straightforward for decision tree–based models.\n",
        "\n",
        "\n",
        "Different classes of models are good at modeling different types of underlying patterns in data. So a good first step is to quickly test out a few different classes of models to know which ones capture the underlying structure of the dataset most efficiently. We will follow this approach while performing model selection in all our supervised learning–based case studies."
      ],
      "metadata": {
        "id": "QaKr69Q9INxZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Chapter Summary\n",
        "In this chapter, we discussed the importance of supervised learning models in finance, followed by a brief introduction to several supervised learning models, including linear and logistic regression, SVM, decision trees, ensemble, KNN, LDA, and ANN. We demonstrated training and tuning of these models in a few lines of code using sklearn and Keras libraries.\n",
        "\n",
        "We discussed the most common error metrics for regression and classification models, explained the bias-variance trade-off, and illustrated the various tools for managing the model selection process using cross validation.\n",
        "\n",
        "\n",
        "We introduced the strengths and weaknesses of each model and discussed the factors to consider when selecting the best model. We also discussed the trade-off between model performance and interpretability.\n",
        "\n",
        "\n",
        "Page 2 of 2\n",
        "\n"
      ],
      "metadata": {
        "id": "kSpSJgXMIUCA"
      }
    }
  ]
}