Testing_concordance_indexes.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Testing_concordance_indexes.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/alonsosilvaallende/Survival_Analysis/blob/master/Testing_concordance_indexes.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yujaTHwquzdV",
        "colab_type": "text"
      },
      "source": [
        "# Testing concordance indexes of lifelines, scikit-survival, and pysurvival\n",
        "[Alonso Silva](http://www.dim.uchile.cl/~alsilva/) [(@alonsosilva)](https://twitter.com/alonsosilva) | 2019/09/30"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Tg64n0Jjw-3Z",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# install scikit-survival, lifelines, and pysurvival\n",
        "pip install -q scikit-survival lifelines pysurvival"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EIejd1FnxF0T",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# imports\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "from matplotlib import pyplot as plt\n",
        "\n",
        "%matplotlib inline"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X2usPhPIuAkz",
        "colab_type": "text"
      },
      "source": [
        "The objective of this notebook is to compare the concordance indexes of three different libraries on survival analysis."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ohcR8VX0dEgZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from lifelines.utils import concordance_index as ci_lifelines\n",
        "from pysurvival.utils.metrics import concordance_index as ci_pysurvival\n",
        "from sksurv.metrics import concordance_index_censored as ci_scikit"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yj0u5J8uyOLZ",
        "colab_type": "text"
      },
      "source": [
        "Lifelines concordance index inputs are:\n",
        "\n",
        "+ event/censoring times\n",
        "+ predicted scores\n",
        "+ event observed (*optional*). The default assumes all events observed.\n",
        "\n",
        "Lifelines concordance index output is:\n",
        "\n",
        "+ concordance index"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rzzUqeesdIGB",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "9ed83043-adfc-472a-c326-c047cf77e127"
      },
      "source": [
        "events = [0, 1, 2, 3, 4]\n",
        "preds_scores = [0, 1, 2, 3, 4]\n",
        "event_obs = [True, True, True, True, True]\n",
        "\n",
        "print(f'Concordance-index (lifelines): {ci_lifelines(events, preds_scores, event_obs)}')"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Concordance-index (lifelines): 1.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V7xuP687dV2J",
        "colab_type": "text"
      },
      "source": [
        "Scikit-survival's concordance index inputs are:\n",
        "+ event observed\n",
        "+ event/censoring times\n",
        "+ predicted risks\n",
        "+ tied tolerance (float, optional, default: 1e-8) – The tolerance value for considering ties. If the absolute difference between risk scores is smaller or equal than tied_tol, risk scores are considered tied. \n",
        "\n",
        "Scikit-survival's concordance index outputs are:\n",
        "\n",
        "+ Concordance index\n",
        "+ Number of concordant pairs\n",
        "+ Number of discordant pairs\n",
        "+ Number of pairs having tied estimated risks\n",
        "+ Number of comparable pairs sharing the same time"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DEFgM_FKxIzk",
        "colab_type": "code",
        "outputId": "34aa8dfc-81f6-43a3-c747-12425f81599c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "events = [0, 1, 2, 3, 4]\n",
        "preds_risks = [4, 3, 2, 1, 0]\n",
        "event_obs = [True, True, True, True, True]\n",
        "\n",
        "print(f'Concordance-index (scikit-survival): {ci_scikit(event_obs, events, preds_risks)[0]}')"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Concordance-index (scikit-survival): 1.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EyT74Q4i3Kpy",
        "colab_type": "text"
      },
      "source": [
        "As mentioned above, scikit survival also gives the number of concordant pairs, number of discordant pairs, number of pairs having tied estimated risks, and the number of comparable pairs sharing the same time."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zxxz9nUd2_ok",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "outputId": "0a19a962-0b4e-4b75-e5e8-b07e78eaa6a5"
      },
      "source": [
        "print(f'Number of concordant pairs (scikit-survival): {ci_scikit(event_obs, events, preds_risks)[1]}')\n",
        "print(f'Number of discordant pairs (scikit-survival): {ci_scikit(event_obs, events, preds_risks)[2]}')\n",
        "print(f'Number of pairs having tied estimated risks (scikit-survival): {ci_scikit(event_obs, events, preds_risks)[3]}')\n",
        "print(f'Number of comparable pairs sharing the same time (scikit-survival): {ci_scikit(event_obs, events, preds_risks)[4]}')"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of concordant pairs (scikit-survival): 10\n",
            "Number of discordant pairs (scikit-survival): 0\n",
            "Number of pairs having tied estimated risks (scikit-survival): 0\n",
            "Number of comparable pairs sharing the same time (scikit-survival): 0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NMY28CGB15IL",
        "colab_type": "text"
      },
      "source": [
        "Lifelines gives the concordance between the actuals and the predicted *scores*, while scikit-survival gives the concordance between the actuals and the predicted *risks*, so over the same lists, they are the complete opposite.\n",
        "\n",
        "$$\n",
        "\\textrm{Concordance-index (lifelines)}= 1-\\textrm{Concordance-index (scikit-survival)}\n",
        "$$\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X0GnTAwLxXOL",
        "colab_type": "code",
        "outputId": "5f2b335e-cb53-4530-99f8-2f7c97bf6cd9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "actuals = [0, 1, 2, 3, 4]\n",
        "preds = [4, 3, 2, 1, 0]\n",
        "event_obs = [True, True, True, True, True]\n",
        "\n",
        "print(f'Concordance-index (lifelines): {ci_lifelines(actuals, preds, event_obs)}')\n",
        "print(f'Concordance-index (scikit-survival): {ci_scikit(event_obs, actuals, preds)[0]}')"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Concordance-index (lifelines): 0.0\n",
            "Concordance-index (scikit-survival): 1.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eokykj3s2V3T",
        "colab_type": "text"
      },
      "source": [
        "Let's see a more interesting example:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XDm3EGEX9Z-e",
        "colab_type": "code",
        "outputId": "ca8b8fe6-5656-4782-a86a-16962b1d3ac4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 272
        }
      },
      "source": [
        "actuals = [0, 1, 2, 3, 4]\n",
        "preds = [4, 3, 1, 2, 0]\n",
        "event_obs = [True, True, True, True, True]\n",
        "\n",
        "print(f'Concordance-index (lifelines): {ci_lifelines(actuals, preds, event_obs):.3f}')\n",
        "print(f'Concordance-index (scikit-survival): {ci_scikit(event_obs, actuals, preds)[0]:.3f}')\n",
        "print(f'Number of concordant pairs (scikit-survival): {ci_scikit(event_obs, actuals, preds)[1]}')\n",
        "print(f'Number of discordant pairs (scikit-survival): {ci_scikit(event_obs, actuals, preds)[2]}')\n",
        "df = pd.DataFrame(data={'actuals':actuals, 'predictions': preds, 'event_observed': event_obs}); df"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Concordance-index (lifelines): 0.100\n",
            "Concordance-index (scikit-survival): 0.900\n",
            "Number of concordant pairs (scikit-survival): 9\n",
            "Number of discordant pairs (scikit-survival): 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>actuals</th>\n",
              "      <th>predictions</th>\n",
              "      <th>event_observed</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>4</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>0</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   actuals  predictions  event_observed\n",
              "0        0            4            True\n",
              "1        1            3            True\n",
              "2        2            1            True\n",
              "3        3            2            True\n",
              "4        4            0            True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vXn8Kfgg6QJK",
        "colab_type": "text"
      },
      "source": [
        "In the previous example the order of the pair (2,3) has been inversed which causes a discordant pair (in terms of risks). There is 9 concordant pairs (in terms of risks) out of 10 pairs, therefore c-index=0.9.\n",
        "\n",
        "Let's see an example with censoring:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vLj7qt1Os1fD",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 272
        },
        "outputId": "72a2d332-4db2-4213-bbce-8d858317b991"
      },
      "source": [
        "actuals = [0, 1, 2, 3, 4]\n",
        "preds = [4, 1.5, 2, 1, 0]\n",
        "event_obs = [True, True, False, True, True]\n",
        "\n",
        "print(f'Concordance-index (lifelines): {ci_lifelines(actuals, preds, event_obs):.3f}')\n",
        "print(f'Concordance-index (scikit-survival): {ci_scikit(event_obs, actuals, preds)[0]:.3f}')\n",
        "print(f'Number of concordant pairs (scikit-survival): {ci_scikit(event_obs, actuals, preds)[1]}')\n",
        "print(f'Number of discordant pairs (scikit-survival): {ci_scikit(event_obs, actuals, preds)[2]}')\n",
        "df = pd.DataFrame(data={'actuals':actuals, 'predictions': preds, 'event_observed': event_obs}); df"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Concordance-index (lifelines): 0.125\n",
            "Concordance-index (scikit-survival): 0.875\n",
            "Number of concordant pairs (scikit-survival): 7\n",
            "Number of discordant pairs (scikit-survival): 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>actuals</th>\n",
              "      <th>predictions</th>\n",
              "      <th>event_observed</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>1.5</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>2.0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>1.0</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>0.0</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   actuals  predictions  event_observed\n",
              "0        0          4.0            True\n",
              "1        1          1.5            True\n",
              "2        2          2.0           False\n",
              "3        3          1.0            True\n",
              "4        4          0.0            True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VmZwjqXUh2P9",
        "colab_type": "text"
      },
      "source": [
        "Since the event $2$ is censored, the pairs $(2,3)$ and $(2,4)$ don't appear in the potential comparisons pairs. There is only one discordant pair $(1,2)$ since in the actuals $2$ lived longer than $1$, while in the predictions $1$ has lower risk than $2$."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VfMR2uf66fUN",
        "colab_type": "text"
      },
      "source": [
        "Pysurvival concordance index inputs are:\n",
        "+ a model\n",
        "+ input samples\n",
        "+ event/censoring times\n",
        "+ event observed\n",
        "\n",
        "Pysurvival concordance index outputs are:\n",
        "+ the concordance index\n",
        "+ if additional_results=True \n",
        "  - Number of pairs\n",
        "  - Number of concordant pairs\n",
        "\n",
        "Pysurvival concordance index needs a model as input so it's harder to see what does the concordance index do. Let's try it with a Random survival forest:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "97IVylML5qCE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "np.random.seed(42)\n",
        "x = np.linspace(0,10)\n",
        "y = x + np.random.normal(0, 1, x.shape)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1lrZnxVb5wgb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x1 = x[...,None] # we add a dimension so that RandomSurvivalForestModel doesn't complain\n",
        "x_trn, x_val = x1[:40], x1[40:]\n",
        "y_trn, y_val = y[:40], y[40:]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fPpS7bzWs-Tq",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "a93793d2-6304-469e-f397-47f61cfbbdfc"
      },
      "source": [
        "from pysurvival.models.survival_forest import RandomSurvivalForestModel\n",
        "\n",
        "rsf = RandomSurvivalForestModel(num_trees=10)\n",
        "%time rsf.fit(x_trn, y_trn, np.ones(len(x_trn)), seed=0)"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "CPU times: user 9.65 ms, sys: 2.71 ms, total: 12.4 ms\n",
            "Wall time: 21.4 ms\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "RandomSurvivalForestModel"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2s3PkDnG53QU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "event_obs = [True for i in range(len(x_trn))]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gBot2Zzu57N0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "ec463b67-354a-4e13-d5f2-8896716f897d"
      },
      "source": [
        "ci_pysurvival(rsf, x_trn, y_trn, event_obs, additional_results=True)"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{0: 0.8243589743589743, 1: 1560.0, 2: 1286.0}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iYXC0Grr6Co-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "400c7121-06e3-4b4e-b412-96d03ecafc9d"
      },
      "source": [
        "ci_scikit(event_obs, y_trn, rsf.predict_risk(x_trn))"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0.8243589743589743, 545, 39, 196, 0)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KYQ9LTXj5_B0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "9e03ee34-aacb-420f-c5dd-443b05801f8e"
      },
      "source": [
        "1-ci_lifelines(y_trn, rsf.predict_risk(x_trn), event_obs)"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.8243589743589743"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y_HjJzb9twOM",
        "colab_type": "text"
      },
      "source": [
        "I have no idea why the concordance index find more concordance pairs and more pairs overall but the result in the three cases is the same."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WDJYb_Tn4KWM",
        "colab_type": "text"
      },
      "source": [
        "Let's do random predictions and see what concordance index do we obtain."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sQx4zxpt0sCA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "actuals = np.arange(0,10,1)\n",
        "\n",
        "# choose 20 random seeds\n",
        "np.random.seed(0)\n",
        "seeds = np.random.permutation(1000)[:100]\n",
        "\n",
        "ci_random = []\n",
        "for _seed in seeds:\n",
        "  np.random.seed(_seed); preds_random = np.random.permutation(np.arange(0,10,1))\n",
        "  ci_random.append(ci_lifelines(actuals,preds_random))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jQsr3t_248xc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "e52d25b2-6fa8-4880-f884-e91ae6d44dab"
      },
      "source": [
        "print(f'Mean concordance index of random predictions: {np.mean(ci_random):.3f}')"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mean concordance index of random predictions: 0.517\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wc4yEkVG4vlk",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 513
        },
        "outputId": "7d2a69cb-c8be-47de-e1f4-d31924fa27b4"
      },
      "source": [
        "data = ci_random\n",
        "green_diamond = dict(markerfacecolor='g', marker='D')\n",
        "\n",
        "fig1, ax1 = plt.subplots(figsize=(7,7))\n",
        "ax1.set_title('Concordance indexes of random predictions')\n",
        "ax1.boxplot(data, notch=False, flierprops=green_diamond)\n",
        "ax1.set_xlabel('Random predictions')\n",
        "ax1.set_ylabel('Concordance index')\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfAAAAHwCAYAAABZrD3mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmcJXV97//X20GEEVSU0SjbEAQV\nE4PagttVxhCDShCNV8ElQryiuWJMNEZy9SKiJi4/jbkJMeICxg2JAe7IRZHoqHELMyguQFDCEgaN\nDAIujArI5/dHVeuZppcDTM3pb/N6Ph7n0V1V36r6nKrT591V9T11UlVIkqS23GnSBUiSpFvPAJck\nqUEGuCRJDTLAJUlqkAEuSVKDDHBJkhpkgGvJSHJSkjdMuo5pSZ6T5FO3cd6VSSrJVpu7riEkeUOS\nq5P814TWf1mSAyax7s0pyeFJvjAy/JMkv34blnObX3tqhwGuWSV5dpJ1/RvI95J8IsljJ11XS6rq\nQ1X1xEnXMbQkuwKvAPauql+bdD1LSVVtV1WXzNdmtn/27iivvTs6A1y3kOTlwDuAvwTuA+wK/D3w\n1EnWNa2Vo9I7kF2BH1TVVeM0vqPsvzvK89TkGODaRJK7A8cBL6mqU6vq+qq6sao+XlWv7NvcJck7\nkny3f7wjyV36afsnWZ/kFUmu6o/ejxhZ/rZJ3pbk8iQ/TPKFJNv20w5Ocn6S65J8NsmDRua7LMmr\nknwDuD7JVkkemuSrSX6c5KPANiPtd0hyRpINSa7tf995ZPpnk7w+yRf7+T+VZMeR6Y9N8qW+liuS\nHD7y3P+/JP+Z5PtJ/mG6/lm25czToZXkxUm+0y/3+CTppy3rl3t1kkuAp8zcL0ne22/PK/tT1sv6\nae9M8s8jbd+c5NMjyz4oyXn9Or+U5CEjbV/VL+/HSS5K8ttzvS6S/GO/PS9P8pokd+pPW58N3K8/\nW3PSLPNOvyZe1Z9iP3Ez7J/n9XX8IMmrZ6xvnNfnn4+8Pg9J8uQk305yTZL/Nds26Oc/qd/nZ/d1\nfS7JbjP28UuSfAf4Tj/ugX37a/pt/MyR9vdKsjrJj5KcA+wxY32V5P7973P97Xy+b35dvw8eNctr\n79FJ1vbzrU3y6HG2dZJtknyw387X9fPeZ67toy2sqnz4+OUDOBC4CdhqnjbHAV8B7g2sAL4EvL6f\ntn8//3HAnYEnAxuBHfrpxwOfBXYClgGPBu4C7AVcD/xOP9+fAxcDW/fzXQacB+wCbAtsDVwO/Gnf\n/hnAjcAb+vb3An4fWA5sD/wTcPrIc/gs8B/9erfth9/UT9sN+DFwWL/sewH79NP+GlgN3LNf7seB\nv5pjOx0OfGFkuIAzgHvQHbVuAA7sp70Y+Pf++d0TWNO336qffhrwLuCu/XY/B3hRP2058O1+ff8N\nuBrYuZ/2UOAqYL9+ez+/35Z3AR4AXAHcr2+7Ethjjufyj8D/7Z/zyn59LxjZ5+vneb3sT/eaeHO/\n3m1v5/7ZG/gJ8Lh+eW/vl3/ArXh9HtPv2xf2++HDfR0PBn4K7D7Hczmpf21Mr/tvZtnHZ/f7cNt+\nf10BHAFs1e+Pq+kuNwCcDJzSt/sN4MpZlnf/Bf52VjLyWpn52utruRZ4Xl/DYf3wvcbY1i+ie40v\n79f5cOBuk36f8tHv50kX4GNxPYDnAP+1QJv/AJ48Mvy7wGX97/v3b4CjbyZXAY+kO+PzU+C3Zlnm\n/wZOGRm+U/9mtn8/fBnwhyPTHwd8F8jIuC/RB/gsy98HuHZk+LPAa0aG/yfwyf73vwBOm2UZofsn\nY4+RcY8CLp1jnb98E+2HC3jsyPApwNH9758BXjwy7YnTb8p0lzF+Dmw7Mv0wYM3I8H7ANXT/1Bw2\nMv6d9OE1Mu4i4PHA/ft9cwBw53n29zLgBvrQ6ce9CPjsyD5fKMBvALaZp82t2T/HACePTLtrv/zp\nAB/n9bmsH96+3877jbQ/FzhkjjpPmrHu7YBfALuM7OMnjEx/FvCvM5bxLuC1/Xa9EXjgyLS/nOU1\nc3/m/9tZyfwB/jzgnBnzfBk4fIxt/Yd0f1cPme89wcdkHl6j0Uw/AHZMslVV3TRHm/vRBcW0y/tx\nv1zGjHk30r3R7Uh3mvs/FlpmVd2c5Aq6o41pV8xof2X17zIjdQCQZDnd0fKBwA796O2TLKuqX/TD\noz2mp2uE7ih4thpX0B2JnNufnYYu1JfN0nYuc63zfmz6/Ea37250R4vfG1nvnUbbV9W/9afe7033\nj8HovM9P8tKRcVvTHXV/LsmfAMcCD05yFvDyqvrujJp37Nc/c5/vxPg2VNXPpgdu5/7ZZFtV1fVJ\nfjDSdpzX5/Q6ftr//P7I9J+OrGs2o+v+SZJrZtQ0uh93A/ZLct3IuK2AD9C9nrZi7v0+ar6/nYXM\n3B7T6xndf3Nt6w/Q/T2cnOQewAeBV1fVjbehDm1mXgPXTF+mO9o7ZJ4236V7Y5q2az9uIVcDP2PG\ndb7Zltlfv92F7ih82mhYfw/YKSOJ1tcx7RV0p4j3q6q70R2xQxe4C7lijhqvpntzf3BV3aN/3L2q\n5nuzH9f36J7vtNHncgXdPtlxZL13q6oHTzdI8hK606nfpbv8MDrvG0fmu0dVLa+qjwBU1Yer6rF0\n277oTnPPdDXdkeLMfX7lLG3nMvNrD2/P/tlkW/X/DNxrZPptfX2Oa3Td29Gdoh5d/uhzvQL43Izt\nv11V/RHdqfubmHu/j5rvb2ehr5ScuT2m17Pg/quu/8vrqmpvulP2BwF/sNB82jIMcG2iqn5Id4ry\n+L5zz/Ikd07ypCRv6Zt9BHhNkhV9Z5dj6P4zX2jZNwPvA96e5H7pOm49qu9gdArwlCS/neTOdG/w\nP6c7fTebL9O9+f1xX9/TgX1Hpm9PF7bXJbkn3SnLcX0IOCDJM9N1lrtXkn36+t8N/HWSewMk2SnJ\n796KZc/llP657JxkB+Do6QlV9T3gU8DbktwtXeexPZI8vq9hL+ANwHPpTpf+eZJ9+tnfDbw4yX7p\n3DXJU5Jsn+QBSZ7Qb/+f0W2vm2cW1h+tngK8sZ9vN+DljLHP53F79s/HgIPSdTTcmu6a9+h72W16\nfd4KTx5Z9+uBr1TVFXO0PQPYK12nuzv3j0ckeVC/XU8Fju3/zvam66NwCwv87Wyg229zfV78zL6G\nZ/ev52fR9SM4Y6EnmmRVkt9M12HyR3T/yN3iNaLJMMB1C1X1Nro36NfQvTlcARwFnN43eQOwDvgG\n8E3gq/24cfxZP89aumu2bwbuVFUX0QXQ39Idbfwe8HtVdcMcNd4APJ3uWt81dNcaTx1p8g66DjlX\n03Vo+uSY9VFV/0nX+e4V/bLPA36rn/wqus51X0nyI+Bf6I4kb693A2cBX6fbnqfOmP4HdKe+L6Dr\ngPQx4L7pPqr0QeDNVfX1qvoO8L+ADyS5S1Wto+uo9Xf9fBfTbTPojtjfRLeN/ovu9PtfzFHfS+mu\n/18CfIGu09f7bsfzvT3753zgJX0N36N7XutHmtye1+c4Pkz3D8c1dJ26njtPrT+m689wKN2R8H/x\nq8580P1dbdePPwk4cZ71zvW3sxF4I/DFvqf4I2fU8AO6I+dX0F0i+3PgoKq6eozn+mt0r7UfARcC\nn6M7ra5FIJteQpQkzSXdx+TWV9VrJl2L5BG4JEkNMsAlSWrQoKfQkxxId6ODZcB7qupNM6bvRncd\nbQXdNZ3nVtX6WyxIkiRtYrAA73stfpvuzlrr6TpeHFZVF4y0+SfgjKp6f5InAEdU1fMGKUiSpCVk\nyBu57AtcXP036SQ5me7LMC4YabM3XW9n6G4deToL2HHHHWvlypWbt1JJkhaJc8899+qqWrFQuyED\nfCc2vcPQerrbPY76Ot1Hgf4GeBrdnZju1X/sYVYrV65k3bp1m7tWSZIWhSRz3ZFvE5PuxPZnwOOT\nfI3u3sxX0t1XeBNJjkz33dTrNmzYsKVrlCRp0RkywK9k01sE7syMW/dV1Xer6ulV9VDg1f240XsG\nT7c7oaqmqmpqxYoFzypIkrTkDRnga4E9k+ze33LwULqvYfylJDsmma7hL7h9d3aSJOkOY7AA77+N\n6ii620NeSPdVkecnOS7JwX2z/YGLknyb7isT3zhUPZIkLSXN3Up1amqq7MQmSVqqkpxbVVMLtZt0\nJzZJknQbGOCSJDXIAJckqUEGuCRJDTLAJUlqkAEuSVKDDHBJkhpkgEuS1CADXJKkBhngkua0Zs0a\nVu65kjVr1ky6FEkzGOCSZrVmzRoOetpBXL775Rz0tIMMcWmRMcAl3cJ0eG88ZCM8BjYestEQlxYZ\nA1zSJjYJ7937kbsb4tJiY4BL2sQRRx7Bxn1Hwnva7rBx340cceQRE6lL0qYMcEmbOPGEE1l+znK4\ndMaES2H5Ocs58YQTJ1KXpE0Z4JI2sWrVKs447QyWnz4S4pfC8tOXc8ZpZ7Bq1aqJ1iepY4BLuoVN\nQvyLhre0GBngkmY1HeK7Xbqb4S0tQltNugBJi9eqVau47DuXTboMSbPwCFySpAYZ4JIkNcgAlySp\nQQa4JEkNMsAlSWqQAS5JUoMMcEmSGmSAS5LUIANckqQGGeCSJDXIAJckqUEGuCRJDTLAJUlqkAEu\nSVKDDHBJkhpkgEuS1CADXJKkBhngkiQ1yACXJKlBBrgkSQ0ywCVJapABLklSgwxwSZIaZIBLktQg\nA1ySpAYZ4JIkNcgAlySpQQa4JEkNGjTAkxyY5KIkFyc5epbpuyZZk+RrSb6R5MlD1iNJ0lIxWIAn\nWQYcDzwJ2Bs4LMneM5q9Bjilqh4KHAr8/VD1SJK0lAx5BL4vcHFVXVJVNwAnA0+d0aaAu/W/3x34\n7oD1SJK0ZGw14LJ3Aq4YGV4P7DejzbHAp5K8FLgrcMBsC0pyJHAkwK677rrZC5WWsiSTLgGAqpp0\nCdKSMulObIcBJ1XVzsCTgQ8kuUVNVXVCVU1V1dSKFSu2eJFSy6rqdj02xzIMb2nzGzLArwR2GRne\nuR836gXAKQBV9WVgG2DHAWuSJGlJGDLA1wJ7Jtk9ydZ0ndRWz2jzn8BvAyR5EF2AbxiwJkmSloTB\nAryqbgKOAs4CLqTrbX5+kuOSHNw3ewXwwiRfBz4CHF6ea5MkaUFDdmKjqs4Ezpwx7piR3y8AHjNk\nDZIkLUWT7sQmSZJuAwNckqQGGeCSJDXIAJckqUEGuCRJDTLAJUlqkAEuSVKDDHBJkhpkgEuS1CAD\nXJKkBhngkiQ1yACXJKlBBrgkSQ0ywCVJapABLklSgwxwSZIaZIBLktQgA1ySpAYZ4JIkNcgAlySp\nQQa4JEkNMsAlSWqQAS5JUoMMcEmSGmSAS5LUIANckqQGGeCSJDXIAJckqUEGuCRJDTLAJUlqkAEu\nSVKDDHBJkhpkgEuS1CADXJKkBhngkiQ1yACXJKlBBrgkSQ0ywCVJapABLklSgwxwSZIaZIBLktQg\nA1ySpAYZ4JIkNcgAlySpQQa4JEkNMsAlSWqQAS5JUoMGDfAkBya5KMnFSY6eZfpfJzmvf3w7yXVD\n1iNJ0lKx1VALTrIMOB74HWA9sDbJ6qq6YLpNVf3pSPuXAg8dqh5JkpaSIY/A9wUurqpLquoG4GTg\nqfO0Pwz4yID1SJK0ZAwZ4DsBV4wMr+/H3UKS3YDdgc/MMf3IJOuSrNuwYcNmL1SSpNYslk5shwIf\nq6pfzDaxqk6oqqmqmlqxYsUWLk2SpMVnyAC/EthlZHjnftxsDsXT55IkjW3IAF8L7Jlk9yRb04X0\n6pmNkjwQ2AH48oC1SJK0pAwW4FV1E3AUcBZwIXBKVZ2f5LgkB480PRQ4uapqqFokSVpqBvsYGUBV\nnQmcOWPcMTOGjx2yBkmSlqLF0olNkiTdCga4JEkNMsAlSWqQAS5JUoMMcEmSGmSAS5LUIANckqQG\nGeCSJDXIAJckqUEGuCRJDTLAJUlqkAEuSVKDDHBJkhpkgEuS1CADXJKkBhngkiQ1yACXJKlBBrgk\nSQ0ywCVJapABLklSgwxwSZIaZIBLktQgA1ySpAYZ4JIkNcgAlySpQQa4JEkNMsAlSWqQAS5JUoMM\ncEmSGmSAS5LUIANckqQGGeCSJDXIAJckqUEGuCRJDTLAJUlqkAEuSVKDDHBJkhpkgEuS1CADXJKk\nBhngkiQ1yACXJKlBBrgkSQ0ywCVJapABLklSgwxwSZIaZIBLktSgQQM8yYFJLkpycZKj52jzzCQX\nJDk/yYeHrEeSpKViq6EWnGQZcDzwO8B6YG2S1VV1wUibPYG/AB5TVdcmufdQ9UiStJQMeQS+L3Bx\nVV1SVTcAJwNPndHmhcDxVXUtQFVdNWA9kiQtGUMG+E7AFSPD6/txo/YC9kryxSRfSXLggPVIkrRk\nDHYK/Vasf09gf2Bn4PNJfrOqrhttlORI4EiAXXfddUvXKE3MPe95T6699tpJl0GSia5/hx124Jpr\nrploDdJiM2SAXwnsMjK8cz9u1Hrg36rqRuDSJN+mC/S1o42q6gTgBICpqakarGJpkbn22mup8iU/\n6X8gpMVoyFPoa4E9k+yeZGvgUGD1jDan0x19k2RHulPqlwxYkyRJS8KCAZ7kBTOGlyV57ULzVdVN\nwFHAWcCFwClVdX6S45Ic3Dc7C/hBkguANcArq+oHt/ZJSJJ0R5OFTs/1n82+B/AC4J7AScDnqurP\nBq9uFlNTU7Vu3bpJrFra4pJ4Ch23g+5YkpxbVVMLtVvwGnhVPTvJs4BvAtcDz66qL26GGiVJ0m00\nzin0PYGXAf8MXA48L8nyoQuTJElzG6cT28eBY6rqRcDjge8wo5e4JEnassb5GNm+VfUjgOouQr0t\nyceHLUuSJM1nnCPwm5L87yTvhl+eUt9r2LIkSdJ8xgnwE4GfA4/qh68E3jBYRZIkaUHjBPgeVfUW\n4EaAqtoIeFskSZImaJwAvyHJtkABJNmD7ohckiRNyDid2F4LfBLYJcmHgMcAhw9ZlCRJmt84N3I5\nO8lXgUfSnTp/WVVdPXhlkiRpTnMGeJKHzRj1vf7nrkl2raqvDleWJEmaz3xH4G/rf24DTAFfpzsC\nfwiwjl/1SpckSVvYnJ3YqmpVVa2iO/J+WFVNVdXDgYdyy+/1liRJW9A4vdAfUFXfnB6oqm8BDxqu\nJEmStJBxeqF/I8l7gA/2w88BvjFcSZIkaSHjBPgRwB/RfSMZwOeBdw5WkSRJWtA4HyP7GfDX/UOS\nJC0CCwZ4kscAxwK7jbavql8frixJkjSfcU6hvxf4U+Bc4BfDliNJksYxToD/sKo+MXglkiRpbOME\n+JokbwVOZeRLTLwTmyRJkzNOgO/X/5waGVfAEzZ/OZIkaRzj9EJftSUKkSRJ45vvy0yeW1UfTPLy\n2aZX1duHK0uSJM1nviPwu/Y/t98ShUiSpPHNGeBV9a7+5+u2XDmSJGkc43yZiSRJWmQMcEmSGmSA\nS5LUoAUDPMl9krw3ySf64b2TvGD40iRJ0lzGOQI/CTgLuF8//G3gT4YqSJIkLWycAN+xqk4Bbgao\nqpvwS00kSZqocW6len2Se9HdPpUkjwR+OGhVkgCo194Njr37pMuYuHrt3SZdgrTojBPgLwdWA3sk\n+SKwAnjGoFVJAiCv+xFVNekyJi4Jdeykq5AWl3Huhf7VJI8HHgAEuKiqbhy8MkmSNKdxeqG/BNiu\nqs6vqm8B2yX5n8OXJkmS5jJOJ7YXVtV10wNVdS3wwuFKkiRJCxknwJclyfRAkmXA1sOVJEmSFjJO\nJ7ZPAh9N8q5++EX9OEmSNCHjBPir6EL7j/rhs4H3DFaRJEla0Di90G8G3tk/JEnSIrBggCd5DHAs\nsFvfPkBV1a8PW5okSZrLOKfQ3wv8KXAu3kJVkqRFYZwA/2FVfWLwSiRJ0tjGCfA1Sd4KnAr8fHpk\nVX11sKokSdK8xgnw/fqfUyPjCnjC5i9HkiSNY5xe6Ku2RCGSJGl84xyBk+QpwIOBbabHVdVxY8x3\nIPA3wDLgPVX1phnTDwfeClzZj/q7qvIz5pIkLWCcj5H9A7AcWEV3A5dnAOeMMd8y4Hjgd4D1wNok\nq6vqghlNP1pVR93awiVJuiMb517oj66qPwCurarXAY8C9hpjvn2Bi6vqkqq6ATgZeOptL1WSJE0b\nJ8B/2v/cmOR+wI3AfceYbyfgipHh9f24mX4/yTeSfCzJLrMtKMmRSdYlWbdhw4YxVi1J0tI2ToCf\nkeQedNeqvwpcBnxkM63/48DKqnoI3T3W3z9bo6o6oaqmqmpqxYoVm2nVkiS1a5xe6K/vf/3nJGcA\n21TVD8dY9pXA6BH1zvyqs9r0sn8wMvge4C1jLFeSpDu8OQM8ydPnmUZVnbrAstcCeybZnS64DwWe\nPWM5962q7/WDBwMXjlW1JEl3cPMdgf9e//PewKOBz/TDq4Av0d2ZbU5VdVOSo4Cz6D5G9r6qOj/J\nccC6qloN/HGSg4GbgGuAw2/rE5Ek6Y4kVTV/g+RTwPOnj5ST3Bc4qap+dwvUdwtTU1O1bt26Saxa\n2uL6s12TLmPi3A66I0lyblVNLdRunE5su4yc5gb4PrDrba5MkiTdbuPcie3TSc7iVz3PnwX8y3Al\nSZKkhYzTC/2oJE8DHtePOqGqThu2LEmSNJ95A7y/Heq/9F9oYmhLkrRIzHsNvKp+Adyc5O5bqB5J\nkjSGca6B/wT4ZpKzgeunR1bVHw9WlSRJmtc4AX4qC3zmW5IkbVnjdGJ7f5Kt+dU3kF1UVTcOW5Yk\nSZrPON8Hvj/dl4xcBgTYJcnzq+rzw5YmSZLmMs4p9LcBT6yqiwCS7EX3mfCHD1mYJEma2zh3Yrvz\ndHgDVNW3gTsPV5IkSVrIOEfg65K8B/hgP/wcwJuRS5I0QeME+B8BLwGmPzb2r8DfD1aRJEla0DgB\nvhXwN1X1dvjl3dnuMmhVkiRpXuNcA/80sO3I8Lb4ZSaSJE3UOAG+TVX9ZHqg/335cCVJkqSFjBPg\n1yd52PRAkocDPx2uJEmStJBxroH/CfBPSb5LdyOXX6P7TnBJkjQh49xKdW2SBwIP6Ed5K1VJkiZs\nnCNwgEcAK/v2D0tCVf3jYFVJkqR5jXMv9A8AewDnAb/oRxdggEuSNCHjHIFPAXtXVQ1djCRJGs84\nvdC/RddxTZIkLRLjHIHvCFyQ5Bzg59Mjq+rgwaqSJEnzGifAjx26CEmSdOuM8zGyzyW5D11PdIBz\nquqqYcuSJEnzWfAaeJJnAucA/x14JvBvSZ4xdGGSJGlu45xCfzXwiOmj7iQr6L7M5GNDFiZJkuY2\nTi/0O804Zf6DMeeTJEkDGecI/JNJzgI+0g8/C/jEcCVJkqSFjNOJ7ZVJng48th91QlWdNmxZkiRp\nPnMGeJL7A/epqi9W1anAqf34xybZo6r+Y0sVKUmSNjXftex3AD+aZfwP+2mSJGlC5gvw+1TVN2eO\n7MetHKwiSZK0oPkC/B7zTNt2cxciSZLGN1+Ar0vywpkjk/wP4NzhSpIkSQuZrxf6nwCnJXkOvwrs\nKWBr4GlDFyZJkuY2Z4BX1feBRydZBfxGP/r/VdVntkhlkiRpTuN8DnwNsGYL1CJpFkkmXcLE7bDD\nDpMuQVp0xrkTm6QJqapJl0CSRVGHpE15T3NJkhpkgEuS1CADXJKkBhngkiQ1yACXJKlBBrgkSQ0y\nwCVJatCgAZ7kwCQXJbk4ydHztPv9JJVkash6JElaKgYL8CTLgOOBJwF7A4cl2XuWdtsDLwP+baha\nJElaaoY8At8XuLiqLqmqG4CTgafO0u71wJuBnw1YiyRJS8qQAb4TcMXI8Pp+3C8leRiwS1X9v/kW\nlOTIJOuSrNuwYcPmr1SSpMZMrBNbkjsBbwdesVDbqjqhqqaqamrFihXDFydJ0iI3ZIBfCewyMrxz\nP27a9nRfU/rZJJcBjwRW25FNkqSFDRnga4E9k+yeZGvgUGD19MSq+mFV7VhVK6tqJfAV4OCqWjdg\nTZIkLQmDBXhV3QQcBZwFXAicUlXnJzkuycFDrVeSpDuCQb8PvKrOBM6cMe6YOdruP2QtkiQtJd6J\nTZKkBhngkiQ1yACXJKlBBrgkSQ0ywCVJapABLklSgwxwSZIaZIBLktQgA1ySpAYZ4JIkNcgAlySp\nQQa4JEkNMsAlSWqQAS5JUoMMcEmSGmSAS5LUIANckqQGGeCSJDXIAJckqUEGuCRJDTLAJUlqkAEu\nSVKDDHBJkhpkgEuS1CADXJKkBhngkiQ1yACXJKlBBrgkSQ0ywCVJapABLklSgwxwSZIaZIBLktQg\nA1ySpAYZ4JIkNcgAlySpQQa4JEkNMsAlSWqQAS5JUoMMcEmSGmSAS5LUIANckqQGGeCSJDXIAJck\nqUEGuCRJDTLAJUlqkAEuSVKDBg3wJAcmuSjJxUmOnmX6i5N8M8l5Sb6QZO8h65EkaakYLMCTLAOO\nB54E7A0cNktAf7iqfrOq9gHeArx9qHokSVpKhjwC3xe4uKouqaobgJOBp442qKofjQzeFagB65Ek\nacnYasBl7wRcMTK8HthvZqMkLwFeDmwNPGHAeiRJWjIm3omtqo6vqj2AVwGvma1NkiOTrEuybsOG\nDVu2QEmSFqEhA/xKYJeR4Z37cXM5GThktglVdUJVTVXV1IoVKzZjiZIktWnIAF8L7Jlk9yRbA4cC\nq0cbJNlzZPApwHcGrEeSpCVjsGvgVXVTkqOAs4BlwPuq6vwkxwHrqmo1cFSSA4AbgWuB5w9VjyRJ\nS8mQndioqjOBM2eMO2bk95cNuX5JkpaqiXdikyRJt54BLklSgwxwSZIaZIBLktQgA1ySpAYZ4JIk\nNcgAlySpQQa4JEkNMsAlSWqQAS5JUoMMcEmSGmSAS5LUIANckqQGGeCSJDXIAJckqUEGuCRJDTLA\nJUlqkAEuSVKDDHBJkhpkgEuS1CADXJKkBhngkiQ1yACXJKlBBrgkSQ0ywCVJapABLklSgwxwSZIa\nZIBLktQgA1ySpAYZ4JIkNcgAlySpQQa4JEkNMsAlSWqQAS5JUoMMcEmSGmSAS5LUIANckqQGGeCS\nJDVoq0kXIGlYSRbFMqrqdi9D0q8Y4NISZ3BKS5On0CVJapABLklSgwxwSZIaZIBLktQgA1ySpAYZ\n4JIkNcgAlySpQYMGeJIDk1w8xt3tAAAHiklEQVSU5OIkR88y/eVJLkjyjSSfTrLbkPVIkrRUDBbg\nSZYBxwNPAvYGDkuy94xmXwOmquohwMeAtwxVjyRJS8mQR+D7AhdX1SVVdQNwMvDU0QZVtaaqNvaD\nXwF2HrAeSZKWjCEDfCfgipHh9f24ubwA+MRsE5IcmWRdknUbNmzYjCVKktSmRdGJLclzgSngrbNN\nr6oTqmqqqqZWrFixZYuTJGkRGvLLTK4EdhkZ3rkft4kkBwCvBh5fVT8fsB5JkpaMIY/A1wJ7Jtk9\nydbAocDq0QZJHgq8Czi4qq4asBZJkpaUwQK8qm4CjgLOAi4ETqmq85Mcl+Tgvtlbge2Af0pyXpLV\ncyxOkiSNGPT7wKvqTODMGeOOGfn9gCHXL0nSUrUoOrFJkqRbxwCXJKlBBrgkSQ0ywCVJapABLklS\ngwxwSZIaZIBLktQgA1ySpAYZ4JIkNcgAlySpQQa4JEkNMsAlSWqQAS5JUoMMcEmSGmSAS5LUIANc\n0pzWrFnDyj1XsmbNmkmXImkGA1zSrNasWcNBTzuIy3e/nIOedpAhLi0yBrikW5gO742HbITHwMZD\nNhri0iJjgEvaxCbhvXs/cndDXFpsDHBJmzjiyCPYuO9IeE/bHTbuu5EjjjxiInVJ2pQBLmkTJ55w\nIsvPWQ6XzphwKSw/ZzknnnDiROqStCkDXNImVq1axRmnncHy00dC/FJYfvpyzjjtDFatWjXR+iR1\nDHBJt7BJiH/R8JYWIwNc0qymQ3y3S3czvKVFaKtJFyBp8Vq1ahWXfeeySZchaRYegUuS1CADXJKk\nBhngkiQ1yACXJKlBBrgkSQ0ywCVJapABLklSgwxwSZIaZIBLktQgA1ySpAYZ4JIkNcgAlySpQQa4\nJEkNMsAlSWpQqmrSNdwqSTYAl0+6DukOZEfg6kkXId2B7FZVKxZq1FyAS9qykqyrqqlJ1yFpU55C\nlySpQQa4JEkNMsAlLeSESRcg6Za8Bi5JUoM8ApckqUEGuCRJDTLAJc0qyfuSXJXkW5OuRdItGeCS\n5nIScOCki5A0OwNc0qyq6vPANZOuQ9LsDHBJkhpkgEuS1CADXJKkBhngkiQ1yACXNKskHwG+DDwg\nyfokL5h0TZJ+xVupSpLUII/AJUlqkAEuSVKDDHBJkhpkgEuS1CADXJKkBhng0oCS/CLJeUm+leTj\nSe6xmZa7soVvCUtybJI/638/LskB87TdJ8mTR4YPTnL0lqhTapEBLg3rp1W1T1X9Bt0Xg7xk0gXd\nXkm2ui3zVdUxVfUv8zTZB3jySPvVVfWm27Iu6Y7AAJe2nC8DOwEk2S7Jp5N8Nck3kzy1H78yyYVJ\n3p3k/CSfSrJtP+3hSb6e5OuM/COQZJskJ/bL+VqSVf34w5OcnuTsJJclOSrJy/s2X0lyz5kFJjkp\nyT8kWZfk20kOGlnW6iSfAT7dj3tlkrVJvpHkdSPLeHU/7xeAB8xY9jP63x+R5Ev98zknyd2B44Bn\n9WcsntWv8+9Gtstn+nV9OsmuI8v8P/2yLhlZ/n2TfH7k7Md/21w7UVosDHBpC0iyDPhtYHU/6mfA\n06rqYcAq4G1J0k/bEzi+qh4MXAf8fj/+ROClVfVbMxb/EqCq6jeBw4D3J9mmn/YbwNOBRwBvBDZW\n1UPp/pn4gznKXQnsCzwF+IeRZT0MeEZVPT7JE/s696U7cn54kscleThwKL86mn7ELNtia+CjwMv6\n53IAcD1wDPDR/ozFR2fM9rfA+6vqIcCHgP8zMu2+wGOBg4DpI/ZnA2dV1T7AbwHnzfFcpWbdplNh\nksa2bZLz6I68LwTO7scH+MskjwNu7qffp592aVVNB865wMr+2vk9+u/oBvgA8KT+98fSBRxV9e9J\nLgf26qetqaofAz9O8kPg4/34bwIPmaPmU6rqZuA7SS4BHtiPP7uqpr8f/In942v98HZ0gb49cFpV\nbQRIsppbegDwvapa29f8o77tHOUA8Ci6f0Smn/tbRqad3td7QZLpbbgWeF+SO/fTDXAtOR6BS8P6\naX8UuBtdaE+f+n4OsAJ4eD/9+8D0ke7PR+b/BbfvH+3RZd08MnzzPMudeX/l6eHrR8YF+Kv+aHmf\nqrp/Vb33dtR5e4w+xwD0/+g8DrgSOCnJXGcbpGYZ4NIW0B+R/jHwir4T2N2Bq6rqxv6a9W4LzH8d\ncF2Sx/ajnjMy+V+nh5PsBewKXHQ7yv3vSe6UZA/g1+dY1lnAHybZrl/vTknuDXweOCTJtkm2B35v\nlnkvAu6b5BH9vNv32+THdEfws/kS3al56J7rv873BJLsBny/qt4NvIfu9L+0pHgKXdpCquprSb5B\nd536Q8DHk3wTWAf8+xiLOILutHABnxoZ//fAO/tl3QQcXlU/X+CU9Hz+EzgHuBvw4qr62cxlVdWn\nkjwI+HI/7SfAc6vqq0k+CnwduIruVPbMeW9I8izgb/sOej+luw6+Bji6v+TwVzNmeylwYpJXAhv6\nbTGf/YFXJrmxr80jcC05fhuZpF9KchJwRlV9bNK1SJqfp9AlSWqQR+CSJDXII3BJkhpkgEuS1CAD\nXJKkBhngkiQ1yACXJKlB/z/R8Rd7R92BhgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 504x504 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nRE2p7s1b_Rn",
        "colab_type": "text"
      },
      "source": [
        "**Sources**\n",
        "\n",
        "+ [Lifelines' concordance index](https://lifelines.readthedocs.io/en/latest/lifelines.utils.html#lifelines.utils.concordance_index)\n",
        "+ [Scikit-survival's concordance index](https://scikit-survival.readthedocs.io/en/latest/generated/sksurv.metrics.concordance_index_censored.html#sksurv.metrics.concordance_index_censored)\n",
        "+ [Pysurvival's concordance index](https://square.github.io/pysurvival/metrics/c_index.html)"
      ]
    }
  ]
}