examples/notebooks/hyperparameter_optimization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Hyperparameter optimization"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Import packages"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-06-29 13:47:44.314433: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.\n",
      "2023-06-29 13:47:44.343799: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.\n",
      "2023-06-29 13:47:44.344425: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
      "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2023-06-29 13:47:44.885955: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n"
     ]
    }
   ],
   "source": [
    "from rdkit import RDLogger\n",
    "import logging\n",
    "import warnings\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "logger = logging.getLogger()\n",
    "logger.setLevel(logging.CRITICAL)\n",
    "RDLogger.DisableLog('rdApp.*')\n",
    "\n",
    "from sklearn.metrics import accuracy_score\n",
    "from deepmol.metrics import Metric\n",
    "from sklearn.svm import SVC\n",
    "from deepmol.parameter_optimization import HyperparameterOptimizerValidation\n",
    "\n",
    "from deepmol.splitters import RandomSplitter\n",
    "from deepmol.compound_featurization import MorganFingerprint\n",
    "from deepmol.loaders import SDFLoader\n",
    "\n",
    "\n",
    "from tensorflow.keras.layers import Dropout\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T12:47:46.609909992Z",
     "start_time": "2023-06-29T12:47:43.943184770Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## First of all, let's load the data"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-06-29 13:48:10,595 — INFO — Assuming classification since there are less than 10 unique y values. If otherwise, explicitly set the mode to 'regression'!\n"
     ]
    }
   ],
   "source": [
    "dataset = SDFLoader(\"../data/CHEMBL217_conformers.sdf\", id_field=\"_ID\", labels_fields=[\"_Class\"]).create_dataset()\n",
    "train_dataset, valid_dataset, test_dataset = RandomSplitter().train_valid_test_split(dataset, frac_train=0.8, frac_valid=0.1, frac_test=0.1)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T12:48:10.619861670Z",
     "start_time": "2023-06-29T12:48:09.403757762Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Let's featurize the data"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "outputs": [],
   "source": [
    "morgan_fingerprints = MorganFingerprint()\n",
    "morgan_fingerprints.featurize(train_dataset, inplace=True)\n",
    "morgan_fingerprints.featurize(valid_dataset, inplace=True)\n",
    "morgan_fingerprints.featurize(test_dataset, inplace=True)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T12:48:19.291459568Z",
     "start_time": "2023-06-29T12:48:15.501965505Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Hyperparameter tuning with scikit-learn"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "DeepMol provide methods to perform hyperparameter tuning with scikit-learn models. The hyperparameter tuning can be performed with a validation set, previously created, or with cross validation. Anyway, the hyperparameter tuning is performed with a random search if the number of iterations is specified, otherwise a grid search is performed.\n",
    "\n",
    "Moreover, for each method the user have to specify the metric to optimize and if the metric has to be maximized or minimized. The user must specify the parameters to optimize and the values to try for each parameter.\n",
    "\n",
    "The parameters must be specified as a dictionary, where the keys are the parameters names and the values are the values to try.\n",
    "\n",
    "Let's see how to perform hyperparameter tuning of a SVM with a validation set."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-06-29 13:48:33,584 — INFO — Fitting 2 random models from a space of 9 possible models.\n",
      "2023-06-29 13:48:33,585 — INFO — Fitting model 1/2\n",
      "2023-06-29 13:48:33,586 — INFO — hyperparameters: {'C': 1.0, 'kernel': 'rbf'}\n",
      "2023-06-29 13:49:09,693 — INFO — Model 1/2, Metric accuracy_score, Validation set 1: 0.983755\n",
      "2023-06-29 13:49:09,693 — INFO — \tbest_validation_score so far: 0.983755\n",
      "2023-06-29 13:49:09,694 — INFO — Fitting model 2/2\n",
      "2023-06-29 13:49:09,694 — INFO — hyperparameters: {'C': 0.8, 'kernel': 'rbf'}\n",
      "2023-06-29 13:49:46,486 — INFO — Model 2/2, Metric accuracy_score, Validation set 2: 0.982551\n",
      "2023-06-29 13:49:46,486 — INFO — \tbest_validation_score so far: 0.983755\n",
      "2023-06-29 13:50:27,809 — INFO — Best hyperparameters: {'C': 1.0, 'kernel': 'rbf'}\n",
      "2023-06-29 13:50:27,810 — INFO — train_score: 0.994811\n",
      "2023-06-29 13:50:27,810 — INFO — validation_score: 0.983755\n"
     ]
    }
   ],
   "source": [
    "params_dict_svc = {\"C\": [1.0, 1.2, 0.8], \"kernel\": ['linear', 'poly', 'rbf']} # The keys are the parameters names and the values are the values to try\n",
    "optimizer = HyperparameterOptimizerValidation(SVC,\n",
    "                                              metric=Metric(accuracy_score),\n",
    "                                              maximize_metric=True,\n",
    "                                              n_iter_search=2,\n",
    "                                              params_dict=params_dict_svc,\n",
    "                                              model_type=\"sklearn\")\n",
    "best_svm, best_hyperparams, all_results = optimizer.fit(train_dataset=train_dataset, valid_dataset=valid_dataset)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T12:50:27.869110989Z",
     "start_time": "2023-06-29T12:48:33.586107933Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "In the end, we can check the performance of the best model on the test set."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "outputs": [
    {
     "data": {
      "text/plain": "({'accuracy_score': 0.9873722188815394}, {})"
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_svm.evaluate(test_dataset, metrics = [Metric(accuracy_score)])"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T16:12:53.059934610Z",
     "start_time": "2023-06-29T16:12:48.154838951Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can also check the best combination of hyperparameters found."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "outputs": [
    {
     "data": {
      "text/plain": "{'C': 1.0, 'kernel': 'rbf'}"
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_hyperparams"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T16:12:57.175222348Z",
     "start_time": "2023-06-29T16:12:57.165621622Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can also check the performance of all the models trained during the hyperparameter tuning. Each model is defined by the name of the parameters followed by the value of the parameter."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "outputs": [
    {
     "data": {
      "text/plain": "{'_C_1.000000_kernel_rbf': 0.983754512635379,\n '_C_0.800000_kernel_rbf': 0.9825511432009627}"
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_results"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T16:12:59.339428954Z",
     "start_time": "2023-06-29T16:12:59.335874202Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Finally, save your best model for deployment and new predictions!"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "outputs": [],
   "source": [
    "best_svm.save(\"my_model\")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T16:13:01.122906113Z",
     "start_time": "2023-06-29T16:13:01.086667238Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Bring it back to life and make predictions!"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "outputs": [
    {
     "data": {
      "text/plain": "array([0., 0., 0., ..., 1., 0., 0.])"
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from deepmol.models import SklearnModel\n",
    "\n",
    "SklearnModel.load(\"my_model\").predict(test_dataset)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T16:13:07.823718426Z",
     "start_time": "2023-06-29T16:13:02.591602899Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Now let's try with cross validation"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-06-29 17:13:36,013 — INFO — MODEL TYPE: sklearn\n",
      "2023-06-29 17:13:36,014 — INFO — Fitting 2 random models from a space of 9 possible models.\n",
      "2023-06-29 17:16:14,964 — INFO — \n",
      " \n",
      " Best <function accuracy_score at 0x7f55eba87b80>: 0.981200 using {'kernel': 'poly', 'C': 0.8}\n",
      "2023-06-29 17:16:14,964 — INFO — \n",
      " <function accuracy_score at 0x7f55eba87b80>: 0.981200 (0.000907) with: {'kernel': 'poly', 'C': 0.8} \n",
      "\n",
      "2023-06-29 17:16:14,965 — INFO — \n",
      " <function accuracy_score at 0x7f55eba87b80>: 0.972627 (0.001673) with: {'kernel': 'linear', 'C': 1.0} \n",
      "\n",
      "2023-06-29 17:16:14,966 — INFO — Fitting best model!\n",
      "2023-06-29 17:16:53,755 — INFO — SklearnModel(mode='classification', model=SVC(C=0.8, kernel='poly'),\n",
      "             model_dir='/tmp/tmpybobff7g')\n"
     ]
    }
   ],
   "source": [
    "from deepmol.parameter_optimization import HyperparameterOptimizerCV\n",
    "\n",
    "params_dict_svc = {\"C\": [1.0, 1.2, 0.8], \"kernel\": ['linear', 'poly', 'rbf']}\n",
    "optimizer = HyperparameterOptimizerCV(SVC, metric=Metric(accuracy_score),\n",
    "                                          maximize_metric=True,\n",
    "                                          cv=3,\n",
    "                                          n_iter_search=2,\n",
    "                                          params_dict=params_dict_svc,\n",
    "                                          model_type=\"sklearn\")\n",
    "best_svm, best_hyperparams, all_results = optimizer.fit(train_dataset=train_dataset)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T16:16:53.805914351Z",
     "start_time": "2023-06-29T16:13:36.014811377Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Then, we can check the performance of the best model on the test set."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "outputs": [
    {
     "data": {
      "text/plain": "({'accuracy_score': 0.9849669272399278}, {})"
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_svm.evaluate(test_dataset, metrics = [Metric(accuracy_score)])"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T16:17:16.605476584Z",
     "start_time": "2023-06-29T16:17:11.951043378Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can also check the best combination of hyperparameters found."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "outputs": [
    {
     "data": {
      "text/plain": "{'kernel': 'poly', 'C': 0.8}"
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_hyperparams"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T16:17:16.631272142Z",
     "start_time": "2023-06-29T16:17:16.587213775Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can also check the performance of all the models trained during the hyperparameter tuning. Each model is defined by the name of the parameters followed by the value of the parameter."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "outputs": [
    {
     "data": {
      "text/plain": "{'mean_fit_time': array([20.26896993,  7.89151009]),\n 'std_fit_time': array([0.27499022, 0.29743476]),\n 'mean_score_time': array([9.4404827 , 2.50866826]),\n 'std_score_time': array([0.2355606 , 0.10412309]),\n 'param_kernel': masked_array(data=['poly', 'linear'],\n              mask=[False, False],\n        fill_value='?',\n             dtype=object),\n 'param_C': masked_array(data=[0.8, 1.0],\n              mask=[False, False],\n        fill_value='?',\n             dtype=object),\n 'params': [{'kernel': 'poly', 'C': 0.8}, {'kernel': 'linear', 'C': 1.0}],\n 'split0_test_score': array([0.9799233 , 0.97496052]),\n 'split1_test_score': array([0.98172795, 0.97180239]),\n 'split2_test_score': array([0.98194946, 0.97111913]),\n 'mean_test_score': array([0.98120024, 0.97262735]),\n 'std_test_score': array([0.00090745, 0.00167322]),\n 'rank_test_score': array([1, 2], dtype=int32)}"
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_results"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T16:18:19.663528953Z",
     "start_time": "2023-06-29T16:18:19.618441113Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Finally, save your best model for deployment and new predictions!"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "outputs": [],
   "source": [
    "best_svm.save(\"my_model\")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T16:18:22.686360386Z",
     "start_time": "2023-06-29T16:18:22.551821848Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Bring it back to life and make predictions!"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "outputs": [
    {
     "data": {
      "text/plain": "array([0., 0., 0., ..., 1., 0., 0.])"
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from deepmol.models import SklearnModel\n",
    "\n",
    "SklearnModel.load(\"my_model\").predict(test_dataset)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T16:18:28.806983385Z",
     "start_time": "2023-06-29T16:18:24.559846975Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Hyperparameter tuning with keras"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "DeepMol provide methods to perform hyperparameter tuning with keras models. The hyperparameter tuning can be performed with a validation set, previously created, or with cross validation. Anyway, the hyperparameter tuning is performed with a random search if the number of iterations is specified, otherwise a grid search is performed."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "As explained in the models section, to create a Keras model one have to define a function with the model architecture and the parameters to optimize."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "outputs": [],
   "source": [
    "def create_model(input_dim, optimizer='adam', dropout=0.5):\n",
    "    # create model\n",
    "    inputs = layers.Input(shape=input_dim)\n",
    "\n",
    "    # Define the shared layers\n",
    "    shared_layer_1 = layers.Dense(64, activation=\"relu\")\n",
    "    dropout_1 = Dropout(dropout)\n",
    "    shared_layer_2 = layers.Dense(32, activation=\"relu\")\n",
    "\n",
    "    # Define the shared layers for the inputs\n",
    "    x = shared_layer_1(inputs)\n",
    "    x = dropout_1(x)\n",
    "    x = shared_layer_2(x)\n",
    "\n",
    "    task_output = layers.Dense(1, activation=\"sigmoid\")(x)\n",
    "\n",
    "    # Define the model that outputs the predictions for each task\n",
    "    model = keras.Model(inputs=inputs, outputs=task_output)\n",
    "    # Compile the model with different loss functions and metrics for each task\n",
    "    model.compile(\n",
    "        optimizer=optimizer, loss=\"binary_crossentropy\", metrics=[\"accuracy\"]\n",
    "    )\n",
    "    return model\n"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T16:18:36.197038425Z",
     "start_time": "2023-06-29T16:18:36.193730148Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's see how to perform hyperparameter tuning of a DNN with a validation set."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-06-29 17:18:47,645 — INFO — Fitting 2 random models from a space of 3 possible models.\n",
      "2023-06-29 17:18:47,646 — INFO — Fitting model 1/2\n",
      "2023-06-29 17:18:47,647 — INFO — hyperparameters: {'input_dim': 2048, 'dropout': 0.5, 'optimizer': 'adam'}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-06-29 17:18:47.688410: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:266] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected\n",
      "2023-06-29 17:18:47.732633: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 108937216 exceeds 10% of free system memory.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "52/52 [==============================] - 0s 589us/step\n",
      "2023-06-29 17:21:47,008 — INFO — Model 1/2, Metric accuracy_score, Validation set 1: 0.980144\n",
      "2023-06-29 17:21:47,008 — INFO — \tbest_validation_score so far: 0.980144\n",
      "2023-06-29 17:21:47,009 — INFO — Fitting model 2/2\n",
      "2023-06-29 17:21:47,009 — INFO — hyperparameters: {'input_dim': 2048, 'dropout': 0.7, 'optimizer': 'adam'}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-06-29 17:21:47.047236: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 108937216 exceeds 10% of free system memory.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "52/52 [==============================] - 0s 579us/step\n",
      "2023-06-29 17:24:49,640 — INFO — Model 2/2, Metric accuracy_score, Validation set 2: 0.977738\n",
      "2023-06-29 17:24:49,640 — INFO — \tbest_validation_score so far: 0.980144\n",
      "270/416 [==================>...........] - ETA: 0s"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-06-29 17:24:49.657093: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 108937216 exceeds 10% of free system memory.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "416/416 [==============================] - 0s 567us/step\n",
      "2023-06-29 17:24:50,068 — INFO — Best hyperparameters: {'input_dim': 2048, 'dropout': 0.5, 'optimizer': 'adam'}\n",
      "2023-06-29 17:24:50,068 — INFO — train_score: 0.999925\n",
      "2023-06-29 17:24:50,069 — INFO — validation_score: 0.980144\n"
     ]
    }
   ],
   "source": [
    "params_dict_dense = {\n",
    "                   \"input_dim\": [train_dataset.X.shape[1]],\n",
    "                   \"dropout\": [0.5, 0.6, 0.7],\n",
    "                   \"optimizer\": [\"adam\"]\n",
    "                   }\n",
    "\n",
    "optimizer = HyperparameterOptimizerValidation(create_model,\n",
    "                                              metric=Metric(accuracy_score),\n",
    "                                              maximize_metric=True,\n",
    "                                              n_iter_search=2,\n",
    "                                              params_dict=params_dict_dense,\n",
    "                                              model_type=\"keras\")\n",
    "\n",
    "\n",
    "best_dnn, best_hyperparams, all_results = optimizer.fit(train_dataset=train_dataset, valid_dataset=valid_dataset)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T16:24:50.072370188Z",
     "start_time": "2023-06-29T16:18:47.646999614Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "In the end, we can check the performance of the best model on the test set."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "52/52 [==============================] - 0s 586us/step\n"
     ]
    },
    {
     "data": {
      "text/plain": "({'accuracy_score': 0.9825616355983163}, {})"
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_dnn.evaluate(test_dataset, metrics = [Metric(accuracy_score)])"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T16:27:38.335198225Z",
     "start_time": "2023-06-29T16:27:38.234518351Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can also check the best combination of hyperparameters found."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "outputs": [
    {
     "data": {
      "text/plain": "{'input_dim': 2048, 'dropout': 0.5, 'optimizer': 'adam'}"
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_hyperparams"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T16:27:40.061029636Z",
     "start_time": "2023-06-29T16:27:40.057185286Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can also check the performance of all the models trained during the hyperparameter tuning. Each model is defined by the name of the parameters followed by the value of the parameter."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "outputs": [
    {
     "data": {
      "text/plain": "{'_dropout_0.500000_input_dim_2048_optimizer_adam': 0.98014440433213,\n '_dropout_0.700000_input_dim_2048_optimizer_adam': 0.9777376654632972}"
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_results"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T16:27:46.751790476Z",
     "start_time": "2023-06-29T16:27:46.745998973Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-06-29 17:27:56,111 — INFO — MODEL TYPE: keras\n",
      "2023-06-29 17:27:56,112 — INFO — Fitting 2 random models from a space of 3 possible models.\n",
      "278/278 [==============================] - 1s 1ms/step - loss: 0.1939 - accuracy: 0.9261\n",
      "139/139 [==============================] - 0s 557us/step\n",
      "278/278 [==============================] - 1s 1ms/step - loss: 0.1911 - accuracy: 0.9294\n",
      "139/139 [==============================] - 0s 552us/step\n",
      "278/278 [==============================] - 1s 1ms/step - loss: 0.1807 - accuracy: 0.9350\n",
      "139/139 [==============================] - 0s 560us/step\n",
      "278/278 [==============================] - 1s 1ms/step - loss: 0.2029 - accuracy: 0.9290\n",
      "139/139 [==============================] - 0s 571us/step\n",
      "278/278 [==============================] - 1s 1ms/step - loss: 0.2123 - accuracy: 0.9208\n",
      "139/139 [==============================] - 0s 590us/step\n",
      "278/278 [==============================] - 1s 1ms/step - loss: 0.2133 - accuracy: 0.9192\n",
      "139/139 [==============================] - 0s 566us/step\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-06-29 17:28:02.068092: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 108937216 exceeds 10% of free system memory.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "416/416 [==============================] - 1s 1ms/step - loss: 0.1524 - accuracy: 0.9462\n",
      "2023-06-29 17:28:02,977 — INFO — \n",
      " \n",
      " Best <function accuracy_score at 0x7f55eba87b80>: 0.974808 using {'optimizer': 'adam', 'input_dim': 2048, 'dropout': 0.5}\n",
      "2023-06-29 17:28:02,978 — INFO — \n",
      " <function accuracy_score at 0x7f55eba87b80>: 0.974808 (0.001972) with: {'optimizer': 'adam', 'input_dim': 2048, 'dropout': 0.5} \n",
      "\n",
      "2023-06-29 17:28:02,978 — INFO — \n",
      " <function accuracy_score at 0x7f55eba87b80>: 0.973605 (0.000641) with: {'optimizer': 'adam', 'input_dim': 2048, 'dropout': 0.6} \n",
      "\n",
      "2023-06-29 17:28:02,979 — INFO — Fitting best model!\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-06-29 17:28:03.016279: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 108937216 exceeds 10% of free system memory.\n"
     ]
    }
   ],
   "source": [
    "optimizer = HyperparameterOptimizerCV(create_model,\n",
    "                                      metric=Metric(accuracy_score),\n",
    "                                      maximize_metric=True,\n",
    "                                      cv=3,\n",
    "                                      n_iter_search=2,\n",
    "                                      params_dict=params_dict_dense,\n",
    "                                      model_type=\"keras\")\n",
    "params_dict_dense = {\n",
    "                   \"input_dim\": [train_dataset.X.shape[1]],\n",
    "                   \"dropout\": [0.5, 0.6, 0.7],\n",
    "                   \"optimizer\": [\"adam\"]\n",
    "                   }\n",
    "best_dnn, best_hyperparams, all_results = optimizer.fit(train_dataset=train_dataset)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T16:31:00.899203318Z",
     "start_time": "2023-06-29T16:27:56.113056563Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Finally, save your best model for deployment and new predictions!"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "outputs": [],
   "source": [
    "best_dnn.save(\"my_model\")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T16:32:32.910511495Z",
     "start_time": "2023-06-29T16:32:32.889771Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Bring it back to life and make predictions!"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "outputs": [],
   "source": [
    "from deepmol.models import KerasModel\n",
    "\n",
    "best_dnn = KerasModel.load(\"my_model\")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T16:32:34.625379527Z",
     "start_time": "2023-06-29T16:32:34.578023390Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "52/52 [==============================] - 0s 589us/step\n"
     ]
    },
    {
     "data": {
      "text/plain": "array([0, 0, 0, ..., 1, 0, 0])"
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_dnn.predict(test_dataset)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T16:32:36.175211876Z",
     "start_time": "2023-06-29T16:32:36.065747314Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Hyperparameter tuning with deepchem models"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "DeepMol provide methods to perform hyperparameter tuning with deepchem models. The hyperparameter tuning can be performed with a validation set, previously created, or with cross validation. Anyway, the hyperparameter tuning is performed with a random search if the number of iterations is specified, otherwise a grid search is performed."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "As explained in the models section, to create a DeepChem model one have to define a function with the model architecture and the parameters to optimize. It is MANDATORY to pass the **model_type** parameter to the model.\n",
    "\n",
    "Let's see how to perform hyperparameter tuning of a GraphConvModel with a validation set."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "First, we have to featurize the dataset. These features are specific for the model we want to train. In this case, we want to train a GraphConvModel, so we have to featurize the dataset with ConvMolFeat.\n",
    "For more documentation on this matter check the [DeepChem documentation](https://deepchem.readthedocs.io/en/latest/api_reference/featurizers.html)."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "outputs": [],
   "source": [
    "from deepmol.compound_featurization import ConvMolFeat\n",
    "\n",
    "ConvMolFeat().featurize(train_dataset, inplace=True)\n",
    "ConvMolFeat().featurize(valid_dataset, inplace=True)\n",
    "ConvMolFeat().featurize(test_dataset, inplace=True)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T16:32:46.696292947Z",
     "start_time": "2023-06-29T16:32:39.836872519Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-06-29 17:32:49,453 — INFO — Fitting 2 random models from a space of 2 possible models.\n",
      "2023-06-29 17:32:49,454 — INFO — Fitting model 1/2\n",
      "2023-06-29 17:32:49,455 — INFO — hyperparameters: {'graph_conv_layers': [64, 64]}\n",
      "2023-06-29 17:33:14,791 — INFO — Model 1/2, Metric accuracy_score, Validation set 1: 0.951264\n",
      "2023-06-29 17:33:14,792 — INFO — \tbest_validation_score so far: 0.951264\n",
      "2023-06-29 17:33:14,792 — INFO — Fitting model 2/2\n",
      "2023-06-29 17:33:14,792 — INFO — hyperparameters: {'graph_conv_layers': [32, 32]}\n",
      "WARNING:tensorflow:5 out of the last 220 calls to <function KerasModel._create_gradient_fn.<locals>.apply_gradient_for_batch at 0x7f54ec52a040> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.\n",
      "2023-06-29 17:33:36,465 — INFO — Model 2/2, Metric accuracy_score, Validation set 2: 0.943442\n",
      "2023-06-29 17:33:36,466 — INFO — \tbest_validation_score so far: 0.951264\n",
      "2023-06-29 17:33:37,373 — INFO — Best hyperparameters: {'graph_conv_layers': [64, 64]}\n",
      "2023-06-29 17:33:37,373 — INFO — train_score: 0.955106\n",
      "2023-06-29 17:33:37,374 — INFO — validation_score: 0.951264\n"
     ]
    }
   ],
   "source": [
    "from deepmol.parameter_optimization import HyperparameterOptimizerValidation\n",
    "from deepmol.parameter_optimization import HyperparameterOptimizerCV\n",
    "\n",
    "from sklearn.metrics import roc_auc_score, precision_score\n",
    "from deepmol.models import DeepChemModel\n",
    "from deepchem.models import GraphConvModel\n",
    "\n",
    "def graphconv_builder(graph_conv_layers, batch_size=256, epochs=5):\n",
    "    graph = GraphConvModel(n_tasks=1, graph_conv_layers=graph_conv_layers, batch_size=batch_size,\n",
    "                           mode='classification')\n",
    "    return DeepChemModel(graph, epochs=epochs)\n",
    "\n",
    "model_graph = HyperparameterOptimizerValidation(model_builder=graphconv_builder,\n",
    "                                                metric=Metric(accuracy_score),\n",
    "                                                maximize_metric=True,\n",
    "                                                n_iter_search=2,\n",
    "                                                params_dict={'graph_conv_layers': [[64, 64], [32, 32]]},\n",
    "                                                model_type=\"deepchem\")\n",
    "\n",
    "best_model, best_hyperparams, all_results = model_graph.fit(train_dataset=train_dataset,valid_dataset=valid_dataset)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-29T16:33:37.377095126Z",
     "start_time": "2023-06-29T16:32:49.454861712Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "In the end, we can check the performance of the best model on the test set."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "outputs": [
    {
     "data": {
      "text/plain": "({'accuracy_score': 0.9428743235117258}, {})"
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_model.evaluate(test_dataset, metrics = [Metric(accuracy_score)])"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-02T10:38:57.942235200Z",
     "start_time": "2023-06-02T10:38:57.855864029Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can also check the best combination of hyperparameters found."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "outputs": [
    {
     "data": {
      "text/plain": "{'graph_conv_layers': [64, 64]}"
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_hyperparams"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-02T10:39:01.753659620Z",
     "start_time": "2023-06-02T10:39:01.743171073Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can also check the performance of all the models trained during the hyperparameter tuning. Each model is defined by the name of the parameters followed by the value of the parameter."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "outputs": [
    {
     "data": {
      "text/plain": "{'_graph_conv_layers_[64, 64]': 0.9494584837545126,\n '_graph_conv_layers_[32, 32]': 0.9199759326113117}"
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_results"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-02T10:39:06.399971211Z",
     "start_time": "2023-06-02T10:39:06.357539431Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Finally, save your best model for deployment and new predictions!"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "outputs": [],
   "source": [
    "best_model.save(\"my_model\")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-02T10:39:17.587723495Z",
     "start_time": "2023-06-02T10:39:17.553320485Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Bring it back to life and make predictions!"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "outputs": [],
   "source": [
    "best_model = DeepChemModel.load(\"my_model\")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-02T10:39:21.399279245Z",
     "start_time": "2023-06-02T10:39:21.377246729Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "outputs": [
    {
     "data": {
      "text/plain": "array([[0.06773414, 0.9322658 ],\n       [0.08101802, 0.9189819 ],\n       [0.10755827, 0.8924417 ],\n       ...,\n       [0.00282426, 0.9971757 ],\n       [0.00113406, 0.99886596],\n       [0.98225605, 0.01774395]], dtype=float32)"
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_model.predict(test_dataset)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-02T10:39:25.820561636Z",
     "start_time": "2023-06-02T10:39:24.849099565Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Now, let's try hyperparameter tuning with deepchem models and cross validation"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "def graphconv_builder(graph_conv_layers, batch_size=256, epochs=5):\n",
    "    graph = GraphConvModel(n_tasks=1, graph_conv_layers=graph_conv_layers, batch_size=batch_size,\n",
    "                           mode='classification')\n",
    "    return DeepChemModel(graph, epochs=epochs)\n",
    "\n",
    "model_graph = HyperparameterOptimizerCV(model_builder=graphconv_builder,\n",
    "                                        metric=Metric(roc_auc_score),\n",
    "                                        n_iter_search=2,\n",
    "                                        maximize_metric=True,\n",
    "                                        cv = 2,\n",
    "                                        params_dict={'graph_conv_layers': [[64, 64], [32, 32]]},\n",
    "                                        model_type=\"deepchem\")\n",
    "\n",
    "best_model, best_hyperparameters, all_results = model_graph.fit(train_dataset=train_dataset)\n"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "In the end, we can check the performance of the best model on the test set."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "outputs": [
    {
     "data": {
      "text/plain": "({'roc_auc_score': 0.9905489702285326,\n  'precision_score': 0.9622166246851386,\n  'accuracy_score': 0.9542994588093806},\n {})"
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_preds = best_model.predict(test_dataset)\n",
    "\n",
    "metrics = [Metric(roc_auc_score), Metric(precision_score), Metric(accuracy_score)]\n",
    "\n",
    "best_model.evaluate(test_dataset, metrics)\n"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-02T10:43:46.673286175Z",
     "start_time": "2023-06-02T10:43:45.825683971Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can also check the best combination of hyperparameters found."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "outputs": [
    {
     "data": {
      "text/plain": "{'graph_conv_layers': [64, 64]}"
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_hyperparameters"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-02T10:44:03.896968799Z",
     "start_time": "2023-06-02T10:44:03.831112713Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can also check the performance of all the models trained during the hyperparameter tuning. Each model is defined by the name of the parameters followed by the value of the parameter."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "outputs": [
    {
     "data": {
      "text/plain": "defaultdict(list,\n            {'params': [{'graph_conv_layers': [64, 64]},\n              {'graph_conv_layers': [32, 32]}],\n             'mean_train_score': [0.9795196072718013, 0.980248484393072],\n             'mean_test_score': [0.9758974414293784, 0.9754465835745805],\n             'std_train_score': [0.003408778455179895, 0.000607063345097747],\n             'std_test_score': [0.0029443151372258725, 0.0011259723129122823],\n             'split0_train_score': [0.9829283857269812, 0.9796414210479742],\n             'split0_test_score': [0.9788417565666043, 0.9743206112616681],\n             'split1_train_score': [0.9761108288166214, 0.9808555477381697],\n             'split1_test_score': [0.9729531262921526, 0.9765725558874927]})"
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_results"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-02T12:50:26.567094777Z",
     "start_time": "2023-06-02T12:50:26.556995348Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "outputs": [
    {
     "data": {
      "text/plain": "[{'graph_conv_layers': [64, 64]}, {'graph_conv_layers': [32, 32]}]"
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_results[\"params\"]"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-02T12:50:28.567960975Z",
     "start_time": "2023-06-02T12:50:28.558430169Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Finally, save your best model for deployment and new predictions!"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "outputs": [],
   "source": [
    "best_model.save(\"my_model\")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-02T10:44:12.508470582Z",
     "start_time": "2023-06-02T10:44:12.474032585Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Bring it back to life and make predictions!"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "outputs": [],
   "source": [
    "best_model = DeepChemModel.load(\"my_model\")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-02T10:44:13.321022677Z",
     "start_time": "2023-06-02T10:44:13.301040420Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "outputs": [
    {
     "data": {
      "text/plain": "array([[0.04331122, 0.95668876],\n       [0.7563593 , 0.24364069],\n       [0.00733165, 0.9926683 ],\n       ...,\n       [0.0438629 , 0.9561371 ],\n       [0.06368961, 0.9363104 ],\n       [0.9964748 , 0.00352522]], dtype=float32)"
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_model.predict(test_dataset)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-06-02T10:44:15.021993198Z",
     "start_time": "2023-06-02T10:44:14.024687085Z"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}