kaggle_comp_llm_exam_new-dataset-deberta-v3-large-training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cb2af121",
   "metadata": {
    "papermill": {
     "duration": 0.007095,
     "end_time": "2023-07-23T04:28:26.171953",
     "exception": false,
     "start_time": "2023-07-23T04:28:26.164858",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "\n",
    "200 examples in the public dataset leaves very little room for training!\n",
    "\n",
    "Using `gpt-3.5-turbo` I created another 500 high quality examples at my expense [that I share freely here](https://www.kaggle.com/datasets/radek1/additional-train-data-for-llm-science-exam). They are what (as of now) pushes this notebook to the highest achieved score across the public notebooks (`0.723`)!\n",
    "\n",
    "If you find the additional training examples useful, please upvote the dataset! 😊\n",
    "\n",
    "👉 [additional train data for LLM Science Exam 🥳](https://www.kaggle.com/datasets/radek1/additional-train-data-for-llm-science-exam)\n",
    "\n",
    "Thank you! Appreciate your help! 🙏🙏🙏\n",
    "\n",
    "This notebook builds on a [notebook](https://www.kaggle.com/code/leonidkulyk/lb-0-709-llm-se-deberta-v3-large-i-1k-wiki) by LEONID KULYK. Among some of the changes are:\n",
    "\n",
    "* use of a new high quality dataset for training\n",
    "* modified training procedure carried out in the notebook\n",
    "* general streamlining of code/training for readability"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de38f065",
   "metadata": {
    "papermill": {
     "duration": 0.008859,
     "end_time": "2023-07-23T04:28:26.187582",
     "exception": false,
     "start_time": "2023-07-23T04:28:26.178723",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Preprocessing the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ce257ae3",
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2023-07-23T04:28:26.203082Z",
     "iopub.status.busy": "2023-07-23T04:28:26.202032Z",
     "iopub.status.idle": "2023-07-23T04:28:39.825376Z",
     "shell.execute_reply": "2023-07-23T04:28:39.824418Z"
    },
    "papermill": {
     "duration": 13.633747,
     "end_time": "2023-07-23T04:28:39.827844",
     "exception": false,
     "start_time": "2023-07-23T04:28:26.194097",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.23.5\n",
      "  warnings.warn(f\"A NumPy version >={np_minversion} and <{np_maxversion}\"\n",
      "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/__init__.py:98: UserWarning: unable to load libtensorflow_io_plugins.so: unable to open file: libtensorflow_io_plugins.so, from paths: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io_plugins.so']\n",
      "caused by: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io_plugins.so: undefined symbol: _ZN3tsl6StatusC1EN10tensorflow5error4CodeESt17basic_string_viewIcSt11char_traitsIcEENS_14SourceLocationE']\n",
      "  warnings.warn(f\"unable to load libtensorflow_io_plugins.so: {e}\")\n",
      "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/__init__.py:104: UserWarning: file system plugins are not loaded: unable to open file: libtensorflow_io.so, from paths: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io.so']\n",
      "caused by: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io.so: undefined symbol: _ZTVN10tensorflow13GcsFileSystemE']\n",
      "  warnings.warn(f\"file system plugins are not loaded: {e}\")\n"
     ]
    }
   ],
   "source": [
    "from typing import Optional, Union\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import torch\n",
    "from datasets import Dataset\n",
    "from dataclasses import dataclass\n",
    "from transformers import AutoTokenizer\n",
    "from transformers.tokenization_utils_base import PreTrainedTokenizerBase, PaddingStrategy\n",
    "from transformers import AutoModelForMultipleChoice, TrainingArguments, Trainer, AutoModel\n",
    "\n",
    "deberta_v3_large = '/kaggle/input/deberta-v3-large-hf-weights'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51a49221",
   "metadata": {
    "papermill": {
     "duration": 0.006663,
     "end_time": "2023-07-23T04:28:39.841901",
     "exception": false,
     "start_time": "2023-07-23T04:28:39.835238",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We begin by loading and processing the train data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f2d92e48",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-07-23T04:28:39.856798Z",
     "iopub.status.busy": "2023-07-23T04:28:39.856453Z",
     "iopub.status.idle": "2023-07-23T04:28:39.884142Z",
     "shell.execute_reply": "2023-07-23T04:28:39.883247Z"
    },
    "papermill": {
     "duration": 0.037673,
     "end_time": "2023-07-23T04:28:39.886397",
     "exception": false,
     "start_time": "2023-07-23T04:28:39.848724",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(200, 7)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train = pd.read_csv('/kaggle/input/kaggle-llm-science-exam/train.csv')\n",
    "df_train = df_train.drop(columns=\"id\")\n",
    "df_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34465887",
   "metadata": {
    "papermill": {
     "duration": 0.006768,
     "end_time": "2023-07-23T04:28:39.900970",
     "exception": false,
     "start_time": "2023-07-23T04:28:39.894202",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Let's add another 500 examples to the train set!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8219ad7e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-07-23T04:28:39.915935Z",
     "iopub.status.busy": "2023-07-23T04:28:39.915642Z",
     "iopub.status.idle": "2023-07-23T04:28:39.936159Z",
     "shell.execute_reply": "2023-07-23T04:28:39.935017Z"
    },
    "papermill": {
     "duration": 0.030803,
     "end_time": "2023-07-23T04:28:39.938606",
     "exception": false,
     "start_time": "2023-07-23T04:28:39.907803",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(700, 7)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train = pd.concat([\n",
    "    df_train,\n",
    "    pd.read_csv('/kaggle/input/additional-train-data-for-llm-science-exam/extra_train_set.csv'),\n",
    "])\n",
    "df_train.reset_index(inplace=True, drop=True)\n",
    "df_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb8d54db",
   "metadata": {
    "papermill": {
     "duration": 0.006843,
     "end_time": "2023-07-23T04:28:39.953200",
     "exception": false,
     "start_time": "2023-07-23T04:28:39.946357",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Now that we have gone from 200 -> 700 train examples, let us preprocess the data and begin training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d0c5bc40",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-07-23T04:28:39.969318Z",
     "iopub.status.busy": "2023-07-23T04:28:39.968395Z",
     "iopub.status.idle": "2023-07-23T04:28:39.981983Z",
     "shell.execute_reply": "2023-07-23T04:28:39.981120Z"
    },
    "papermill": {
     "duration": 0.023667,
     "end_time": "2023-07-23T04:28:39.984061",
     "exception": false,
     "start_time": "2023-07-23T04:28:39.960394",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "option_to_index = {option: idx for idx, option in enumerate('ABCDE')}\n",
    "index_to_option = {v: k for k,v in option_to_index.items()}\n",
    "\n",
    "def preprocess(example):\n",
    "    first_sentence = [example['prompt']] * 5\n",
    "    second_sentences = [example[option] for option in 'ABCDE']\n",
    "    tokenized_example = tokenizer(first_sentence, second_sentences, truncation=True)\n",
    "    tokenized_example['label'] = option_to_index[example['answer']]\n",
    "    \n",
    "    return tokenized_example\n",
    "\n",
    "@dataclass\n",
    "class DataCollatorForMultipleChoice:\n",
    "    tokenizer: PreTrainedTokenizerBase\n",
    "    padding: Union[bool, str, PaddingStrategy] = True\n",
    "    max_length: Optional[int] = None\n",
    "    pad_to_multiple_of: Optional[int] = None\n",
    "    \n",
    "    def __call__(self, features):\n",
    "        label_name = 'label' if 'label' in features[0].keys() else 'labels'\n",
    "        labels = [feature.pop(label_name) for feature in features]\n",
    "        batch_size = len(features)\n",
    "        num_choices = len(features[0]['input_ids'])\n",
    "        flattened_features = [\n",
    "            [{k: v[i] for k, v in feature.items()} for i in range(num_choices)] for feature in features\n",
    "        ]\n",
    "        flattened_features = sum(flattened_features, [])\n",
    "        \n",
    "        batch = self.tokenizer.pad(\n",
    "            flattened_features,\n",
    "            padding=self.padding,\n",
    "            max_length=self.max_length,\n",
    "            pad_to_multiple_of=self.pad_to_multiple_of,\n",
    "            return_tensors='pt',\n",
    "        )\n",
    "        batch = {k: v.view(batch_size, num_choices, -1) for k, v in batch.items()}\n",
    "        batch['labels'] = torch.tensor(labels, dtype=torch.int64)\n",
    "        return batch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ed6e127",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-07-21T03:38:13.805215Z",
     "iopub.status.busy": "2023-07-21T03:38:13.804849Z",
     "iopub.status.idle": "2023-07-21T03:38:13.814629Z",
     "shell.execute_reply": "2023-07-21T03:38:13.813302Z",
     "shell.execute_reply.started": "2023-07-21T03:38:13.805184Z"
    },
    "papermill": {
     "duration": 0.006882,
     "end_time": "2023-07-23T04:28:39.998047",
     "exception": false,
     "start_time": "2023-07-23T04:28:39.991165",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We first create a HuggingFace `Dataset`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "41b1348a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-07-23T04:28:40.014002Z",
     "iopub.status.busy": "2023-07-23T04:28:40.013174Z",
     "iopub.status.idle": "2023-07-23T04:28:41.205491Z",
     "shell.execute_reply": "2023-07-23T04:28:41.204562Z"
    },
    "papermill": {
     "duration": 1.202444,
     "end_time": "2023-07-23T04:28:41.207634",
     "exception": false,
     "start_time": "2023-07-23T04:28:40.005190",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n",
      "/opt/conda/lib/python3.10/site-packages/transformers/convert_slow_tokenizer.py:454: UserWarning: The sentencepiece tokenizer that you are converting to a fast tokenizer uses the byte fallback option which is not implemented in the fast tokenizers. In practice this means that the fast version of the tokenizer can produce unknown tokens whereas the sentencepiece version would have converted these unknown tokens into a sequence of byte tokens matching the original piece of text.\n",
      "  warnings.warn(\n",
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['prompt', 'A', 'B', 'C', 'D', 'E', 'answer'],\n",
       "    num_rows: 700\n",
       "})"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(deberta_v3_large)\n",
    "\n",
    "dataset = Dataset.from_pandas(df_train)\n",
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "930db860",
   "metadata": {
    "papermill": {
     "duration": 0.007303,
     "end_time": "2023-07-23T04:28:41.222519",
     "exception": false,
     "start_time": "2023-07-23T04:28:41.215216",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "And let us now preprocess the examples for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ff0dc6cd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-07-23T04:28:41.238954Z",
     "iopub.status.busy": "2023-07-23T04:28:41.238662Z",
     "iopub.status.idle": "2023-07-23T04:28:42.107505Z",
     "shell.execute_reply": "2023-07-23T04:28:42.106466Z"
    },
    "papermill": {
     "duration": 0.879604,
     "end_time": "2023-07-23T04:28:42.109718",
     "exception": false,
     "start_time": "2023-07-23T04:28:41.230114",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "95d2ea3f21464d4b835cc0863315b61c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/700 [00:00<?, ?ex/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['input_ids', 'token_type_ids', 'attention_mask', 'label'],\n",
       "    num_rows: 700\n",
       "})"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenized_dataset = dataset.map(preprocess, remove_columns=['prompt', 'A', 'B', 'C', 'D', 'E', 'answer'])\n",
    "tokenized_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87a89fd9",
   "metadata": {
    "papermill": {
     "duration": 0.007618,
     "end_time": "2023-07-23T04:28:42.125222",
     "exception": false,
     "start_time": "2023-07-23T04:28:42.117604",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "3ea3c71f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-07-23T04:28:42.142252Z",
     "iopub.status.busy": "2023-07-23T04:28:42.141677Z",
     "iopub.status.idle": "2023-07-23T04:38:45.338092Z",
     "shell.execute_reply": "2023-07-23T04:38:45.337212Z"
    },
    "papermill": {
     "duration": 603.20716,
     "end_time": "2023-07-23T04:38:45.340060",
     "exception": false,
     "start_time": "2023-07-23T04:28:42.132900",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at /kaggle/input/deberta-v3-large-hf-weights were not used when initializing DebertaV2ForMultipleChoice: ['lm_predictions.lm_head.LayerNorm.bias', 'mask_predictions.classifier.weight', 'mask_predictions.classifier.bias', 'mask_predictions.dense.weight', 'mask_predictions.LayerNorm.weight', 'mask_predictions.LayerNorm.bias', 'lm_predictions.lm_head.dense.weight', 'lm_predictions.lm_head.dense.bias', 'lm_predictions.lm_head.bias', 'mask_predictions.dense.bias', 'lm_predictions.lm_head.LayerNorm.weight']\n",
      "- This IS expected if you are initializing DebertaV2ForMultipleChoice from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DebertaV2ForMultipleChoice from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DebertaV2ForMultipleChoice were not initialized from the model checkpoint at /kaggle/input/deberta-v3-large-hf-weights and are newly initialized: ['classifier.weight', 'classifier.bias', 'pooler.dense.bias', 'pooler.dense.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "/opt/conda/lib/python3.10/site-packages/transformers/optimization.py:411: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "You're using a DebertaV2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='2100' max='2100' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [2100/2100 09:44, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>500</td>\n",
       "      <td>1.616200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1000</td>\n",
       "      <td>1.600700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1500</td>\n",
       "      <td>1.356700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2000</td>\n",
       "      <td>1.096100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=2100, training_loss=1.3949325597853888, metrics={'train_runtime': 586.2184, 'train_samples_per_second': 3.582, 'train_steps_per_second': 3.582, 'total_flos': 823233082746960.0, 'train_loss': 1.3949325597853888, 'epoch': 3.0})"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_args = TrainingArguments(\n",
    "    warmup_ratio=0.8,\n",
    "    learning_rate=5e-6,\n",
    "    per_device_train_batch_size=1,\n",
    "    per_device_eval_batch_size=2,\n",
    "    num_train_epochs=3,\n",
    "    report_to='none',\n",
    "    output_dir='.'\n",
    ")\n",
    "\n",
    "model = AutoModelForMultipleChoice.from_pretrained(deberta_v3_large)\n",
    "\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    tokenizer=tokenizer,\n",
    "    data_collator=DataCollatorForMultipleChoice(tokenizer=tokenizer),\n",
    "    train_dataset=tokenized_dataset,\n",
    ")\n",
    "\n",
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dd42c85",
   "metadata": {
    "papermill": {
     "duration": 0.008385,
     "end_time": "2023-07-23T04:38:45.357078",
     "exception": false,
     "start_time": "2023-07-23T04:38:45.348693",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Predicting on the test set"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69c7225a",
   "metadata": {
    "papermill": {
     "duration": 0.008284,
     "end_time": "2023-07-23T04:38:45.374944",
     "exception": false,
     "start_time": "2023-07-23T04:38:45.366660",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Now that we have trained our model, let us predict on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0344e380",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-07-23T04:38:45.393877Z",
     "iopub.status.busy": "2023-07-23T04:38:45.392897Z",
     "iopub.status.idle": "2023-07-23T04:38:45.676885Z",
     "shell.execute_reply": "2023-07-23T04:38:45.675937Z"
    },
    "papermill": {
     "duration": 0.296002,
     "end_time": "2023-07-23T04:38:45.679206",
     "exception": false,
     "start_time": "2023-07-23T04:38:45.383204",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b5240cec750244c881ad4339feb44be6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/200 [00:00<?, ?ex/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_df = pd.read_csv('/kaggle/input/kaggle-llm-science-exam/test.csv')\n",
    "test_df['answer'] = 'A' # dummy answer that allows us to preprocess the test dataset just like we preprocessed the train dataset\n",
    "\n",
    "tokenized_test_dataset = Dataset.from_pandas(test_df.drop(columns=['id'])).map(preprocess, remove_columns=['prompt', 'A', 'B', 'C', 'D', 'E'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9e069960",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-07-23T04:38:45.698164Z",
     "iopub.status.busy": "2023-07-23T04:38:45.697873Z",
     "iopub.status.idle": "2023-07-23T04:38:55.994864Z",
     "shell.execute_reply": "2023-07-23T04:38:55.993870Z"
    },
    "papermill": {
     "duration": 10.309536,
     "end_time": "2023-07-23T04:38:55.997395",
     "exception": false,
     "start_time": "2023-07-23T04:38:45.687859",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "array([[-5.919389 , -3.1346128, -5.882382 ,  1.061752 , -5.349337 ],\n",
       "       [ 1.0768496,  0.6444651, -3.657922 , -4.1297135, -2.3705304],\n",
       "       [ 2.539788 , -3.852208 ,  0.4875411, -5.173388 , -4.1340165],\n",
       "       [-5.267994 , -4.3641067,  1.7363403, -3.7330499, -4.3782735]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_predictions = trainer.predict(tokenized_test_dataset).predictions\n",
    "test_predictions[:4]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fed43b0",
   "metadata": {
    "papermill": {
     "duration": 0.008532,
     "end_time": "2023-07-23T04:38:56.014902",
     "exception": false,
     "start_time": "2023-07-23T04:38:56.006370",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The predictions are values output from the last layer of our neural network.\n",
    "\n",
    "Let's obtain the predicted answer ids by sorting them from largest to the smallest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "74dab8e7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-07-23T04:38:56.033930Z",
     "iopub.status.busy": "2023-07-23T04:38:56.033634Z",
     "iopub.status.idle": "2023-07-23T04:38:56.042389Z",
     "shell.execute_reply": "2023-07-23T04:38:56.041402Z"
    },
    "papermill": {
     "duration": 0.020524,
     "end_time": "2023-07-23T04:38:56.044391",
     "exception": false,
     "start_time": "2023-07-23T04:38:56.023867",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[3, 1, 4, 2, 0],\n",
       "       [0, 1, 4, 2, 3],\n",
       "       [0, 2, 1, 4, 3]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions_as_ids = np.argsort(-test_predictions, 1)\n",
    "predictions_as_ids[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80a52e06",
   "metadata": {
    "papermill": {
     "duration": 0.008845,
     "end_time": "2023-07-23T04:38:56.061962",
     "exception": false,
     "start_time": "2023-07-23T04:38:56.053117",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Let us now assign a letter corresponding to each predicted id (0 -> 'A', 1 -> 'B', etc). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e58aab9c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-07-23T04:38:56.081516Z",
     "iopub.status.busy": "2023-07-23T04:38:56.080869Z",
     "iopub.status.idle": "2023-07-23T04:38:56.088431Z",
     "shell.execute_reply": "2023-07-23T04:38:56.087487Z"
    },
    "papermill": {
     "duration": 0.019781,
     "end_time": "2023-07-23T04:38:56.090430",
     "exception": false,
     "start_time": "2023-07-23T04:38:56.070649",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([['D', 'B', 'E', 'C', 'A'],\n",
       "       ['A', 'B', 'E', 'C', 'D'],\n",
       "       ['A', 'C', 'B', 'E', 'D']], dtype='<U1')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions_as_answer_letters = np.array(list('ABCDE'))[predictions_as_ids]\n",
    "predictions_as_answer_letters[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a87caf1",
   "metadata": {
    "papermill": {
     "duration": 0.008772,
     "end_time": "2023-07-23T04:38:56.108329",
     "exception": false,
     "start_time": "2023-07-23T04:38:56.099557",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "And let us now go from this representation to outputting a string with 3 highest rated answers seperated by a space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "febec6ec",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-07-23T04:38:56.127923Z",
     "iopub.status.busy": "2023-07-23T04:38:56.127147Z",
     "iopub.status.idle": "2023-07-23T04:38:56.135357Z",
     "shell.execute_reply": "2023-07-23T04:38:56.134377Z"
    },
    "papermill": {
     "duration": 0.020215,
     "end_time": "2023-07-23T04:38:56.137526",
     "exception": false,
     "start_time": "2023-07-23T04:38:56.117311",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['D B E', 'A B E', 'A C B']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions_as_string = test_df['prediction'] = [\n",
    "    ' '.join(row) for row in predictions_as_answer_letters[:, :3]\n",
    "]\n",
    "predictions_as_string[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "879255ee",
   "metadata": {
    "papermill": {
     "duration": 0.008796,
     "end_time": "2023-07-23T04:38:56.155397",
     "exception": false,
     "start_time": "2023-07-23T04:38:56.146601",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "And we are done! 🥳\n",
    "\n",
    "Let us now output our submission."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a5b216fb",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-07-23T04:38:56.175390Z",
     "iopub.status.busy": "2023-07-23T04:38:56.174686Z",
     "iopub.status.idle": "2023-07-23T04:38:56.196725Z",
     "shell.execute_reply": "2023-07-23T04:38:56.195010Z"
    },
    "papermill": {
     "duration": 0.035019,
     "end_time": "2023-07-23T04:38:56.199664",
     "exception": false,
     "start_time": "2023-07-23T04:38:56.164645",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>prediction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>D B E</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>A B E</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>A C B</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>C D B</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>D A B</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id prediction\n",
       "0   0      D B E\n",
       "1   1      A B E\n",
       "2   2      A C B\n",
       "3   3      C D B\n",
       "4   4      D A B"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "submission = test_df[['id', 'prediction']]\n",
    "submission.to_csv('submission.csv', index=False)\n",
    "\n",
    "pd.read_csv('submission.csv').head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f70c0bd9",
   "metadata": {
    "papermill": {
     "duration": 0.009309,
     "end_time": "2023-07-23T04:38:56.218498",
     "exception": false,
     "start_time": "2023-07-23T04:38:56.209189",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "I hope you enjoyed this notebook!\n",
    "\n",
    "If you found it useful, please upvote 👉 [the corresponding dataset with 500 new training examples](https://www.kaggle.com/datasets/radek1/additional-train-data-for-llm-science-exam) 👈\n",
    "\n",
    "Thank you, appreciate your help! 🙏😊\n",
    "\n",
    "Thank you for reading and happy Kaggling!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 643.013387,
   "end_time": "2023-07-23T04:38:59.273046",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2023-07-23T04:28:16.259659",
   "version": "2.4.0"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {
     "0d673ec0a2ce49cbaa81988f6148e0d0": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_e7cdd856023c4bcf8075c3918b3f7ddb",
       "placeholder": "​",
       "style": "IPY_MODEL_64d5560392bc4410ac7fe18c943f366e",
       "value": "100%"
      }
     },
     "13ea4663e38d49fbb3e196cab307b927": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "2edb03f15fed46ff84f6db6cee8eb603": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "44295248d6d34bc296921eb15fcccf19": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "528ac93e462c4c39982f95e4e84cd421": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": ""
      }
     },
     "53083cf851c44d6ba235822c3f1dfc73": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "574d276cd74845cc8a83315149126e24": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "5b3c2ba26d75437483fbae138f034151": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_53083cf851c44d6ba235822c3f1dfc73",
       "placeholder": "​",
       "style": "IPY_MODEL_13ea4663e38d49fbb3e196cab307b927",
       "value": " 700/700 [00:00&lt;00:00, 1028.31ex/s]"
      }
     },
     "5d23e0555c5941688693b95e1660739f": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "64d5560392bc4410ac7fe18c943f366e": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "65245958873443df8ceec387a968a74c": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_574d276cd74845cc8a83315149126e24",
       "placeholder": "​",
       "style": "IPY_MODEL_cb07eec068954790bf8f3b3830c6ba92",
       "value": "100%"
      }
     },
     "72c596bcf4a84eff8915a7b2881266fc": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "95d2ea3f21464d4b835cc0863315b61c": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_65245958873443df8ceec387a968a74c",
        "IPY_MODEL_9dad4120e41e47979ca652e1256fbf7d",
        "IPY_MODEL_5b3c2ba26d75437483fbae138f034151"
       ],
       "layout": "IPY_MODEL_b059dede90454843a1a363a13afac76c"
      }
     },
     "9a28293c1ad04ddbbfd24cf173a062d7": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_44295248d6d34bc296921eb15fcccf19",
       "placeholder": "​",
       "style": "IPY_MODEL_fee8f47d6ee94a369221916de6c9dbc0",
       "value": " 200/200 [00:00&lt;00:00, 973.95ex/s]"
      }
     },
     "9dad4120e41e47979ca652e1256fbf7d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_2edb03f15fed46ff84f6db6cee8eb603",
       "max": 700.0,
       "min": 0.0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_d3da942e19584c13b72c325fc7d182c5",
       "value": 700.0
      }
     },
     "b059dede90454843a1a363a13afac76c": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "b5240cec750244c881ad4339feb44be6": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_0d673ec0a2ce49cbaa81988f6148e0d0",
        "IPY_MODEL_c86e5813432446aa941ef67a83ea8226",
        "IPY_MODEL_9a28293c1ad04ddbbfd24cf173a062d7"
       ],
       "layout": "IPY_MODEL_5d23e0555c5941688693b95e1660739f"
      }
     },
     "c86e5813432446aa941ef67a83ea8226": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_72c596bcf4a84eff8915a7b2881266fc",
       "max": 200.0,
       "min": 0.0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_528ac93e462c4c39982f95e4e84cd421",
       "value": 200.0
      }
     },
     "cb07eec068954790bf8f3b3830c6ba92": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "d3da942e19584c13b72c325fc7d182c5": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": ""
      }
     },
     "e7cdd856023c4bcf8075c3918b3f7ddb": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "fee8f47d6ee94a369221916de6c9dbc0": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     }
    },
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}