evaluations_SG_SS_phase1+2_best2models_trial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Trial set evaluations for SS Phase 2, on best 2 models resulting from SG and SS step phase 1:\n",
    "This notebook corresponds to the results presented in section 4.3 of the thesis. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### For model 'roberta-base':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-06-15T16:49:40.922645Z",
     "start_time": "2023-06-15T16:49:36.941069Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "data, substitutes_df = get_data_and_create_empty_df()\n",
    "\n",
    "model = 'roberta-base'\n",
    "model_name_str = get_str_for_file_name(model)\n",
    "\n",
    "nlp, lm_tokenizer, lm_model, fill_mask = instantiate_spacy_tokenizer_model_pipeline(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-06-15T16:45:42.386920Z",
     "start_time": "2023-06-15T16:45:38.130694Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SS_phase2_option1_SharedSyns_robertabase exported to csv in path './predictions/trial/SS_phase2_option1_SharedSyns_robertabase.tsv'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Substitute Selection phase 2, option 1: substitutes that are synonyms of the complex word first (lemmatized substitutes that share the same synset as the lemmatized complex word)\n",
    "substitutes_df = substitute_selection_phase_2_option_1(data, substitutes_df, lm_tokenizer, fill_mask, model_name_str, nlp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "python tsar_eval.py --gold_file ./data/trial/tsar2022_en_trial_gold_no_noise.tsv --predictions_file ./predictions/trial/SS_phase2_option1_SharedSyns_robertabase.tsv --output_file ./output/trial/SS_phase2_option1_SharedSyns_robertabase.tsv"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "=========   EVALUATION config.=========\n",
    "GOLD file = ./data/trial/tsar2022_en_trial_gold_no_noise.tsv\n",
    "PREDICTION LABELS file = ./predictions/trial/SS_phase2_option1_SharedSyns_robertabase.tsv\n",
    "OUTPUT file = ./output/trial/SS_phase2_option1_SharedSyns_robertabase.tsv\n",
    "===============   RESULTS  =============\n",
    "\n",
    "MAP@1/Potential@1/Precision@1 = 0.4\n",
    "\n",
    "MAP@3 = 0.3388\n",
    "MAP@5 = 0.2323\n",
    "MAP@10 = 0.1504\n",
    "\n",
    "Potential@3 = 0.8\n",
    "Potential@5 = 1.0\n",
    "Potential@10 = 1.0\n",
    "\n",
    "Accuracy@1@top_gold_1 = 0.3\n",
    "Accuracy@2@top_gold_1 = 0.5\n",
    "Accuracy@3@top_gold_1 = 0.6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SS_phase2_option2_SharedHyper1_robertabase exported to csv in path './predictions/trial/SS_phase2_option2_SharedHyper1_robertabase.tsv'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Substitute Selection phase 2, option 2a: sort the substitutes that share their 1st level hypernyms with the complex word first: (retrieving the 1-level hypernyms for all possible synsets of the complex word and substitutes, and then compare these hypernyms to identify shared hypernyms between the complex word and substitutes. these will be prioritized in the resulting list).\n",
    "substitutes_df = substitute_selection_phase_2_option_2(data, substitutes_df, lm_tokenizer, fill_mask, model_name_str, nlp, levels=[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "python tsar_eval.py --gold_file ./data/trial/tsar2022_en_trial_gold_no_noise.tsv --predictions_file ./predictions/trial/SS_phase2_option2_SharedHyper1_robertabase.tsv --output_file ./output/trial/SS_phase2_option2_SharedHyper1_robertabase.tsv"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "=========   EVALUATION config.=========\n",
    "GOLD file = ./data/trial/tsar2022_en_trial_gold_no_noise.tsv\n",
    "PREDICTION LABELS file = ./predictions/trial/SS_phase2_option2_SharedHyper1_robertabase.tsv\n",
    "OUTPUT file = ./output/trial/SS_phase2_option2_SharedHyper1_robertabase.tsv\n",
    "===============   RESULTS  =============\n",
    "\n",
    "MAP@1/Potential@1/Precision@1 = 0.3\n",
    "\n",
    "MAP@3 = 0.2222\n",
    "MAP@5 = 0.1883\n",
    "MAP@10 = 0.1245\n",
    "\n",
    "Potential@3 = 0.7\n",
    "Potential@5 = 0.9\n",
    "Potential@10 = 1.0\n",
    "\n",
    "Accuracy@1@top_gold_1 = 0.2\n",
    "Accuracy@2@top_gold_1 = 0.4\n",
    "Accuracy@3@top_gold_1 = 0.5\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SS_phase2_option2_SharedHyper2_robertabase exported to csv in path './predictions/trial/SS_phase2_option2_SharedHyper2_robertabase.tsv'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Substitute Selection phase 2, option 2b: sort the substitutes that share their indirect hypernyms (2 levels up) with the complex word first\n",
    "substitutes_df = substitute_selection_phase_2_option_2(data, substitutes_df, lm_tokenizer, fill_mask, model_name_str, nlp, levels=[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "python tsar_eval.py --gold_file ./data/trial/tsar2022_en_trial_gold_no_noise.tsv --predictions_file ./predictions/trial/SS_phase2_option2_SharedHyper2_robertabase.tsv --output_file ./output/trial/SS_phase2_option2_SharedHyper2_robertabase.tsv"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "=========   EVALUATION config.=========\n",
    "GOLD file = ./data/trial/tsar2022_en_trial_gold_no_noise.tsv\n",
    "PREDICTION LABELS file = ./predictions/trial/SS_phase2_option2_SharedHyper2_robertabase.tsv\n",
    "OUTPUT file = ./output/trial/SS_phase2_option2_SharedHyper2_robertabase.tsv\n",
    "===============   RESULTS  =============\n",
    "\n",
    "MAP@1/Potential@1/Precision@1 = 0.5\n",
    "\n",
    "MAP@3 = 0.3277\n",
    "MAP@5 = 0.3046\n",
    "MAP@10 = 0.1692\n",
    "\n",
    "Potential@3 = 0.8\n",
    "Potential@5 = 1.0\n",
    "Potential@10 = 1.0\n",
    "\n",
    "Accuracy@1@top_gold_1 = 0.4\n",
    "Accuracy@2@top_gold_1 = 0.6\n",
    "Accuracy@3@top_gold_1 = 0.7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SS_phase2_option2_SharedHyper1_2_robertabase exported to csv in path './predictions/trial/SS_phase2_option2_SharedHyper1_2_robertabase.tsv'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Substitute Selection phase 2, option 2c: sort the substitutes that share either their direct (1 level up) or their indirect (2 levels up) hypernyms with the complex word first\n",
    "substitutes_df = substitute_selection_phase_2_option_2(data, substitutes_df, lm_tokenizer, fill_mask, model_name_str, nlp, levels=[1,2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "python tsar_eval.py --gold_file ./data/trial/tsar2022_en_trial_gold_no_noise.tsv --predictions_file ./predictions/trial/SS_phase2_option2_SharedHyper1_2_robertabase.tsv --output_file ./output/trial/SS_phase2_option2_SharedHyper1_2_robertabase.tsv"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "=========   EVALUATION config.=========\n",
    "GOLD file = ./data/trial/tsar2022_en_trial_gold_no_noise.tsv\n",
    "PREDICTION LABELS file = ./predictions/trial/SS_phase2_option2_SharedHyper1_2_robertabase.tsv\n",
    "OUTPUT file = ./output/trial/SS_phase2_option2_SharedHyper1_2_robertabase.tsv\n",
    "===============   RESULTS  =============\n",
    "\n",
    "MAP@1/Potential@1/Precision@1 = 0.4\n",
    "\n",
    "MAP@3 = 0.2722\n",
    "MAP@5 = 0.2363\n",
    "MAP@10 = 0.1466\n",
    "\n",
    "Potential@3 = 0.7\n",
    "Potential@5 = 0.9\n",
    "Potential@10 = 1.0\n",
    "\n",
    "Accuracy@1@top_gold_1 = 0.3\n",
    "Accuracy@2@top_gold_1 = 0.4\n",
    "Accuracy@3@top_gold_1 = 0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-06-15T16:46:27.281341Z",
     "start_time": "2023-06-15T16:45:58.376364Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SS_phase2_option3a_BSbertbase_robertabase exported to csv in path './predictions/trial/SS_phase2_option3a_BSbertbase_robertabase.tsv'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Substitute Selection phase 2, option 3: substitutes ranked on their BertScores (Bertscore was applied by using all 6 models):\n",
    "# option 3a\n",
    "score_model = 'bert-base-uncased'\n",
    "letter = 'a'\n",
    "substitutes_df, score_model_name_str = substitute_selection_phase_2_option_3(data, substitutes_df, lm_tokenizer, fill_mask, model_name_str, nlp, score_model, letter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "python tsar_eval.py --gold_file ./data/trial/tsar2022_en_trial_gold_no_noise.tsv --predictions_file ./predictions/trial/SS_phase2_option3a_BSbertbase_robertabase.tsv --output_file ./output/trial/SS_phase2_option3a_BSbertbase_robertabase.tsv"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "=========   EVALUATION config.=========\n",
    "GOLD file = ./data/trial/tsar2022_en_trial_gold_no_noise.tsv\n",
    "PREDICTION LABELS file = ./predictions/trial/SS_phase2_option3a_BSbertbase_robertabase.tsv\n",
    "OUTPUT file = ./output/trial/SS_phase2_option3a_BSbertbase_robertabase.tsv\n",
    "===============   RESULTS  =============\n",
    "\n",
    "MAP@1/Potential@1/Precision@1 = 0.5\n",
    "\n",
    "MAP@3 = 0.2333\n",
    "MAP@5 = 0.1739\n",
    "MAP@10 = 0.1153\n",
    "\n",
    "Potential@3 = 0.6\n",
    "Potential@5 = 0.9\n",
    "Potential@10 = 1.0\n",
    "\n",
    "Accuracy@1@top_gold_1 = 0.3\n",
    "Accuracy@2@top_gold_1 = 0.3\n",
    "Accuracy@3@top_gold_1 = 0.4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-06-15T16:48:19.743184Z",
     "start_time": "2023-06-15T16:46:27.285341Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SS_phase2_option3b_BSbertlarge_robertabase exported to csv in path './predictions/trial/SS_phase2_option3b_BSbertlarge_robertabase.tsv'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Option 3b: Bs with bertlarge:\n",
    "score_model = 'bert-large-uncased'\n",
    "letter = 'b'\n",
    "substitutes_df, score_model_name_str = substitute_selection_phase_2_option_3(data, substitutes_df, lm_tokenizer, fill_mask, model_name_str, nlp, score_model, letter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "python tsar_eval.py --gold_file ./data/trial/tsar2022_en_trial_gold_no_noise.tsv --predictions_file ./predictions/trial/SS_phase2_option3b_BSbertlarge_robertabase.tsv --output_file ./output/trial/SS_phase2_option3b_BSbertlarge_robertabase.tsv"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "=========   EVALUATION config.=========\n",
    "GOLD file = ./data/trial/tsar2022_en_trial_gold_no_noise.tsv\n",
    "PREDICTION LABELS file = ./predictions/trial/SS_phase2_option3b_BSbertlarge_robertabase.tsv\n",
    "OUTPUT file = ./output/trial/SS_phase2_option3b_BSbertlarge_robertabase.tsv\n",
    "===============   RESULTS  =============\n",
    "\n",
    "MAP@1/Potential@1/Precision@1 = 0.4\n",
    "\n",
    "MAP@3 = 0.2944\n",
    "MAP@5 = 0.1816\n",
    "MAP@10 = 0.1257\n",
    "\n",
    "Potential@3 = 0.6\n",
    "Potential@5 = 0.7\n",
    "Potential@10 = 0.9\n",
    "\n",
    "Accuracy@1@top_gold_1 = 0.2\n",
    "Accuracy@2@top_gold_1 = 0.4\n",
    "Accuracy@3@top_gold_1 = 0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-06-15T16:48:35.342035Z",
     "start_time": "2023-06-15T16:48:19.740183Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SS_phase2_option3c_BSelectrabase_robertabase exported to csv in path './predictions/trial/SS_phase2_option3c_BSelectrabase_robertabase.tsv'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Option 3c: Bertscore with electrabase:\n",
    "score_model = 'google/electra-base-generator'\n",
    "letter = 'c'\n",
    "substitutes_df, score_model_name_str = substitute_selection_phase_2_option_3(data, substitutes_df, lm_tokenizer, fill_mask, model_name_str, nlp, score_model, letter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "python tsar_eval.py --gold_file ./data/trial/tsar2022_en_trial_gold_no_noise.tsv --predictions_file ./predictions/trial/SS_phase2_option3c_BSelectrabase_robertabase.tsv --output_file ./output/trial/SS_phase2_option3c_BSelectrabase_robertabase.tsv"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "=========   EVALUATION config.=========\n",
    "GOLD file = ./data/trial/tsar2022_en_trial_gold_no_noise.tsv\n",
    "PREDICTION LABELS file = ./predictions/trial/SS_phase2_option3c_BSelectrabase_robertabase.tsv\n",
    "OUTPUT file = ./output/trial/SS_phase2_option3c_BSelectrabase_robertabase.tsv\n",
    "===============   RESULTS  =============\n",
    "\n",
    "MAP@1/Potential@1/Precision@1 = 0.4\n",
    "\n",
    "MAP@3 = 0.2666\n",
    "MAP@5 = 0.193\n",
    "MAP@10 = 0.1275\n",
    "\n",
    "Potential@3 = 0.6\n",
    "Potential@5 = 0.9\n",
    "Potential@10 = 1.0\n",
    "\n",
    "Accuracy@1@top_gold_1 = 0.2\n",
    "Accuracy@2@top_gold_1 = 0.4\n",
    "Accuracy@3@top_gold_1 = 0.4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-06-15T16:41:23.270123Z",
     "start_time": "2023-06-15T16:41:03.822152Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SS_phase2_option3d_BSelectralarge_robertabase exported to csv in path './predictions/trial/SS_phase2_option3d_BSelectralarge_robertabase.tsv'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Option 3d: Bertscore with electralarge:\n",
    "score_model = 'google/electra-large-generator'\n",
    "letter = 'd'\n",
    "substitutes_df, score_model_name_str = substitute_selection_phase_2_option_3(data, substitutes_df, lm_tokenizer, fill_mask, model_name_str, nlp, score_model, letter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "python tsar_eval.py --gold_file ./data/trial/tsar2022_en_trial_gold_no_noise.tsv --predictions_file ./predictions/trial/SS_phase2_option3d_BSelectralarge_robertabase.tsv --output_file ./output/trial/SS_phase2_option3d_BSelectralarge_robertabase.tsv"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "=========   EVALUATION config.=========\n",
    "GOLD file = ./data/trial/tsar2022_en_trial_gold_no_noise.tsv\n",
    "PREDICTION LABELS file = ./predictions/trial/SS_phase2_option3d_BSelectralarge_robertabase.tsv\n",
    "OUTPUT file = ./output/trial/SS_phase2_option3d_BSelectralarge_robertabase.tsv\n",
    "===============   RESULTS  =============\n",
    "\n",
    "MAP@1/Potential@1/Precision@1 = 0.4\n",
    "\n",
    "MAP@3 = 0.25\n",
    "MAP@5 = 0.198\n",
    "MAP@10 = 0.1296\n",
    "\n",
    "Potential@3 = 0.6\n",
    "Potential@5 = 0.9\n",
    "Potential@10 = 0.9\n",
    "\n",
    "Accuracy@1@top_gold_1 = 0.3\n",
    "Accuracy@2@top_gold_1 = 0.3\n",
    "Accuracy@3@top_gold_1 = 0.5\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-06-15T16:41:54.884416Z",
     "start_time": "2023-06-15T16:41:23.269130Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SS_phase2_option3e_BSrobertabase_robertabase exported to csv in path './predictions/trial/SS_phase2_option3e_BSrobertabase_robertabase.tsv'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Option 3e: Bertscore with robertabase:\n",
    "score_model = 'roberta-base'\n",
    "letter = 'e'\n",
    "substitutes_df, score_model_name_str = substitute_selection_phase_2_option_3(data, substitutes_df, lm_tokenizer, fill_mask, model_name_str, nlp, score_model, letter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "python tsar_eval.py --gold_file ./data/trial/tsar2022_en_trial_gold_no_noise.tsv --predictions_file ./predictions/trial/SS_phase2_option3e_BSrobertabase_robertabase.tsv --output_file ./output/trial/SS_phase2_option3e_BSrobertabase_robertabase.tsv"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "=========   EVALUATION config.=========\n",
    "GOLD file = ./data/trial/tsar2022_en_trial_gold_no_noise.tsv\n",
    "PREDICTION LABELS file = ./predictions/trial/SS_phase2_option3e_BSrobertabase_robertabase.tsv\n",
    "OUTPUT file = ./output/trial/SS_phase2_option3e_BSrobertabase_robertabase.tsv\n",
    "===============   RESULTS  =============\n",
    "\n",
    "MAP@1/Potential@1/Precision@1 = 0.6\n",
    "\n",
    "MAP@3 = 0.2833\n",
    "MAP@5 = 0.188\n",
    "MAP@10 = 0.1323\n",
    "\n",
    "Potential@3 = 0.7\n",
    "Potential@5 = 0.9\n",
    "Potential@10 = 0.9\n",
    "\n",
    "Accuracy@1@top_gold_1 = 0.4\n",
    "Accuracy@2@top_gold_1 = 0.4\n",
    "Accuracy@3@top_gold_1 = 0.4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-06-15T16:43:31.775415Z",
     "start_time": "2023-06-15T16:41:54.888416Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SS_phase2_option3f_BSrobertalarge_robertabase exported to csv in path './predictions/trial/SS_phase2_option3f_BSrobertalarge_robertabase.tsv'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Option 3f: Bertscore with robertalarge:\n",
    "score_model = 'roberta-large'\n",
    "letter = 'f'\n",
    "substitutes_df, score_model_name_str = substitute_selection_phase_2_option_3(data, substitutes_df, lm_tokenizer, fill_mask, model_name_str, nlp, score_model, letter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "python tsar_eval.py --gold_file ./data/trial/tsar2022_en_trial_gold_no_noise.tsv --predictions_file ./predictions/trial/SS_phase2_option3f_BSrobertalarge_robertabase.tsv --output_file ./output/trial/SS_phase2_option3f_BSrobertalarge_robertabase.tsv"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-06-15T16:43:31.786415Z",
     "start_time": "2023-06-15T16:43:31.775415Z"
    },
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "=========   EVALUATION config.=========\n",
    "GOLD file = ./data/trial/tsar2022_en_trial_gold_no_noise.tsv\n",
    "PREDICTION LABELS file = ./predictions/trial/SS_phase2_option3f_BSrobertalarge_robertabase.tsv\n",
    "OUTPUT file = ./output/trial/SS_phase2_option3f_BSrobertalarge_robertabase.tsv\n",
    "===============   RESULTS  =============\n",
    "\n",
    "MAP@1/Potential@1/Precision@1 = 0.5\n",
    "\n",
    "MAP@3 = 0.2944\n",
    "MAP@5 = 0.2256\n",
    "MAP@10 = 0.1456\n",
    "\n",
    "Potential@3 = 0.7\n",
    "Potential@5 = 0.8\n",
    "Potential@10 = 1.0\n",
    "\n",
    "Accuracy@1@top_gold_1 = 0.4\n",
    "Accuracy@2@top_gold_1 = 0.6\n",
    "Accuracy@3@top_gold_1 = 0.6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### For model 'google/electra-large-generator':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "data, substitutes_df = get_data_and_create_empty_df()\n",
    "\n",
    "model = 'google/electra-large-generator'\n",
    "model_name_str = get_str_for_file_name(model)\n",
    "\n",
    "nlp, lm_tokenizer, lm_model, fill_mask = instantiate_spacy_tokenizer_model_pipeline(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SS_phase2_option1_SharedSyns_electralarge exported to csv in path './predictions/trial/SS_phase2_option1_SharedSyns_electralarge.tsv'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Substitute Selection phase 2, option 1: substitutes that are synonyms of the complex word first (lemmatized substitutes that share the same synset as the lemmatized complex word)\n",
    "substitutes_df = substitute_selection_phase_2_option_1(data, substitutes_df, lm_tokenizer, fill_mask, model_name_str, nlp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "python tsar_eval.py --gold_file ./data/trial/tsar2022_en_trial_gold_no_noise.tsv --predictions_file ./predictions/trial/SS_phase2_option1_SharedSyns_electralarge.tsv --output_file ./output/trial/SS_phase2_option1_SharedSyns_electralarge.tsv"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "=========   EVALUATION config.=========\n",
    "GOLD file = ./data/trial/tsar2022_en_trial_gold_no_noise.tsv\n",
    "PREDICTION LABELS file = ./predictions/trial/SS_phase2_option1_SharedSyns_electralarge.tsv\n",
    "OUTPUT file = ./output/trial/SS_phase2_option1_SharedSyns_electralarge.tsv\n",
    "===============   RESULTS  =============\n",
    "\n",
    "MAP@1/Potential@1/Precision@1 = 0.5\n",
    "\n",
    "MAP@3 = 0.3833\n",
    "MAP@5 = 0.286\n",
    "MAP@10 = 0.1709\n",
    "\n",
    "Potential@3 = 0.8\n",
    "Potential@5 = 0.8\n",
    "Potential@10 = 0.9\n",
    "\n",
    "Accuracy@1@top_gold_1 = 0.3\n",
    "Accuracy@2@top_gold_1 = 0.4\n",
    "Accuracy@3@top_gold_1 = 0.6\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SS_phase2_option2_SharedHyper1_electralarge exported to csv in path './predictions/trial/SS_phase2_option2_SharedHyper1_electralarge.tsv'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Substitute Selection phase 2, option 2a: sort the substitutes that share their 1st level hypernyms with the complex word first: (retrieving the 1-level hypernyms for all possible synsets of the complex word and substitutes, and then compare these hypernyms to identify shared hypernyms between the complex word and substitutes. these will be prioritized in the resulting list).\n",
    "substitutes_df = substitute_selection_phase_2_option_2(data, substitutes_df, lm_tokenizer, fill_mask, model_name_str, nlp, levels=[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "python tsar_eval.py --gold_file ./data/trial/tsar2022_en_trial_gold_no_noise.tsv --predictions_file ./predictions/trial/SS_phase2_option2_SharedHyper1_electralarge.tsv --output_file ./output/trial/SS_phase2_option2_SharedHyper1_electralarge.tsv"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "=========   EVALUATION config.=========\n",
    "GOLD file = ./data/trial/tsar2022_en_trial_gold_no_noise.tsv\n",
    "PREDICTION LABELS file = ./predictions/trial/SS_phase2_option2_SharedHyper1_electralarge.tsv\n",
    "OUTPUT file = ./output/trial/SS_phase2_option2_SharedHyper1_electralarge.tsv\n",
    "===============   RESULTS  =============\n",
    "\n",
    "MAP@1/Potential@1/Precision@1 = 0.6\n",
    "\n",
    "MAP@3 = 0.3055\n",
    "MAP@5 = 0.2843\n",
    "MAP@10 = 0.1743\n",
    "\n",
    "Potential@3 = 0.8\n",
    "Potential@5 = 0.9\n",
    "Potential@10 = 0.9\n",
    "\n",
    "Accuracy@1@top_gold_1 = 0.1\n",
    "Accuracy@2@top_gold_1 = 0.2\n",
    "Accuracy@3@top_gold_1 = 0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SS_phase2_option2_SharedHyper2_electralarge exported to csv in path './predictions/trial/SS_phase2_option2_SharedHyper2_electralarge.tsv'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Substitute Selection phase 2, option 2b: sort the substitutes that share their indirect hypernyms (2 levels up) with the complex word first\n",
    "substitutes_df = substitute_selection_phase_2_option_2(data, substitutes_df, lm_tokenizer, fill_mask, model_name_str, nlp, levels=[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "python tsar_eval.py --gold_file ./data/trial/tsar2022_en_trial_gold_no_noise.tsv --predictions_file ./predictions/trial/SS_phase2_option2_SharedHyper2_electralarge.tsv --output_file ./output/trial/SS_phase2_option2_SharedHyper2_electralarge.tsv"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "=========   EVALUATION config.=========\n",
    "GOLD file = ./data/trial/tsar2022_en_trial_gold_no_noise.tsv\n",
    "PREDICTION LABELS file = ./predictions/trial/SS_phase2_option2_SharedHyper2_electralarge.tsv\n",
    "OUTPUT file = ./output/trial/SS_phase2_option2_SharedHyper2_electralarge.tsv\n",
    "===============   RESULTS  =============\n",
    "\n",
    "MAP@1/Potential@1/Precision@1 = 0.6\n",
    "\n",
    "MAP@3 = 0.3388\n",
    "MAP@5 = 0.2853\n",
    "MAP@10 = 0.1688\n",
    "\n",
    "Potential@3 = 0.8\n",
    "Potential@5 = 0.9\n",
    "Potential@10 = 0.9\n",
    "\n",
    "Accuracy@1@top_gold_1 = 0.3\n",
    "Accuracy@2@top_gold_1 = 0.3\n",
    "Accuracy@3@top_gold_1 = 0.5\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SS_phase2_option2_SharedHyper1_2_electralarge exported to csv in path './predictions/trial/SS_phase2_option2_SharedHyper1_2_electralarge.tsv'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Substitute Selection phase 2, option 2c: sort the substitutes that share either their direct (1 level up) or their indirect (2 levels up) hypernyms with the complex word first\n",
    "substitutes_df = substitute_selection_phase_2_option_2(data, substitutes_df, lm_tokenizer, fill_mask, model_name_str, nlp, levels=[1,2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "python tsar_eval.py --gold_file ./data/trial/tsar2022_en_trial_gold_no_noise.tsv --predictions_file ./predictions/trial/SS_phase2_option2_SharedHyper1_2_electralarge.tsv --output_file ./output/trial/SS_phase2_option2_SharedHyper1_2_electralarge.tsv"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "=========   EVALUATION config.=========\n",
    "GOLD file = ./data/trial/tsar2022_en_trial_gold_no_noise.tsv\n",
    "PREDICTION LABELS file = ./predictions/trial/SS_phase2_option2_SharedHyper1_2_electralarge.tsv\n",
    "OUTPUT file = ./output/trial/SS_phase2_option2_SharedHyper1_2_electralarge.tsv\n",
    "===============   RESULTS  =============\n",
    "\n",
    "MAP@1/Potential@1/Precision@1 = 0.6\n",
    "\n",
    "MAP@3 = 0.2833\n",
    "MAP@5 = 0.229\n",
    "MAP@10 = 0.1564\n",
    "\n",
    "Potential@3 = 0.7\n",
    "Potential@5 = 0.9\n",
    "Potential@10 = 0.9\n",
    "\n",
    "Accuracy@1@top_gold_1 = 0.2\n",
    "Accuracy@2@top_gold_1 = 0.2\n",
    "Accuracy@3@top_gold_1 = 0.2\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SS_phase2_option3a_BSbertbase_electralarge exported to csv in path './predictions/trial/SS_phase2_option3a_BSbertbase_electralarge.tsv'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Substitute Selection phase 2, option 3: substitutes ranked on their BertScores (Bertscore was applied by using all 6 models):\n",
    "# option 3a\n",
    "score_model = 'bert-base-uncased'\n",
    "letter = 'a'\n",
    "substitutes_df, score_model_name_str = substitute_selection_phase_2_option_3(data, substitutes_df, lm_tokenizer, fill_mask, model_name_str, nlp, score_model, letter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "python tsar_eval.py --gold_file ./data/trial/tsar2022_en_trial_gold_no_noise.tsv --predictions_file ./predictions/trial/SS_phase2_option3a_BSbertbase_electralarge.tsv --output_file ./output/trial/SS_phase2_option3a_BSbertbase_electralarge.tsv"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "=========   EVALUATION config.=========\n",
    "GOLD file = ./data/trial/tsar2022_en_trial_gold_no_noise.tsv\n",
    "PREDICTION LABELS file = ./predictions/trial/SS_phase2_option3a_BSbertbase_electralarge.tsv\n",
    "OUTPUT file = ./output/trial/SS_phase2_option3a_BSbertbase_electralarge.tsv\n",
    "===============   RESULTS  =============\n",
    "\n",
    "MAP@1/Potential@1/Precision@1 = 0.6\n",
    "\n",
    "MAP@3 = 0.2777\n",
    "MAP@5 = 0.1796\n",
    "MAP@10 = 0.1395\n",
    "\n",
    "Potential@3 = 0.6\n",
    "Potential@5 = 0.9\n",
    "Potential@10 = 0.9\n",
    "\n",
    "Accuracy@1@top_gold_1 = 0.4\n",
    "Accuracy@2@top_gold_1 = 0.4\n",
    "Accuracy@3@top_gold_1 = 0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SS_phase2_option3b_BSbertlarge_electralarge exported to csv in path './predictions/trial/SS_phase2_option3b_BSbertlarge_electralarge.tsv'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Option 3b: Bs with bertlarge:\n",
    "score_model = 'bert-large-uncased'\n",
    "letter = 'b'\n",
    "substitutes_df, score_model_name_str = substitute_selection_phase_2_option_3(data, substitutes_df, lm_tokenizer, fill_mask, model_name_str, nlp, score_model, letter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "python tsar_eval.py --gold_file ./data/trial/tsar2022_en_trial_gold_no_noise.tsv --predictions_file ./predictions/trial/SS_phase2_option3b_BSbertlarge_electralarge.tsv --output_file ./output/trial/SS_phase2_option3b_BSbertlarge_electralarge.tsv"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "=========   EVALUATION config.=========\n",
    "GOLD file = ./data/trial/tsar2022_en_trial_gold_no_noise.tsv\n",
    "PREDICTION LABELS file = ./predictions/trial/SS_phase2_option3b_BSbertlarge_electralarge.tsv\n",
    "OUTPUT file = ./output/trial/SS_phase2_option3b_BSbertlarge_electralarge.tsv\n",
    "===============   RESULTS  =============\n",
    "\n",
    "MAP@1/Potential@1/Precision@1 = 0.5\n",
    "\n",
    "MAP@3 = 0.2999\n",
    "MAP@5 = 0.193\n",
    "MAP@10 = 0.1312\n",
    "\n",
    "Potential@3 = 0.7\n",
    "Potential@5 = 0.8\n",
    "Potential@10 = 0.8\n",
    "\n",
    "Accuracy@1@top_gold_1 = 0.4\n",
    "Accuracy@2@top_gold_1 = 0.5\n",
    "Accuracy@3@top_gold_1 = 0.5\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SS_phase2_option3c_BSelectrabase_electralarge exported to csv in path './predictions/trial/SS_phase2_option3c_BSelectrabase_electralarge.tsv'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Option 3c: Bertscore with electrabase:\n",
    "score_model = 'google/electra-base-generator'\n",
    "letter = 'c'\n",
    "substitutes_df, score_model_name_str = substitute_selection_phase_2_option_3(data, substitutes_df, lm_tokenizer, fill_mask, model_name_str, nlp, score_model, letter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "python tsar_eval.py --gold_file ./data/trial/tsar2022_en_trial_gold_no_noise.tsv --predictions_file ./predictions/trial/SS_phase2_option3c_BSelectrabase_electralarge.tsv --output_file ./output/trial/SS_phase2_option3c_BSelectrabase_electralarge.tsv"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "=========   EVALUATION config.=========\n",
    "GOLD file = ./data/trial/tsar2022_en_trial_gold_no_noise.tsv\n",
    "PREDICTION LABELS file = ./predictions/trial/SS_phase2_option3c_BSelectrabase_electralarge.tsv\n",
    "OUTPUT file = ./output/trial/SS_phase2_option3c_BSelectrabase_electralarge.tsv\n",
    "===============   RESULTS  =============\n",
    "\n",
    "MAP@1/Potential@1/Precision@1 = 0.6\n",
    "\n",
    "MAP@3 = 0.3722\n",
    "MAP@5 = 0.2523\n",
    "MAP@10 = 0.1683\n",
    "\n",
    "Potential@3 = 0.7\n",
    "Potential@5 = 0.8\n",
    "Potential@10 = 0.9\n",
    "\n",
    "Accuracy@1@top_gold_1 = 0.2\n",
    "Accuracy@2@top_gold_1 = 0.4\n",
    "Accuracy@3@top_gold_1 = 0.4\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SS_phase2_option3d_BSelectralarge_electralarge exported to csv in path './predictions/trial/SS_phase2_option3d_BSelectralarge_electralarge.tsv'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Option 3d: Bertscore with electralarge:\n",
    "score_model = 'google/electra-large-generator'\n",
    "letter = 'd'\n",
    "substitutes_df, score_model_name_str = substitute_selection_phase_2_option_3(data, substitutes_df, lm_tokenizer, fill_mask, model_name_str, nlp, score_model, letter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "python tsar_eval.py --gold_file ./data/trial/tsar2022_en_trial_gold_no_noise.tsv --predictions_file ./predictions/trial/SS_phase2_option3d_BSelectralarge_electralarge.tsv --output_file ./output/trial/SS_phase2_option3d_BSelectralarge_electralarge.tsv"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "=========   EVALUATION config.=========\n",
    "GOLD file = ./data/trial/tsar2022_en_trial_gold_no_noise.tsv\n",
    "PREDICTION LABELS file = ./predictions/trial/SS_phase2_option3d_BSelectralarge_electralarge.tsv\n",
    "OUTPUT file = ./output/trial/SS_phase2_option3d_BSelectralarge_electralarge.tsv\n",
    "===============   RESULTS  =============\n",
    "\n",
    "MAP@1/Potential@1/Precision@1 = 0.4\n",
    "\n",
    "MAP@3 = 0.2777\n",
    "MAP@5 = 0.2226\n",
    "MAP@10 = 0.1445\n",
    "\n",
    "Potential@3 = 0.7\n",
    "Potential@5 = 0.9\n",
    "Potential@10 = 0.9\n",
    "\n",
    "Accuracy@1@top_gold_1 = 0.4\n",
    "Accuracy@2@top_gold_1 = 0.4\n",
    "Accuracy@3@top_gold_1 = 0.4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SS_phase2_option3e_BSrobertabase_electralarge exported to csv in path './predictions/trial/SS_phase2_option3e_BSrobertabase_electralarge.tsv'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Option 3e: Bertscore with robertabase:\n",
    "score_model = 'roberta-base'\n",
    "letter = 'e'\n",
    "substitutes_df, score_model_name_str = substitute_selection_phase_2_option_3(data, substitutes_df, lm_tokenizer, fill_mask, model_name_str, nlp, score_model, letter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "python tsar_eval.py --gold_file ./data/trial/tsar2022_en_trial_gold_no_noise.tsv --predictions_file ./predictions/trial/SS_phase2_option3e_BSrobertabase_electralarge.tsv --output_file ./output/trial/SS_phase2_option3e_BSrobertabase_electralarge.tsv"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "=========   EVALUATION config.=========\n",
    "GOLD file = ./data/trial/tsar2022_en_trial_gold_no_noise.tsv\n",
    "PREDICTION LABELS file = ./predictions/trial/SS_phase2_option3e_BSrobertabase_electralarge.tsv\n",
    "OUTPUT file = ./output/trial/SS_phase2_option3e_BSrobertabase_electralarge.tsv\n",
    "===============   RESULTS  =============\n",
    "\n",
    "MAP@1/Potential@1/Precision@1 = 0.5\n",
    "\n",
    "MAP@3 = 0.2944\n",
    "MAP@5 = 0.2066\n",
    "MAP@10 = 0.1535\n",
    "\n",
    "Potential@3 = 0.9\n",
    "Potential@5 = 0.9\n",
    "Potential@10 = 0.9\n",
    "\n",
    "Accuracy@1@top_gold_1 = 0.4\n",
    "Accuracy@2@top_gold_1 = 0.4\n",
    "Accuracy@3@top_gold_1 = 0.4\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SS_phase2_option3f_BSrobertalarge_electralarge exported to csv in path './predictions/trial/SS_phase2_option3f_BSrobertalarge_electralarge.tsv'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Option 3f: Bertscore with robertalarge:\n",
    "score_model = 'roberta-large'\n",
    "letter = 'f'\n",
    "substitutes_df, score_model_name_str = substitute_selection_phase_2_option_3(data, substitutes_df, lm_tokenizer, fill_mask, model_name_str, nlp, score_model, letter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "python tsar_eval.py --gold_file ./data/trial/tsar2022_en_trial_gold_no_noise.tsv --predictions_file ./predictions/trial/SS_phase2_option3f_BSrobertalarge_electralarge.tsv --output_file ./output/trial/SS_phase2_option3f_BSrobertalarge_electralarge.tsv"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "=========   EVALUATION config.=========\n",
    "GOLD file = ./data/trial/tsar2022_en_trial_gold_no_noise.tsv\n",
    "PREDICTION LABELS file = ./predictions/trial/SS_phase2_option3f_BSrobertalarge_electralarge.tsv\n",
    "OUTPUT file = ./output/trial/SS_phase2_option3f_BSrobertalarge_electralarge.tsv\n",
    "===============   RESULTS  =============\n",
    "\n",
    "MAP@1/Potential@1/Precision@1 = 0.6\n",
    "\n",
    "MAP@3 = 0.3055\n",
    "MAP@5 = 0.2633\n",
    "MAP@10 = 0.1698\n",
    "\n",
    "Potential@3 = 0.7\n",
    "Potential@5 = 0.9\n",
    "Potential@10 = 0.9\n",
    "\n",
    "Accuracy@1@top_gold_1 = 0.4\n",
    "Accuracy@2@top_gold_1 = 0.5\n",
    "Accuracy@3@top_gold_1 = 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Results\n",
    "Based on the accumulated scores, the best model per strategy, resulting from the SS step phase 2, proceeds to the SR step (refer to table 4.12 in section 4.3.4 of the thesis):\n",
    "\n",
    "- Regarding shared synsets: SS_phase2_option1_SharedSyns_robertabase (accum. score: 5.3215). Model name in thesis: RB_Syns-shared.\n",
    "- Regarding shared hypernyms: SS_phase2_option2_SharedHyper2_robertabase (accum. score: 5.8015). Model name in thesis: RB_Hyper2-shared.\n",
    "- Regarding BERTScore similarity scores: SS_phase2_option3f_BSrobertalarge_robertabase (accum. score: 5.2656). Model name in thesis: RB_BSrl."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tensorflow_env",
   "language": "python",
   "name": "tensorflow_env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}