replication.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Replication experiment\n",
    "\n",
    "This is the main notebook required to obtain the results of our replication study. The notebook is build around the concepts of predefined configuration files. These configuration files can be found within the codebase. The configuration files for different datasets and different explainers can be chosen by changing the parameters in the second codeblock. \n",
    "\n",
    "When loaded, the configuration for a replication experiment is passed to the replication function. This function is responsible for running all parts of the evaluation; quantitative, qualitative and efficiency. The results for the quantitative and efficiency studies are returned by the replication method and also stored in the `results` folder. The results of the qualitative study are stored in the folder name `qualitative`. \n",
    "\n",
    "**Be aware that the replication function can take very long to completed**. This is caused by the method averaging all scores over ten runs. If speed is required over accuracy the last line of the 2nd codeblock can be uncommented. This will make the evaluation run over one run only. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ExplanationEvaluation.configs.selector import Selector\n",
    "from ExplanationEvaluation.tasks.replication import replication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_dataset = 'bashapes' # One of: bashapes, bacommunity, treecycles, treegrids, ba2motifs, mutag\n",
    "_explainer = 'pgexplainer' # One of: pgexplainer, gnnexplainer\n",
    "\n",
    "# Parameters below should only be changed if you want to run any of the experiments in the supplementary\n",
    "_folder = 'replication' # One of: replication, extension\n",
    "\n",
    "# PGExplainer\n",
    "config_path = f\"./ExplanationEvaluation/configs/{_folder}/explainers/{_explainer}/{_dataset}.json\"\n",
    "\n",
    "config = Selector(config_path)\n",
    "extension = (_folder == 'extension')\n",
    "\n",
    "# config.args.explainer.seeds = [0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/10 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading syn1 dataset\n",
      "This model obtained: Train Acc: 0.9696, Val Acc: 1.0000, Test Acc: 1.0000.\n",
      "Run 0 with seed 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 10/10 [00:06<00:00,  1.54it/s]\n",
      "100%|██████████| 60/60 [00:02<00:00, 22.66it/s]\n",
      "  0%|          | 0/10 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "score: 0.9993016799871388\n",
      "time_elased: 44.1603144009908\n",
      "Run 1 with seed 1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 10/10 [00:06<00:00,  1.53it/s]\n",
      "100%|██████████| 60/60 [00:02<00:00, 22.39it/s]\n",
      "  0%|          | 0/10 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "score: 0.999546271416104\n",
      "time_elased: 44.68813339869181\n",
      "Run 2 with seed 2\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 10/10 [00:06<00:00,  1.48it/s]\n",
      " 25%|██▌       | 15/60 [00:00<00:02, 16.76it/s]"
     ]
    }
   ],
   "source": [
    "(auc, auc_std), inf_time = replication(config.args.explainer, extension)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print((auc, auc_std), inf_time)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}