inference_deeper_impact.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from src.llama2.generate import LLamaQueryGenerator\n",
    "from src.utils.utils import merge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Llama 2 Expansion Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Option 1: Meta Website"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the Llama 2 7B pre-trained weights by visiting [Llama 2 GitHub Page](https://github.com/meta-llama/llama) and following the instructions.\n",
    "\n",
    "We need to convert the checkpoint from its original format into the Hugging Face format. For that, run:\n",
    "\n",
    "```bash\n",
    "pip install -r requirements.txt\n",
    "```\n",
    "\n",
    "Assuming the downloaded checkpoint resides under `./llama2/7B`, run the following:\n",
    "\n",
    "```bash\n",
    "TRANSFORM=`python -c \"import transformers;print('/'.join(transformers.__file__.split('/')[:-1])+'/models/llama/convert_llama_weights_to_hf.py')\"`\n",
    "python ${TRANSFORM} --input_dir llama2 --model_size 7B --output_dir llama2/hf/7B\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "LLAMA2_PATH = './llama2/hf/7B'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Option 2: Hugging Face\n",
    "\n",
    "Request access to the model by acknowledging the license and filling the form in the model card at [https://huggingface.co/meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf) and use the link directly.\n",
    "\n",
    "This option might take more time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# LLAMA2_PATH = 'meta-llama/Llama-2-7b-hf'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hyperparameters used for expansion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_INPUT_TOKEN_LEN = 512\n",
    "NUM_QUERIES = 80\n",
    "MAX_NEW_TOKENS = 50\n",
    "TOP_K = 50\n",
    "TOP_P = 0.95"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loading Llama 2 with LoRA weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5f8b3f4ce4494199bf8fede4fbf44788",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "generator = LLamaQueryGenerator(\n",
    "    llama_path=LLAMA2_PATH,\n",
    "    max_tokens=MAX_INPUT_TOKEN_LEN,\n",
    "    peft_path='soyuj/llama2-doc2query'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Document Expansion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "document = \"The presence of communication amid scientific minds was equally important to the success of the Manhattan Project as scientific intellect was. The only cloud hanging over the impressive achievement of the atomic researchers and engineers is what their success truly meant; hundreds of thousands of innocent lives obliterated.\"\n",
    "inputs = [document]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.\n"
     ]
    }
   ],
   "source": [
    "expansions = generator.generate(\n",
    "    inputs,\n",
    "    num_return_sequences=NUM_QUERIES,\n",
    "    max_new_tokens=MAX_NEW_TOKENS,\n",
    "    do_sample=True,\n",
    "    top_k=TOP_K,\n",
    "    top_p=TOP_P\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "who was involved in the manhattan project\n",
      "what is the meaning of the manhattan project\n",
      "how important was communication in the manhattan project\n",
      "what was the importance of communications in the success of the manhattan project?\n",
      "why was the manhattan project a success\n",
      "what was the scientific achievement of the atomic bomb\n",
      "manhattan project research team\n",
      "was the manhattan project an success or failure\n",
      "what does it mean to be involved with the manhattan project\n",
      "what was the manhattan project and why was it successful\n",
      "what was the manhattan project and what did it accomplish\n",
      "which scientific discipline had the most impact on the success of the manhattan project\n",
      "how does the scientists feel about atomic bomb\n",
      "what is the manhattan project?\n",
      "why was science important to the manhattan project\n",
      "which project included the development of nuclear weapons\n",
      "who was behind the manhattan project\n",
      "what was the manhattan project\n",
      "was the manhattan project important\n",
      "why was communication important in the manhattan project\n",
      "what contributed to the success of the manhattan project\n",
      "what manhattan project scientists were at the beginning of the project\n",
      "what was the significance of manhattan project\n",
      "what was the name of the secret place that conducted the manhattan project?\n",
      "how did the atomic bomb succeed\n",
      "was scientific knowledge gained during the project manhattan\n",
      "what was the manhattan project\n",
      "what was the importance of the atomic bombing of hiroshima?\n",
      "what does the manhattan project mean\n",
      "why was the manhattan project important\n",
      "was the manhattan project important\n",
      "what manhattan project project was manhattan a project\n",
      "who was involved in manhattan project\n",
      "was the atomic bomb truly necessary\n",
      "why was communication with other scientists so vital in the early stages of the manhattan project\n",
      "what manhattan project\n",
      "why was the manhattan project successful\n",
      "was the manhattan project successful\n",
      "was the manhattan project a success\n",
      "what was the role of communication on the manhattan project\n",
      "what was the manhattan project created for?\n",
      "who is responsible for manhattan project\n",
      "what is the manhattan project\n",
      "what role did communication and social interaction play in the project?\n",
      "was the manhattan project justified or not\n",
      "what was the significance of the manhattan project\n",
      "the manhattan project impact\n",
      "what was significant about manhattan project\n",
      "why was the manhattan project so important\n",
      "why was scientific communication so important during the manhattan project\n",
      "how was the manhattan project created\n",
      "who made the atomic bomb\n",
      "what were the roles of leaders for the manhattan project\n",
      "was the manhattan project a success or failure\n",
      "what was the most important goal of the manhattan project?\n",
      "what was the purpose of the manhattan project\n",
      "how important was communication during the development of the manhattan project?\n",
      "was the manhattan project successful\n",
      "how did the success of the manhattan project reflect\n",
      "what led to the success of the manhattan project\n",
      "what is the significance of the manhattan project\n",
      "which statement is most characteristic of the atomic bombing of japan?\n",
      "why is the manhattan project an important event\n",
      "why was the manhattan project important\n",
      "what scientific background was necessary to work on the manhattan project?\n",
      "what was the purpose of the manhattan project\n",
      "what was the success of the manhattan project\n",
      "what role does the manhattan project have in a science fiction novel\n",
      "did the atomic scientists consider the ramifications of their work when they worked on the manhattan project\n",
      "which achievement was most important to the scientific progress of world war ii?\n",
      "what made the manhattan project a success?\n",
      "what did the manhattan project do\n",
      "what made the manhattan project successful\n",
      "what does the manhattan project mean\n",
      "what was the purpose of the manhattan project\n",
      "was the manhattan project successful?\n",
      "what was the manhattan project important\n",
      "what made the manhattan project so successful\n",
      "what was the manhattan project?\n",
      "why was the manhattan project important\n"
     ]
    }
   ],
   "source": [
    "for expansion in expansions[0]:\n",
    "    print(expansion)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Appending new expansion terms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The presence of communication amid scientific minds was equally important to the success of the Manhattan Project as scientific intellect was. The only cloud hanging over the impressive achievement of the atomic researchers and engineers is what their success truly meant; hundreds of thousands of innocent lives obliterated. team does place led leaders created weapons at so when how name be feel included failure discipline impact do did were fiction about who worked gained hiroshima mean most on stages ramifications significant science which that characteristic event nuclear social research war succeed secret significance early work ii bombing successful novel bomb meaning involved development accomplish it play goal behind for japan knowledge contributed a had during made roles statement have vital not an other interaction or importance world communications scientists background conducted purpose progress responsible reflect role they with in why justified necessary consider beginning\n"
     ]
    }
   ],
   "source": [
    "expanded_document = merge(document, expansions[0])\n",
    "print(expanded_document)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inference with DeeperImpact"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "from src.deep_impact.models import DeepImpact"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "deeper_impact = DeepImpact.from_pretrained('soyuj/deeper-impact')\n",
    "deeper_impact.eval();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('the', 0.65567946), ('presence', 2.607322), ('of', 0.48188052), ('communication', 3.691589), ('amid', 3.4650323), ('scientific', 2.6603425), ('minds', 2.482521), ('was', 0.7672796), ('equally', 2.8385103), ('important', 2.6603296), ('to', 0.67902374), ('success', 3.1919205), ('manhattan', 4.527264), ('project', 2.7042632), ('as', 0.31514826), ('intellect', 3.555631), ('only', 1.3839537), ('cloud', 3.512771), ('hanging', 2.7201285), ('over', 1.6333817), ('impressive', 2.7397997), ('achievement', 2.761028), ('atomic', 3.3154886), ('researchers', 1.7769612), ('and', 0.23050006), ('engineers', 2.0636017), ('is', 0.3808705), ('what', 0.44478962), ('their', 0.41361985), ('truly', 1.799064), ('meant', 0.88288176), ('hundreds', 1.0101533), ('thousands', 0.8593701), ('innocent', 2.6967251), ('lives', 1.3273858), ('obliterated', 3.5495284), ('team', 1.418661), ('does', 0.51023555), ('place', 0.5456477), ('led', 0.70278203), ('leaders', 0.79924226), ('created', 0.0), ('weapons', 0.40899017), ('at', 0.14039692), ('so', 0.2498331), ('when', 0.0), ('how', 0.09834278), ('name', 0.0), ('be', 0.12824894), ('feel', 0.52665937), ('included', 0.020318337), ('failure', 0.9496788), ('discipline', 0.16453774), ('impact', 0.63975656), ('do', 0.5500505), ('did', 1.0159733), ('were', 0.5095451), ('fiction', 0.116154835), ('about', 0.088284165), ('who', 0.0), ('worked', 0.37038213), ('gained', 0.3308649), ('hiroshima', 2.694856), ('mean', 0.43517774), ('most', 0.26356035), ('on', 0.17517641), ('stages', 0.0), ('ramifications', 0.61331236), ('significant', 1.6739875), ('science', 2.1209028), ('which', 0.0), ('that', 0.16778676), ('characteristic', 0.39264068), ('event', 0.4357824), ('nuclear', 2.059705), ('social', 0.1820672), ('research', 1.9943281), ('war', 1.0510502), ('succeed', 2.348107), ('secret', 0.9140662), ('significance', 1.4335434), ('early', 0.0), ('work', 0.45690867), ('ii', 0.79559934), ('bombing', 0.7679534), ('successful', 2.9322484), ('novel', 0.0), ('bomb', 1.888563), ('meaning', 0.0), ('involved', 0.27117136), ('development', 0.5459063), ('accomplish', 1.1552042), ('it', 0.031300217), ('play', 0.024912462), ('goal', 0.68410623), ('behind', 0.16158186), ('for', 0.30054682), ('japan', 0.9849607), ('knowledge', 1.0114889), ('contributed', 0.21641459), ('a', 0.3267625), ('had', 0.15509938), ('during', 0.0348548), ('made', 0.0978632), ('roles', 0.0), ('statement', 0.0), ('have', 0.23918313), ('vital', 1.2186114), ('not', 0.2140626), ('an', 0.27995324), ('other', 0.099128336), ('interaction', 0.502367), ('or', 0.15023825), ('importance', 2.8616989), ('world', 1.0633146), ('communications', 2.809551), ('scientists', 2.1175172), ('background', 0.1521832), ('conducted', 0.39807206), ('purpose', 0.56767094), ('progress', 1.0624642), ('responsible', 0.23610064), ('reflect', 0.0), ('role', 0.28581366), ('they', 0.009961218), ('with', 0.0), ('in', 0.27379692), ('why', 1.0819248), ('justified', 0.0), ('necessary', 0.71488917), ('consider', 0.043811038), ('beginning', 0.0)]\n"
     ]
    }
   ],
   "source": [
    "impact_scores = deeper_impact.get_impact_scores(expanded_document)\n",
    "print(impact_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For performing expansions and inferences for larger collection of documents, refer to the `README.md`."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}