V1/engineering-best-practises/Business Agility - migration between investment systems.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <form action=\"javascript:code_toggle()\"><input type=\"submit\" id=\"toggleButton\" value=\"Toggle Docstring\"></form>\n",
       "    \n",
       "         <script>\n",
       "         function code_toggle() {\n",
       "             if ($('div.cell.code_cell.rendered.selected div.input').css('display')!='none'){\n",
       "                 $('div.cell.code_cell.rendered.selected div.input').hide();\n",
       "             } else {\n",
       "                 $('div.cell.code_cell.rendered.selected div.input').show();\n",
       "             }\n",
       "         }\n",
       "         </script>\n",
       "\n",
       "     "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from lusidtools.jupyter_tools import toggle_code\n",
    "\n",
    "\"\"\"Business agility\n",
    "\n",
    "Demonstration of how to migrate your data from one system to another.\n",
    "\n",
    "Attributes\n",
    "----------\n",
    "transactions\n",
    "holdings\n",
    "set holdings\n",
    "instruments\n",
    "properties\n",
    "transaction configuration\n",
    "reconciliations\n",
    "\"\"\"\n",
    "\n",
    "toggle_code(\"Toggle Docstring\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Challenge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the biggest challenges in migrating your investment records from one system to another is in ensuring that your transactions are represented correctly in the new system.\n",
    "\n",
    "It is critical that the new system can interpret each transaction appropriately and determine what the underlying movements in cash and securities are.\n",
    "\n",
    "This can be difficult when transactions are given esoteric types such as '1085' or 'STP' (STP is the MT940 SWIFT transaction type for Stamp Duty, but it could just as easily be an acronym for 'Sale Third Party' or 'Sale Transaction Processed'). \n",
    "\n",
    "The end result is typically a lot of time spent chasing down the handful of people in the organisation that understand the meaning of each transaction type and then laboriously mapping those to the new system's available types. This is one of the reasons that migrations between systems can take so long. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rather than forcing a set of transaction types onto you, LUSID allows you to configure your own set of transaction types. Each type specifies the underlying movements that should occur when a transaction with this type is upserted into LUSID. \n",
    "\n",
    "In this notebook you will set up your own transaction type configuration and simulate how you might adjust this during a migration. You will achieve this by:\n",
    "\n",
    "1) Setting up a Scope to Hold your Portfolio\n",
    "\n",
    "2) Creating your Portfolio\n",
    "\n",
    "3) Creating your Instrument Universe\n",
    "\n",
    "4) Setting your Initial Holdings\n",
    "\n",
    "5) Getting your Output Transactions\n",
    "\n",
    "6) Setting your Transaction Type Configuration\n",
    "\n",
    "7) Loading your Transactions \n",
    "\n",
    "8) Retrieving your Holdings\n",
    "\n",
    "9) Updating your Transaction Type Configuration\n",
    "\n",
    "10) Adjusting your Holdings with the End of Day Report\n",
    "\n",
    "11) Performing a Reconciliation\n",
    "\n",
    "12) Investigating the Reconcilliation Breaks & Updating your Transaction Type Configuration\n",
    "\n",
    "13) Re-running your Reconciliation\n",
    "\n",
    "*Run the cell below to import the libraries and authenticate your LUSID client*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LUSID Environment Initialised\n",
      "LUSID API Version:  0.6.11506.0\n"
     ]
    }
   ],
   "source": [
    "# Import LUSID\n",
    "import lusid\n",
    "import lusid.models as models\n",
    "import lusid_sample_data as import_data\n",
    "from lusidjam import RefreshingToken\n",
    "\n",
    "# Import Libraries\n",
    "import pprint\n",
    "from datetime import datetime, timedelta, time\n",
    "import pytz\n",
    "import printer as prettyprint\n",
    "import pandas as pd\n",
    "import json\n",
    "import lusid\n",
    "import os\n",
    "\n",
    "# Authenticate our user and create our API client\n",
    "secrets_path = os.getenv(\"FBN_SECRETS_PATH\")\n",
    "\n",
    "api_factory = lusid.utilities.ApiClientFactory(\n",
    "    token=RefreshingToken(),\n",
    "    api_secrets_filename = secrets_path,\n",
    "    app_name=\"LusidJupyterNotebook\")\n",
    "\n",
    "\n",
    "print ('LUSID Environment Initialised')\n",
    "print ('LUSID API Version: ', api_factory.build(lusid.api.ApplicationMetadataApi).get_lusid_versions().build_version)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Init](img/migration-lusidinit.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1) Set up your Scope"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will need to create a name for the scope that you will use for the migration. Read more about scopes in the [LUSID Knowledge Base: Scopes](https://support.lusid.com/what-is-a-scope-in-lusid-and-how-is-it-used).\n",
    "\n",
    "*Run the cell below to create a name for your scope*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mScope: \u001b[0minvestment-book-of-records-3bf2-7ffd-98a6-44\n"
     ]
    }
   ],
   "source": [
    "# Generate a unique id for your scope\n",
    "scope_id = import_data.create_scope_id()\n",
    "# Set the name of your scope\n",
    "scope = 'investment-book-of-records-{}'.format(scope_id)\n",
    "prettyprint.heading('Scope', scope)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Scopes](img/migration-scopes.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Set up your Portfolio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have decided on the name for your scope, you can create your portfolio inside this scope. Note that every portfolio can be referenced by a unique code. Read more about portfolios in the [LUSID Knowledge Base: Portfolios](https://support.lusid.com/what-is-2).\n",
    "\n",
    "For further usage of the create portfolio API call refer to the [LUSID API Docs: Create Portfolio](https://docs.lusid.com/#operation/CreatePortfolio).\n",
    "\n",
    "Note that when you create the portolio in the cell below you are creating it with a 'created' date of 1052 days ago. This number is rather arbitary, in practice it should be the date the portfolio came into existence regardless of the system you first created it in, read more about the importance of the created date on a portfolio in the [LUSID Knowledge Base: Importance of Portfolio Creation Date](https://support.lusid.com/importance-of-portfolio-creation-date).\n",
    "\n",
    "*Run the cell below to create your portfolio*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mPortfolio Created\u001b[0m\n",
      "\u001b[1mScope: \u001b[0minvestment-book-of-records-3bf2-7ffd-98a6-44\n",
      "\u001b[1mCode: \u001b[0mGlobal-Strategies\n",
      "\u001b[1mPortfolio Effective From: \u001b[0m2020-08-03 16:38:35.884666+00:00\n",
      "\u001b[1mPortfolio Created On: \u001b[0m2023-06-21 16:38:39.670283+00:00\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Set the code of your portfolio\n",
    "portfolio_code = 'Global-Strategies'\n",
    "\n",
    "# Set the creation date of your portfolio \n",
    "portfolio_creation_date = datetime.now(pytz.UTC) - timedelta(days=1052)\n",
    "\n",
    "# Build your request to create your portfolio\n",
    "request = models.CreateTransactionPortfolioRequest(\n",
    "    display_name='Global Strategies Fund',\n",
    "    code=portfolio_code,\n",
    "    base_currency='GBP',\n",
    "    description=None,\n",
    "    created=portfolio_creation_date,\n",
    "    corporate_action_source_id=None,\n",
    "    accounting_method='AverageCost',\n",
    "    sub_holding_keys=None,\n",
    "    properties=None)\n",
    "\n",
    "# Call LUSID to create your portfolio\n",
    "response = api_factory.build(lusid.api.TransactionPortfoliosApi).create_portfolio(\n",
    "    scope=scope,\n",
    "    create_transaction_portfolio_request=request)\n",
    "\n",
    "# Pretty print the response\n",
    "prettyprint.portfolio_response(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Portfolio](img/migration-portfolio.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3) Add your Instrument Universe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before you can take on any holdings or make any trades you need to ensure that your instrument universe has been populated. In this case you will import your instrument universe from a CSV file. Read more about instruments in LUSID in the [LUSID Knowledge Base: Instruments](https://support.lusid.com/what-is-an-instrument).\n",
    "\n",
    "*Run the cell below to import your instrument universe*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>instrument_name</th>\n",
       "      <th>client_internal</th>\n",
       "      <th>currency</th>\n",
       "      <th>isin</th>\n",
       "      <th>figi</th>\n",
       "      <th>exchange_code</th>\n",
       "      <th>country_issue</th>\n",
       "      <th>ticker</th>\n",
       "      <th>market_sector</th>\n",
       "      <th>security_type</th>\n",
       "      <th>coupon</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Amazon_Nasdaq_AMZN</td>\n",
       "      <td>imd_34634534</td>\n",
       "      <td>USD</td>\n",
       "      <td>US0231351067</td>\n",
       "      <td>BBG000BVPXP1</td>\n",
       "      <td>UN</td>\n",
       "      <td>united_states_america</td>\n",
       "      <td>AMZN</td>\n",
       "      <td>equity</td>\n",
       "      <td>common_stock</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Apple_Nasdaq_AAPL</td>\n",
       "      <td>imd_35345345</td>\n",
       "      <td>USD</td>\n",
       "      <td>US0378331005</td>\n",
       "      <td>BBG000B9XVV8</td>\n",
       "      <td>UN</td>\n",
       "      <td>united_states_america</td>\n",
       "      <td>AAPL</td>\n",
       "      <td>equity</td>\n",
       "      <td>common_stock</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>BP_LondonStockEx_BP</td>\n",
       "      <td>imd_43535553</td>\n",
       "      <td>GBP</td>\n",
       "      <td>GB0007980591</td>\n",
       "      <td>BBG000C05BD1</td>\n",
       "      <td>LN</td>\n",
       "      <td>united_kingdom</td>\n",
       "      <td>BP/</td>\n",
       "      <td>equity</td>\n",
       "      <td>common_stock</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>BurfordCapital_LondonStockEx_BUR</td>\n",
       "      <td>imd_43534356</td>\n",
       "      <td>GBP</td>\n",
       "      <td>GG00B4L84979</td>\n",
       "      <td>BBG000PN88Q7</td>\n",
       "      <td>LN</td>\n",
       "      <td>united_kingdom</td>\n",
       "      <td>BUR</td>\n",
       "      <td>equity</td>\n",
       "      <td>common_stock</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>EKFDiagnostics_LondonStockEx_EKF</td>\n",
       "      <td>imd_34535355</td>\n",
       "      <td>GBP</td>\n",
       "      <td>GB0031509804</td>\n",
       "      <td>BBG000BVNBN3</td>\n",
       "      <td>LN</td>\n",
       "      <td>united_kingdom</td>\n",
       "      <td>EKF</td>\n",
       "      <td>equity</td>\n",
       "      <td>common_stock</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    instrument_name client_internal currency          isin  \\\n",
       "0                Amazon_Nasdaq_AMZN    imd_34634534      USD  US0231351067   \n",
       "1                 Apple_Nasdaq_AAPL    imd_35345345      USD  US0378331005   \n",
       "2               BP_LondonStockEx_BP    imd_43535553      GBP  GB0007980591   \n",
       "3  BurfordCapital_LondonStockEx_BUR    imd_43534356      GBP  GG00B4L84979   \n",
       "4  EKFDiagnostics_LondonStockEx_EKF    imd_34535355      GBP  GB0031509804   \n",
       "\n",
       "           figi exchange_code          country_issue ticker market_sector  \\\n",
       "0  BBG000BVPXP1            UN  united_states_america   AMZN        equity   \n",
       "1  BBG000B9XVV8            UN  united_states_america   AAPL        equity   \n",
       "2  BBG000C05BD1            LN         united_kingdom    BP/        equity   \n",
       "3  BBG000PN88Q7            LN         united_kingdom    BUR        equity   \n",
       "4  BBG000BVNBN3            LN         united_kingdom    EKF        equity   \n",
       "\n",
       "  security_type  coupon  \n",
       "0  common_stock     NaN  \n",
       "1  common_stock     NaN  \n",
       "2  common_stock     NaN  \n",
       "3  common_stock     NaN  \n",
       "4  common_stock     NaN  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "instrument_universe = pd.read_csv('data/instruments.csv')\n",
    "instrument_universe.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have the details for your instruments you can go ahead and create an instrument definition for each instrument. These can then be upserted into LUSID. Read about instrument definitions here [LUSID Knowledge Base: What is an Instrument?](https://support.lusid.com/what-is-an-instrument).\n",
    "\n",
    "As part of this definition you will also attach two alias identifiers to your instruments. Read more about alias identifiers here [LUSID Knowledge Base: What is an Alias Identifier?](https://support.lusid.com/alias-identifiers).\n",
    "\n",
    "You use an upsert method to add instrument definitions to the instrument universe in LUSID. Read more about the behaviour of the upsert method here [LUSID Knowledge Base: Upsert](https://support.lusid.com/upsert-command).\n",
    "\n",
    "For further usage of the upsert instruments API call refer to the [LUSID API Docs: Upserting Instruments](https://docs.lusid.com/#operation/UpsertInstruments).\n",
    "\n",
    "*Run the cell below to upsert your instruments into LUSID*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[91m\u001b[1mInstruments Successfully Upserted: \u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Instrument</th>\n",
       "      <th>ClientInternal ID</th>\n",
       "      <th>LUSID Instrument ID</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Whitebread_LondonStockEx_WTB</td>\n",
       "      <td>imd_35349900</td>\n",
       "      <td>LUID_0000MIP2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>MicroFocus_LondonStockEx_MCRO</td>\n",
       "      <td>imd_34567338</td>\n",
       "      <td>LUID_0000MIP9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>UKGiltTreasury_2.0_2025</td>\n",
       "      <td>imd_34534536</td>\n",
       "      <td>LUID_0000MIP3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Apple_Nasdaq_AAPL</td>\n",
       "      <td>imd_35345345</td>\n",
       "      <td>LUID_0000MIP0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>TESCO_LondonStockEx_TSCO</td>\n",
       "      <td>imd_34634673</td>\n",
       "      <td>LUID_0000NA4Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Kingfisher_LondonStockEx_KGF</td>\n",
       "      <td>imd_34535552</td>\n",
       "      <td>LUID_0000MIOZ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>UKGiltTreasury_3.5_2045</td>\n",
       "      <td>imd_54234532</td>\n",
       "      <td>LUID_0000MIPC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Amazon_Nasdaq_AMZN</td>\n",
       "      <td>imd_34634534</td>\n",
       "      <td>LUID_0000MIOX</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>USTreasury_2.00_2021</td>\n",
       "      <td>imd_34535347</td>\n",
       "      <td>LUID_0000MIPE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>USTreasury_6.875_2025</td>\n",
       "      <td>imd_34534539</td>\n",
       "      <td>LUID_0000MIP1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>UKGiltTreasury_4.5_2034</td>\n",
       "      <td>imd_34534534</td>\n",
       "      <td>LUID_0000MIPA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>EKFDiagnostics_LondonStockEx_EKF</td>\n",
       "      <td>imd_34535355</td>\n",
       "      <td>LUID_0000MIPB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>WPP_LondonStockEx_WPP</td>\n",
       "      <td>imd_34536734</td>\n",
       "      <td>LUID_0000MIPF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Sage_LondonStockEx_SGE</td>\n",
       "      <td>imd_23536673</td>\n",
       "      <td>LUID_0000MIP5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>BurfordCapital_LondonStockEx_BUR</td>\n",
       "      <td>imd_43534356</td>\n",
       "      <td>LUID_0000MIOY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>JustEat_LondonStockEx_JE</td>\n",
       "      <td>imd_35436366</td>\n",
       "      <td>LUID_0000MIP4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Glencore_LondonStockEx_GLEN</td>\n",
       "      <td>imd_34534555</td>\n",
       "      <td>LUID_0000MIPD</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>UKGiltTreasury_3.75_2021</td>\n",
       "      <td>imd_34643653</td>\n",
       "      <td>LUID_0000MIP6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>RELXGroup_LondonStockEx_REL</td>\n",
       "      <td>imd_43532542</td>\n",
       "      <td>LUID_0000MIOW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>BP_LondonStockEx_BP</td>\n",
       "      <td>imd_43535553</td>\n",
       "      <td>LUID_00044OXX</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          Instrument ClientInternal ID LUSID Instrument ID\n",
       "0       Whitebread_LondonStockEx_WTB      imd_35349900       LUID_0000MIP2\n",
       "1      MicroFocus_LondonStockEx_MCRO      imd_34567338       LUID_0000MIP9\n",
       "2            UKGiltTreasury_2.0_2025      imd_34534536       LUID_0000MIP3\n",
       "3                  Apple_Nasdaq_AAPL      imd_35345345       LUID_0000MIP0\n",
       "4           TESCO_LondonStockEx_TSCO      imd_34634673       LUID_0000NA4Q\n",
       "5       Kingfisher_LondonStockEx_KGF      imd_34535552       LUID_0000MIOZ\n",
       "6            UKGiltTreasury_3.5_2045      imd_54234532       LUID_0000MIPC\n",
       "7                 Amazon_Nasdaq_AMZN      imd_34634534       LUID_0000MIOX\n",
       "8               USTreasury_2.00_2021      imd_34535347       LUID_0000MIPE\n",
       "9              USTreasury_6.875_2025      imd_34534539       LUID_0000MIP1\n",
       "10           UKGiltTreasury_4.5_2034      imd_34534534       LUID_0000MIPA\n",
       "11  EKFDiagnostics_LondonStockEx_EKF      imd_34535355       LUID_0000MIPB\n",
       "12             WPP_LondonStockEx_WPP      imd_34536734       LUID_0000MIPF\n",
       "13            Sage_LondonStockEx_SGE      imd_23536673       LUID_0000MIP5\n",
       "14  BurfordCapital_LondonStockEx_BUR      imd_43534356       LUID_0000MIOY\n",
       "15          JustEat_LondonStockEx_JE      imd_35436366       LUID_0000MIP4\n",
       "16       Glencore_LondonStockEx_GLEN      imd_34534555       LUID_0000MIPD\n",
       "17          UKGiltTreasury_3.75_2021      imd_34643653       LUID_0000MIP6\n",
       "18       RELXGroup_LondonStockEx_REL      imd_43532542       LUID_0000MIOW\n",
       "19               BP_LondonStockEx_BP      imd_43535553       LUID_00044OXX"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Initialise your batch upsert request\n",
    "batch_upsert_request = {}\n",
    "\n",
    "# Using your instrument universe create your batch request\n",
    "for index, instrument in instrument_universe.iterrows():\n",
    "\n",
    "    # Specify the columns of your identifiers\n",
    "    identifier_columns = [\n",
    "            ('isin', 'Isin'), \n",
    "            ('figi', 'Figi'), \n",
    "            ('ticker', 'Ticker'),\n",
    "            ('client_internal', 'ClientInternal')\n",
    "    ]\n",
    "    \n",
    "    # Create your identifiers\n",
    "    identifiers = {}\n",
    "    for identifier in identifier_columns:\n",
    "        identifiers[identifier[1]] = models.InstrumentIdValue(\n",
    "            value=instrument[identifier[0]])\n",
    "    \n",
    "    # Build your request and add it to the dictionary\n",
    "    batch_upsert_request[instrument['instrument_name']] = models.InstrumentDefinition(\n",
    "        name=instrument['instrument_name'],\n",
    "        identifiers=identifiers)\n",
    "    \n",
    "# Call LUSID to upsert your instrument defintions\n",
    "instrument_response = api_factory.build(lusid.api.InstrumentsApi).upsert_instruments(request_body=batch_upsert_request)\n",
    "\n",
    "# Pretty print the response\n",
    "prettyprint.instrument_response(instrument_response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Instruments](img/migration-instruments.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4) Set your Initial Holdings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have your instrument universe populated you can load your initial holdings into your portfolio. In this case you will import your holdings from a CSV file. \n",
    "\n",
    "*Run the cell below to import your take on balances*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>portfolio_code</th>\n",
       "      <th>instrument_name</th>\n",
       "      <th>quantity</th>\n",
       "      <th>price</th>\n",
       "      <th>currency</th>\n",
       "      <th>figi</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Global-Strategies</td>\n",
       "      <td>GBP_Cash</td>\n",
       "      <td>8336000</td>\n",
       "      <td>1.00</td>\n",
       "      <td>GBP</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Global-Strategies</td>\n",
       "      <td>Glencore_LondonStockEx_GLEN</td>\n",
       "      <td>905141</td>\n",
       "      <td>2.76</td>\n",
       "      <td>GBP</td>\n",
       "      <td>BBG001MM1KV4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Global-Strategies</td>\n",
       "      <td>Kingfisher_LondonStockEx_KGF</td>\n",
       "      <td>1362038</td>\n",
       "      <td>2.28</td>\n",
       "      <td>GBP</td>\n",
       "      <td>BBG000BKH1W6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Global-Strategies</td>\n",
       "      <td>UKGiltTreasury_2.0_2025</td>\n",
       "      <td>405589</td>\n",
       "      <td>106.64</td>\n",
       "      <td>GBP</td>\n",
       "      <td>BBG0088JSC32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Global-Strategies</td>\n",
       "      <td>UKGiltTreasury_3.5_2045</td>\n",
       "      <td>266169</td>\n",
       "      <td>134.43</td>\n",
       "      <td>GBP</td>\n",
       "      <td>BBG006N6HZM7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      portfolio_code               instrument_name  quantity  price currency  \\\n",
       "0  Global-Strategies                      GBP_Cash   8336000   1.00      GBP   \n",
       "1  Global-Strategies   Glencore_LondonStockEx_GLEN    905141   2.76      GBP   \n",
       "2  Global-Strategies  Kingfisher_LondonStockEx_KGF   1362038   2.28      GBP   \n",
       "3  Global-Strategies       UKGiltTreasury_2.0_2025    405589 106.64      GBP   \n",
       "4  Global-Strategies       UKGiltTreasury_3.5_2045    266169 134.43      GBP   \n",
       "\n",
       "           figi  \n",
       "0           NaN  \n",
       "1  BBG001MM1KV4  \n",
       "2  BBG000BKH1W6  \n",
       "3  BBG0088JSC32  \n",
       "4  BBG006N6HZM7  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "holdings = pd.read_csv('data/demo-holdings.csv')\n",
    "holdings.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have imported your holdings you can add them to LUSID. You can do this by setting the holdings on a portfolio. Read more about how making an adjustment or setting the holdings on a portfolio affects it here [LUSID Knowledge Base: The effect of holding adjustments](https://support.lusid.com/how-do-holding-adjustments-affect-a-portfolio).\n",
    "\n",
    "For further usage of the set holdings API call refer to the [LUSID API Docs: Set Holdings](https://docs.lusid.com/#operation/SetHoldings).\n",
    "\n",
    "*Run the cell below to upsert your holdings into LUSID*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mHoldings Successfully Set for Portfolio\u001b[0m\n",
      "\u001b[1mScope: \u001b[0minvestment-book-of-records-3bf2-7ffd-98a6-44\n",
      "\u001b[1mCode: \u001b[0mGlobal-Strategies\n",
      "\u001b[1mHoldings Effective From: \u001b[0m2020-08-03 16:38:35.884666+00:00\n",
      "\u001b[1mHoldings Created On: \u001b[0m2023-06-21 16:38:46.758672+00:00\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Make the holdings effective from two days ago\n",
    "holdings_effective_date = datetime.now(pytz.UTC) - timedelta(days=2)\n",
    "\n",
    "# Iterate the portfolios in the holdings CSV, note in this case you only have one\n",
    "for portfolio in holdings['portfolio_code'].unique():\n",
    "    # Initialise a list to hold your adjustments\n",
    "    holding_adjustments = []\n",
    "    \n",
    "    # Iterate over the holdings in each portfolio\n",
    "    for index, holding in holdings.loc[holdings['portfolio_code'] == portfolio].iterrows():\n",
    "        \n",
    "        # Set your instrument identifiers based on whether or not instrument is cash\n",
    "        if 'Cash' in holding['instrument_name']:\n",
    "            identifier_key = 'Instrument/default/Currency'\n",
    "            identifer = holding['instrument_name'].split('_')[0]\n",
    "        else:\n",
    "            identifier_key = 'Instrument/default/Figi'\n",
    "            identifer = holding['figi']\n",
    "            \n",
    "        # Create your holding adjustment and append it to your list\n",
    "        holding_adjustments.append(\n",
    "            models.AdjustHoldingRequest(\n",
    "                instrument_identifiers={\n",
    "                    identifier_key: identifer},\n",
    "                tax_lots=[\n",
    "                    models.TargetTaxLotRequest(\n",
    "                        units=holding['quantity'],\n",
    "                        cost=models.CurrencyAndAmount(\n",
    "                            amount=holding['quantity'] * holding['price'],\n",
    "                            currency=holding['currency']),\n",
    "                        portfolio_cost=holding['quantity'] * holding['price'],\n",
    "                        price=holding['price'])\n",
    "                ]\n",
    "            )\n",
    "        )\n",
    "    \n",
    "    # Call LUSID to set your initial holdings\n",
    "    response = api_factory.build(lusid.api.TransactionPortfoliosApi).set_holdings(\n",
    "        scope=scope,\n",
    "        code=portfolio,\n",
    "        effective_at=holdings_effective_date,\n",
    "        adjust_holding_request=holding_adjustments)\n",
    "\n",
    "    # Pretty print our response from LUSID\n",
    "    prettyprint.set_holdings_response(response, scope, portfolio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Instruments](img/migration-holdingsadded.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5) Get your Output Transactions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see how LUSID has adjusted the holdings of our portfolio to meet our initial holdings by generating the output transactions from the portfolio. You can read more about output transactions here [LUSID Knowledge Base: Output Transactions](https://support.lusid.com/what-is-an-output-transaction-in-lusid).\n",
    "\n",
    "For further usage of the build transactions API call refer to the [LUSID API Docs: Build Output Transactions](https://docs.lusid.com/#operation/BuildTransactions).\n",
    "\n",
    "*Run the cell below to generate your output transactions*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[91m\u001b[1mOutput Transactions for Portfolio\u001b[0m\n",
      "\u001b[1mScope: \u001b[0minvestment-book-of-records-3bf2-7ffd-98a6-44\n",
      "\u001b[1mCode: \u001b[0mGlobal-Strategies\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Transaction ID</th>\n",
       "      <th>Transaction Type</th>\n",
       "      <th>Instrument/default/Name</th>\n",
       "      <th>Units</th>\n",
       "      <th>Price</th>\n",
       "      <th>Currency</th>\n",
       "      <th>Transaction Date</th>\n",
       "      <th>Settlement Date</th>\n",
       "      <th>Realised Gain Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2023-06-19T16:38:45.7175870+00:00</td>\n",
       "      <td>AdjustmentIncrease</td>\n",
       "      <td>GBP</td>\n",
       "      <td>8336000.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>GBP</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2023-06-19T16:38:45.7175870+00:00</td>\n",
       "      <td>AdjustmentIncrease</td>\n",
       "      <td>Glencore_LondonStockEx_GLEN</td>\n",
       "      <td>905141.00</td>\n",
       "      <td>2.76</td>\n",
       "      <td>GBP</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2023-06-19T16:38:45.7175870+00:00</td>\n",
       "      <td>AdjustmentIncrease</td>\n",
       "      <td>Kingfisher_LondonStockEx_KGF</td>\n",
       "      <td>1362038.00</td>\n",
       "      <td>2.28</td>\n",
       "      <td>GBP</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2023-06-19T16:38:45.7175870+00:00</td>\n",
       "      <td>AdjustmentIncrease</td>\n",
       "      <td>UKGiltTreasury_2.0_2025</td>\n",
       "      <td>405589.00</td>\n",
       "      <td>106.64</td>\n",
       "      <td>GBP</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2023-06-19T16:38:45.7175870+00:00</td>\n",
       "      <td>AdjustmentIncrease</td>\n",
       "      <td>UKGiltTreasury_3.5_2045</td>\n",
       "      <td>266169.00</td>\n",
       "      <td>134.43</td>\n",
       "      <td>GBP</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2023-06-19T16:38:45.7175870+00:00</td>\n",
       "      <td>AdjustmentIncrease</td>\n",
       "      <td>UKGiltTreasury_3.75_2021</td>\n",
       "      <td>661713.00</td>\n",
       "      <td>108.13</td>\n",
       "      <td>GBP</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2023-06-19T16:38:45.7175870+00:00</td>\n",
       "      <td>AdjustmentIncrease</td>\n",
       "      <td>UKGiltTreasury_4.5_2034</td>\n",
       "      <td>77481.00</td>\n",
       "      <td>140.57</td>\n",
       "      <td>GBP</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2023-06-19T16:38:45.7175870+00:00</td>\n",
       "      <td>AdjustmentIncrease</td>\n",
       "      <td>USTreasury_2.00_2021</td>\n",
       "      <td>1440244.00</td>\n",
       "      <td>97.90</td>\n",
       "      <td>USD</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2023-06-19T16:38:45.7175870+00:00</td>\n",
       "      <td>AdjustmentIncrease</td>\n",
       "      <td>BurfordCapital_LondonStockEx_BUR</td>\n",
       "      <td>853486.00</td>\n",
       "      <td>14.06</td>\n",
       "      <td>GBP</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2023-06-19T16:38:45.7175870+00:00</td>\n",
       "      <td>AdjustmentIncrease</td>\n",
       "      <td>EKFDiagnostics_LondonStockEx_EKF</td>\n",
       "      <td>925925.00</td>\n",
       "      <td>0.27</td>\n",
       "      <td>GBP</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2023-06-19T16:38:45.7175870+00:00</td>\n",
       "      <td>AdjustmentIncrease</td>\n",
       "      <td>JustEat_LondonStockEx_JE</td>\n",
       "      <td>4860907.00</td>\n",
       "      <td>5.46</td>\n",
       "      <td>GBP</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2023-06-19T16:38:45.7175870+00:00</td>\n",
       "      <td>AdjustmentIncrease</td>\n",
       "      <td>MicroFocus_LondonStockEx_MCRO</td>\n",
       "      <td>687994.00</td>\n",
       "      <td>14.54</td>\n",
       "      <td>GBP</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>2023-06-19T16:38:45.7175870+00:00</td>\n",
       "      <td>AdjustmentIncrease</td>\n",
       "      <td>RELXGroup_LondonStockEx_REL</td>\n",
       "      <td>494343.00</td>\n",
       "      <td>15.98</td>\n",
       "      <td>GBP</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>2023-06-19T16:38:45.7175870+00:00</td>\n",
       "      <td>AdjustmentIncrease</td>\n",
       "      <td>Sage_LondonStockEx_SGE</td>\n",
       "      <td>2599653.00</td>\n",
       "      <td>5.77</td>\n",
       "      <td>GBP</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>2023-06-19T16:38:45.7175870+00:00</td>\n",
       "      <td>AdjustmentIncrease</td>\n",
       "      <td>WPP_LondonStockEx_WPP</td>\n",
       "      <td>2956000.00</td>\n",
       "      <td>8.71</td>\n",
       "      <td>GBP</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>2023-06-19T16:38:45.7175870+00:00</td>\n",
       "      <td>AdjustmentIncrease</td>\n",
       "      <td>USD</td>\n",
       "      <td>23000000.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>USD</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>2023-06-19T16:38:45.7175870+00:00</td>\n",
       "      <td>AdjustmentIncrease</td>\n",
       "      <td>USTreasury_6.875_2025</td>\n",
       "      <td>534049.00</td>\n",
       "      <td>124.52</td>\n",
       "      <td>USD</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>2023-06-19 16:38:45.717587+00:00</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>50366732.00</td>\n",
       "      <td>784.01</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>NaT</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       Transaction ID    Transaction Type  \\\n",
       "0   2023-06-19T16:38:45.7175870+00:00  AdjustmentIncrease   \n",
       "1   2023-06-19T16:38:45.7175870+00:00  AdjustmentIncrease   \n",
       "2   2023-06-19T16:38:45.7175870+00:00  AdjustmentIncrease   \n",
       "3   2023-06-19T16:38:45.7175870+00:00  AdjustmentIncrease   \n",
       "4   2023-06-19T16:38:45.7175870+00:00  AdjustmentIncrease   \n",
       "5   2023-06-19T16:38:45.7175870+00:00  AdjustmentIncrease   \n",
       "6   2023-06-19T16:38:45.7175870+00:00  AdjustmentIncrease   \n",
       "7   2023-06-19T16:38:45.7175870+00:00  AdjustmentIncrease   \n",
       "8   2023-06-19T16:38:45.7175870+00:00  AdjustmentIncrease   \n",
       "9   2023-06-19T16:38:45.7175870+00:00  AdjustmentIncrease   \n",
       "10  2023-06-19T16:38:45.7175870+00:00  AdjustmentIncrease   \n",
       "11  2023-06-19T16:38:45.7175870+00:00  AdjustmentIncrease   \n",
       "12  2023-06-19T16:38:45.7175870+00:00  AdjustmentIncrease   \n",
       "13  2023-06-19T16:38:45.7175870+00:00  AdjustmentIncrease   \n",
       "14  2023-06-19T16:38:45.7175870+00:00  AdjustmentIncrease   \n",
       "15  2023-06-19T16:38:45.7175870+00:00  AdjustmentIncrease   \n",
       "16  2023-06-19T16:38:45.7175870+00:00  AdjustmentIncrease   \n",
       "17                                NaN                 NaN   \n",
       "\n",
       "             Instrument/default/Name       Units  Price Currency  \\\n",
       "0                                GBP  8336000.00   1.00      GBP   \n",
       "1        Glencore_LondonStockEx_GLEN   905141.00   2.76      GBP   \n",
       "2       Kingfisher_LondonStockEx_KGF  1362038.00   2.28      GBP   \n",
       "3            UKGiltTreasury_2.0_2025   405589.00 106.64      GBP   \n",
       "4            UKGiltTreasury_3.5_2045   266169.00 134.43      GBP   \n",
       "5           UKGiltTreasury_3.75_2021   661713.00 108.13      GBP   \n",
       "6            UKGiltTreasury_4.5_2034    77481.00 140.57      GBP   \n",
       "7               USTreasury_2.00_2021  1440244.00  97.90      USD   \n",
       "8   BurfordCapital_LondonStockEx_BUR   853486.00  14.06      GBP   \n",
       "9   EKFDiagnostics_LondonStockEx_EKF   925925.00   0.27      GBP   \n",
       "10          JustEat_LondonStockEx_JE  4860907.00   5.46      GBP   \n",
       "11     MicroFocus_LondonStockEx_MCRO   687994.00  14.54      GBP   \n",
       "12       RELXGroup_LondonStockEx_REL   494343.00  15.98      GBP   \n",
       "13            Sage_LondonStockEx_SGE  2599653.00   5.77      GBP   \n",
       "14             WPP_LondonStockEx_WPP  2956000.00   8.71      GBP   \n",
       "15                               USD 23000000.00   1.00      USD   \n",
       "16             USTreasury_6.875_2025   534049.00 124.52      USD   \n",
       "17                               NaN 50366732.00 784.01      NaN   \n",
       "\n",
       "                   Transaction Date                  Settlement Date  \\\n",
       "0  2023-06-19 16:38:45.717587+00:00 2023-06-19 16:38:45.717587+00:00   \n",
       "1  2023-06-19 16:38:45.717587+00:00 2023-06-19 16:38:45.717587+00:00   \n",
       "2  2023-06-19 16:38:45.717587+00:00 2023-06-19 16:38:45.717587+00:00   \n",
       "3  2023-06-19 16:38:45.717587+00:00 2023-06-19 16:38:45.717587+00:00   \n",
       "4  2023-06-19 16:38:45.717587+00:00 2023-06-19 16:38:45.717587+00:00   \n",
       "5  2023-06-19 16:38:45.717587+00:00 2023-06-19 16:38:45.717587+00:00   \n",
       "6  2023-06-19 16:38:45.717587+00:00 2023-06-19 16:38:45.717587+00:00   \n",
       "7  2023-06-19 16:38:45.717587+00:00 2023-06-19 16:38:45.717587+00:00   \n",
       "8  2023-06-19 16:38:45.717587+00:00 2023-06-19 16:38:45.717587+00:00   \n",
       "9  2023-06-19 16:38:45.717587+00:00 2023-06-19 16:38:45.717587+00:00   \n",
       "10 2023-06-19 16:38:45.717587+00:00 2023-06-19 16:38:45.717587+00:00   \n",
       "11 2023-06-19 16:38:45.717587+00:00 2023-06-19 16:38:45.717587+00:00   \n",
       "12 2023-06-19 16:38:45.717587+00:00 2023-06-19 16:38:45.717587+00:00   \n",
       "13 2023-06-19 16:38:45.717587+00:00 2023-06-19 16:38:45.717587+00:00   \n",
       "14 2023-06-19 16:38:45.717587+00:00 2023-06-19 16:38:45.717587+00:00   \n",
       "15 2023-06-19 16:38:45.717587+00:00 2023-06-19 16:38:45.717587+00:00   \n",
       "16 2023-06-19 16:38:45.717587+00:00 2023-06-19 16:38:45.717587+00:00   \n",
       "17                              NaT                              NaT   \n",
       "\n",
       "    Realised Gain Loss  \n",
       "0                  NaN  \n",
       "1                  NaN  \n",
       "2                  NaN  \n",
       "3                  NaN  \n",
       "4                  NaN  \n",
       "5                  NaN  \n",
       "6                  NaN  \n",
       "7                  NaN  \n",
       "8                  NaN  \n",
       "9                  NaN  \n",
       "10                 NaN  \n",
       "11                 NaN  \n",
       "12                 NaN  \n",
       "13                 NaN  \n",
       "14                 NaN  \n",
       "15                 NaN  \n",
       "16                 NaN  \n",
       "17                0.00  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set our query parameters to build your transactions\n",
    "query_params = models.TransactionQueryParameters(\n",
    "    start_date=(holdings_effective_date-timedelta(days=4)).isoformat(),\n",
    "    end_date=(holdings_effective_date+timedelta(days=4)).isoformat(),\n",
    "    query_mode='TradeDate',\n",
    "    show_cancelled_transactions=None)\n",
    "\n",
    "# Call LUSID to build your output transactions\n",
    "response = api_factory.build(lusid.api.TransactionPortfoliosApi).build_transactions(\n",
    "    scope=scope,\n",
    "    code=portfolio_code,\n",
    "    property_keys=['Instrument/default/Name'],\n",
    "    transaction_query_parameters=query_params\n",
    ")\n",
    "\n",
    "# Pretty print your output transactions\n",
    "prettyprint.output_transactions(response, scope, portfolio_code, ['Instrument/default/Name'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Output Transactions](img/migration-outputtransactions.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6) Set your Transaction Type Configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have set your initial holdings you are ready to take on some transactions. You have made several trades over the last day that you would like to add to LUSID. You will import these from a CSV file. \n",
    "\n",
    "*Run the cell below to import your transactions*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>portfolio_code</th>\n",
       "      <th>transaction_id</th>\n",
       "      <th>instrument_name</th>\n",
       "      <th>transaction_description</th>\n",
       "      <th>transaction_type</th>\n",
       "      <th>transaction_units</th>\n",
       "      <th>transaction_price</th>\n",
       "      <th>transaction_currency</th>\n",
       "      <th>transaction_strategy</th>\n",
       "      <th>transaction_cost</th>\n",
       "      <th>figi</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Global-Strategies</td>\n",
       "      <td>tid_329432525234324</td>\n",
       "      <td>Kingfisher_LondonStockEx_KGF</td>\n",
       "      <td>Equity Sale</td>\n",
       "      <td>Sell</td>\n",
       "      <td>325000</td>\n",
       "      <td>2.35</td>\n",
       "      <td>GBP</td>\n",
       "      <td>quantitativeSignal</td>\n",
       "      <td>762125.00</td>\n",
       "      <td>BBG000BKH1W6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Global-Strategies</td>\n",
       "      <td>tid_325452342424500</td>\n",
       "      <td>UKGiltTreasury_4.5_2034</td>\n",
       "      <td>Equity Purchase</td>\n",
       "      <td>Buy</td>\n",
       "      <td>10501</td>\n",
       "      <td>140.57</td>\n",
       "      <td>GBP</td>\n",
       "      <td>incomeRequirements</td>\n",
       "      <td>1476146.57</td>\n",
       "      <td>BBG0000D14P3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Global-Strategies</td>\n",
       "      <td>tid_234295929052090</td>\n",
       "      <td>UKGiltTreasury_3.75_2021</td>\n",
       "      <td>Equity Purchase</td>\n",
       "      <td>Buy</td>\n",
       "      <td>24000</td>\n",
       "      <td>109.13</td>\n",
       "      <td>GBP</td>\n",
       "      <td>incomeRequirements</td>\n",
       "      <td>2619024.00</td>\n",
       "      <td>BBG001KKJLR4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Global-Strategies</td>\n",
       "      <td>tid_234942982496001</td>\n",
       "      <td>USTreasury_2.00_2021</td>\n",
       "      <td>Equity Sale</td>\n",
       "      <td>Sell</td>\n",
       "      <td>57000</td>\n",
       "      <td>97.80</td>\n",
       "      <td>USD</td>\n",
       "      <td>internationalExposure</td>\n",
       "      <td>5574600.00</td>\n",
       "      <td>BBG00FN3B5K8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Global-Strategies</td>\n",
       "      <td>tid_121309590059995</td>\n",
       "      <td>BP_LondonStockEx_BP</td>\n",
       "      <td>Equity Purchase</td>\n",
       "      <td>Buy</td>\n",
       "      <td>50000</td>\n",
       "      <td>5.29</td>\n",
       "      <td>GBP</td>\n",
       "      <td>fundamentalAnalysis</td>\n",
       "      <td>264350.00</td>\n",
       "      <td>BBG000C05BD1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      portfolio_code       transaction_id               instrument_name  \\\n",
       "0  Global-Strategies  tid_329432525234324  Kingfisher_LondonStockEx_KGF   \n",
       "1  Global-Strategies  tid_325452342424500       UKGiltTreasury_4.5_2034   \n",
       "2  Global-Strategies  tid_234295929052090      UKGiltTreasury_3.75_2021   \n",
       "3  Global-Strategies  tid_234942982496001          USTreasury_2.00_2021   \n",
       "4  Global-Strategies  tid_121309590059995           BP_LondonStockEx_BP   \n",
       "\n",
       "  transaction_description transaction_type  transaction_units  \\\n",
       "0             Equity Sale             Sell             325000   \n",
       "1         Equity Purchase              Buy              10501   \n",
       "2         Equity Purchase              Buy              24000   \n",
       "3             Equity Sale             Sell              57000   \n",
       "4         Equity Purchase              Buy              50000   \n",
       "\n",
       "   transaction_price transaction_currency   transaction_strategy  \\\n",
       "0               2.35                  GBP     quantitativeSignal   \n",
       "1             140.57                  GBP     incomeRequirements   \n",
       "2             109.13                  GBP     incomeRequirements   \n",
       "3              97.80                  USD  internationalExposure   \n",
       "4               5.29                  GBP    fundamentalAnalysis   \n",
       "\n",
       "   transaction_cost          figi  \n",
       "0         762125.00  BBG000BKH1W6  \n",
       "1        1476146.57  BBG0000D14P3  \n",
       "2        2619024.00  BBG001KKJLR4  \n",
       "3        5574600.00  BBG00FN3B5K8  \n",
       "4         264350.00  BBG000C05BD1  "
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "daily_transactions = pd.read_csv('data/demo-transactions.csv')\n",
    "daily_transactions.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before you can add these trades to LUSID you need to set up a transaction type configuration. This allows the LUSID movement engine to correctly decompose each transaction into its underlying economic movements. Read more about configuring transaction types here [LUSID Knowledge Base: Configuring Transaction Types](https://support.lusid.com/configuring-transaction-types) and movements in LUSID here [LUSID Knowledge Base: Movements](https://support.lusid.com/what-is-a-movement-in-lusid). \n",
    "\n",
    "You will import the default LUSID transaction type configuration from a JSON file and then use this to set your configuration inside LUSID.\n",
    "\n",
    "*Run the cell below to import the LUSID default transaction mapping configuration*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'transactionConfigRequests': [{'aliases': [{'description': 'Purchase',\n",
      "                                             'transactionClass': 'Basic',\n",
      "                                             'transactionGroup': 'default',\n",
      "                                             'transactionRoles': 'LongLonger',\n",
      "                                             'type': 'Buy'},\n",
      "                                            {'description': 'PURCHASE',\n",
      "                                             'transactionClass': 'Basic',\n",
      "                                             'transactionGroup': 'alt1',\n",
      "                                             'transactionRoles': 'LongLonger',\n",
      "                                             'type': 'BY'}],\n",
      "                                'movements': [{'direction': 1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'StockMovement',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side1'},\n",
      "                                              {'direction': -1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'CashCommitment',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side2'}],\n",
      "                                'properties': []},\n",
      "                               {'aliases': [{'description': 'Sale',\n",
      "                                             'transactionClass': 'Basic',\n",
      "                                             'transactionGroup': 'default',\n",
      "                                             'transactionRoles': 'LongShorter',\n",
      "                                             'type': 'Sell'},\n",
      "                                            {'description': 'SALE',\n",
      "                                             'transactionClass': 'Basic',\n",
      "                                             'transactionGroup': 'alt1',\n",
      "                                             'transactionRoles': 'LongShorter',\n",
      "                                             'type': 'SL'}],\n",
      "                                'movements': [{'direction': -1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'StockMovement',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side1'},\n",
      "                                              {'direction': 1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'CashCommitment',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side2'}],\n",
      "                                'properties': []},\n",
      "                               {'aliases': [{'description': 'Cover Short ',\n",
      "                                             'transactionClass': 'Basic',\n",
      "                                             'transactionGroup': 'default',\n",
      "                                             'transactionRoles': 'ShortLonger',\n",
      "                                             'type': 'CoverShort'},\n",
      "                                            {'description': 'COVER SHORT ',\n",
      "                                             'transactionClass': 'Basic',\n",
      "                                             'transactionGroup': 'alt1',\n",
      "                                             'transactionRoles': 'ShortLonger',\n",
      "                                             'type': 'CS'}],\n",
      "                                'movements': [{'direction': 1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'StockMovement',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side1'},\n",
      "                                              {'direction': -1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'CashCommitment',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side2'}],\n",
      "                                'properties': []},\n",
      "                               {'aliases': [{'description': 'Sell Short',\n",
      "                                             'transactionClass': 'Basic',\n",
      "                                             'transactionGroup': 'default',\n",
      "                                             'transactionRoles': 'ShortShorter',\n",
      "                                             'type': 'SellShort'},\n",
      "                                            {'description': 'SELL SHORT',\n",
      "                                             'transactionClass': 'Basic',\n",
      "                                             'transactionGroup': 'alt1',\n",
      "                                             'transactionRoles': 'ShortShorter',\n",
      "                                             'type': 'SS'}],\n",
      "                                'movements': [{'direction': -1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'StockMovement',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side1'},\n",
      "                                              {'direction': 1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'CashCommitment',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side2'}],\n",
      "                                'properties': []},\n",
      "                               {'aliases': [{'description': 'Transfer In',\n",
      "                                             'transactionClass': 'StockTransfers',\n",
      "                                             'transactionGroup': 'default',\n",
      "                                             'transactionRoles': 'Longer',\n",
      "                                             'type': 'StockIn'},\n",
      "                                            {'description': 'SECURITY IN',\n",
      "                                             'transactionClass': 'StockTransfers',\n",
      "                                             'transactionGroup': 'alt1',\n",
      "                                             'transactionRoles': 'LongLonger',\n",
      "                                             'type': 'SI'},\n",
      "                                            {'description': 'SHORT SECURITY '\n",
      "                                                            'OUT',\n",
      "                                             'transactionClass': 'StockTransfers',\n",
      "                                             'transactionGroup': 'alt1',\n",
      "                                             'transactionRoles': 'ShortLonger',\n",
      "                                             'type': 'SW'}],\n",
      "                                'movements': [{'direction': 1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'StockMovement',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side1'}],\n",
      "                                'properties': []},\n",
      "                               {'aliases': [{'description': 'Transfer Out',\n",
      "                                             'transactionClass': 'StockTransfers',\n",
      "                                             'transactionGroup': 'default',\n",
      "                                             'transactionRoles': 'Shorter',\n",
      "                                             'type': 'StockOut'},\n",
      "                                            {'description': 'SECURITY OUT',\n",
      "                                             'transactionClass': 'StockTransfers',\n",
      "                                             'transactionGroup': 'alt1',\n",
      "                                             'transactionRoles': 'LongShorter',\n",
      "                                             'type': 'SO'},\n",
      "                                            {'description': 'SHORT SECURITY IN',\n",
      "                                             'transactionClass': 'StockTransfers',\n",
      "                                             'transactionGroup': 'alt1',\n",
      "                                             'transactionRoles': 'ShortShorter',\n",
      "                                             'type': 'SA'}],\n",
      "                                'movements': [{'direction': -1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'StockMovement',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side1'}],\n",
      "                                'properties': []},\n",
      "                               {'aliases': [{'description': 'Spot FX Purchase',\n",
      "                                             'transactionClass': 'SpotFx',\n",
      "                                             'transactionGroup': 'default',\n",
      "                                             'transactionRoles': 'Longer',\n",
      "                                             'type': 'FxBuy'}],\n",
      "                                'movements': [{'direction': 1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'CashCommitment',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side1'},\n",
      "                                              {'direction': -1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'CashCommitment',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side2'}],\n",
      "                                'properties': []},\n",
      "                               {'aliases': [{'description': 'Spot FX Sale',\n",
      "                                             'transactionClass': 'SpotFx',\n",
      "                                             'transactionGroup': 'default',\n",
      "                                             'transactionRoles': 'Shorter',\n",
      "                                             'type': 'FxSell'},\n",
      "                                            {'description': 'SPOT FX',\n",
      "                                             'transactionClass': 'SpotFx',\n",
      "                                             'transactionGroup': 'alt1',\n",
      "                                             'transactionRoles': 'Longer',\n",
      "                                             'type': 'SF'}],\n",
      "                                'movements': [{'direction': -1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'CashCommitment',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side1'},\n",
      "                                              {'direction': 1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'CashCommitment',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side2'}],\n",
      "                                'properties': []},\n",
      "                               {'aliases': [{'description': 'Forward FX '\n",
      "                                                            'Purchase',\n",
      "                                             'transactionClass': 'FwdFx',\n",
      "                                             'transactionGroup': 'default',\n",
      "                                             'transactionRoles': 'Longer',\n",
      "                                             'type': 'FwdFxBuy'}],\n",
      "                                'movements': [{'direction': 1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'CashForward',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side1'},\n",
      "                                              {'direction': -1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'CashForward',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side2'}],\n",
      "                                'properties': []},\n",
      "                               {'aliases': [{'description': 'Forward FX Sale',\n",
      "                                             'transactionClass': 'FwdFx',\n",
      "                                             'transactionGroup': 'default',\n",
      "                                             'transactionRoles': 'Shorter',\n",
      "                                             'type': 'FwdFxSell'},\n",
      "                                            {'description': 'FORWARD FX',\n",
      "                                             'transactionClass': 'FwdFx',\n",
      "                                             'transactionGroup': 'alt1',\n",
      "                                             'transactionRoles': 'Shorter',\n",
      "                                             'type': 'FF'}],\n",
      "                                'movements': [{'direction': -1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'CashForward',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side1'},\n",
      "                                              {'direction': 1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'CashForward',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side2'}],\n",
      "                                'properties': []},\n",
      "                               {'aliases': [{'description': 'Deposit New Funds',\n",
      "                                             'transactionClass': 'CashTransfers',\n",
      "                                             'transactionGroup': 'default',\n",
      "                                             'transactionRoles': 'Longer',\n",
      "                                             'type': 'FundsIn'},\n",
      "                                            {'description': 'FUNDS IN',\n",
      "                                             'transactionClass': 'CashTransfers',\n",
      "                                             'transactionGroup': 'alt1',\n",
      "                                             'transactionRoles': 'Longer',\n",
      "                                             'type': 'FI'}],\n",
      "                                'movements': [{'direction': 1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'CashAccrual',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side1'}],\n",
      "                                'properties': []},\n",
      "                               {'aliases': [{'description': 'Withdraw Funds',\n",
      "                                             'transactionClass': 'CashTransfers',\n",
      "                                             'transactionGroup': 'default',\n",
      "                                             'transactionRoles': 'Shorter',\n",
      "                                             'type': 'FundsOut'},\n",
      "                                            {'description': 'FUNDS OUT',\n",
      "                                             'transactionClass': 'CashTransfers',\n",
      "                                             'transactionGroup': 'alt1',\n",
      "                                             'transactionRoles': 'Shorter',\n",
      "                                             'type': 'FO'}],\n",
      "                                'movements': [{'direction': -1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'CashAccrual',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side1'}],\n",
      "                                'properties': []},\n",
      "                               {'aliases': [{'description': 'Open Long Futures '\n",
      "                                                            'Contract',\n",
      "                                             'transactionClass': 'Futures',\n",
      "                                             'transactionGroup': 'default',\n",
      "                                             'transactionRoles': 'LongLonger',\n",
      "                                             'type': 'OLFC'},\n",
      "                                            {'description': 'OPEN LONG FUTURES '\n",
      "                                                            'CONTRACT',\n",
      "                                             'transactionClass': 'Futures',\n",
      "                                             'transactionGroup': 'alt1',\n",
      "                                             'transactionRoles': 'LongLonger',\n",
      "                                             'type': 'O1'}],\n",
      "                                'movements': [{'direction': 1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'StockMovement',\n",
      "                                               'properties': [{'key': 'TransactionConfiguration/default/TaxLotSelectionMethod',\n",
      "                                                               'value': 'FirstInFirstOut'}],\n",
      "                                               'side': 'Side1'},\n",
      "                                              {'direction': -1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'CashCommitment',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side2'}],\n",
      "                                'properties': []},\n",
      "                               {'aliases': [{'description': 'Close Long '\n",
      "                                                            'Futures Contract',\n",
      "                                             'transactionClass': 'Futures',\n",
      "                                             'transactionGroup': 'default',\n",
      "                                             'transactionRoles': 'LongShorter',\n",
      "                                             'type': 'CLFC'},\n",
      "                                            {'description': 'CLOSE LONG '\n",
      "                                                            'FUTURES CONTRACT',\n",
      "                                             'transactionClass': 'Futures',\n",
      "                                             'transactionGroup': 'alt1',\n",
      "                                             'transactionRoles': 'LongShorter',\n",
      "                                             'type': 'C1'}],\n",
      "                                'movements': [{'direction': -1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'StockMovement',\n",
      "                                               'properties': [{'key': 'TransactionConfiguration/default/TaxLotSelectionMethod',\n",
      "                                                               'value': 'FirstInFirstOut'}],\n",
      "                                               'side': 'Side1'},\n",
      "                                              {'direction': 1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'CashCommitment',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side2'}],\n",
      "                                'properties': []},\n",
      "                               {'aliases': [{'description': 'Open Short '\n",
      "                                                            'Futures Contract',\n",
      "                                             'transactionClass': 'Futures',\n",
      "                                             'transactionGroup': 'default',\n",
      "                                             'transactionRoles': 'ShortShorter',\n",
      "                                             'type': 'OSFC'},\n",
      "                                            {'description': 'OPEN SHORT '\n",
      "                                                            'FUTURES CONTRACT',\n",
      "                                             'transactionClass': 'Futures',\n",
      "                                             'transactionGroup': 'alt1',\n",
      "                                             'transactionRoles': 'ShortShorter',\n",
      "                                             'type': 'O3'}],\n",
      "                                'movements': [{'direction': -1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'StockMovement',\n",
      "                                               'properties': [{'key': 'TransactionConfiguration/default/TaxLotSelectionMethod',\n",
      "                                                               'value': 'FirstInFirstOut'}],\n",
      "                                               'side': 'Side1'},\n",
      "                                              {'direction': 1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'CashCommitment',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side2'}],\n",
      "                                'properties': []},\n",
      "                               {'aliases': [{'description': 'Close Short '\n",
      "                                                            'Futures Contract',\n",
      "                                             'transactionClass': 'Futures',\n",
      "                                             'transactionGroup': 'default',\n",
      "                                             'transactionRoles': 'ShortLonger',\n",
      "                                             'type': 'CSFC'},\n",
      "                                            {'description': 'CLOSE SHORT '\n",
      "                                                            'FUTURES CONTRACT',\n",
      "                                             'transactionClass': 'Futures',\n",
      "                                             'transactionGroup': 'alt1',\n",
      "                                             'transactionRoles': 'ShortLonger',\n",
      "                                             'type': 'C3'}],\n",
      "                                'movements': [{'direction': 1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'StockMovement',\n",
      "                                               'properties': [{'key': 'TransactionConfiguration/default/TaxLotSelectionMethod',\n",
      "                                                               'value': 'FirstInFirstOut'}],\n",
      "                                               'side': 'Side1'},\n",
      "                                              {'direction': -1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'CashCommitment',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side2'}],\n",
      "                                'properties': []},\n",
      "                               {'aliases': [{'description': 'A private '\n",
      "                                                            'corporate loan to '\n",
      "                                                            'a company',\n",
      "                                             'transactionClass': 'Loan',\n",
      "                                             'transactionGroup': 'PrivateDebt',\n",
      "                                             'transactionRoles': 'ShortLonger',\n",
      "                                             'type': 'Loan'}],\n",
      "                                'movements': [{'direction': 1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'StockMovement',\n",
      "                                               'properties': [{'key': 'TransactionConfiguration/default/TaxLotSelectionMethod',\n",
      "                                                               'value': 'FirstInFirstOut'}],\n",
      "                                               'side': 'Side1'},\n",
      "                                              {'direction': -1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'CashCommitment',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side2'}],\n",
      "                                'properties': []},\n",
      "                               {'aliases': [{'description': 'A coupon payment '\n",
      "                                                            'from a private '\n",
      "                                                            'corporate loan to '\n",
      "                                                            'a company',\n",
      "                                             'transactionClass': 'Loan',\n",
      "                                             'transactionGroup': 'PrivateDebt',\n",
      "                                             'transactionRoles': 'Longer',\n",
      "                                             'type': 'Coupon'}],\n",
      "                                'movements': [{'direction': 1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'CashCommitment',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side2'}],\n",
      "                                'properties': []},\n",
      "                               {'aliases': [{'description': 'A partial or full '\n",
      "                                                            'repayment from a '\n",
      "                                                            'private corporate '\n",
      "                                                            'loan to a company',\n",
      "                                             'transactionClass': 'Loan',\n",
      "                                             'transactionGroup': 'PrivateDebt',\n",
      "                                             'transactionRoles': 'LongShorter',\n",
      "                                             'type': 'Repayment'}],\n",
      "                                'movements': [{'direction': -1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'StockMovement',\n",
      "                                               'properties': [{'key': 'TransactionConfiguration/default/TaxLotSelectionMethod',\n",
      "                                                               'value': 'FirstInFirstOut'}],\n",
      "                                               'side': 'Side1'},\n",
      "                                              {'direction': 1,\n",
      "                                               'mappings': [],\n",
      "                                               'movementTypes': 'CashCommitment',\n",
      "                                               'properties': [],\n",
      "                                               'side': 'Side2'}],\n",
      "                                'properties': []}]}\n"
     ]
    }
   ],
   "source": [
    "# Import the default transaction type configuration and load it into a dictionary \n",
    "default_transaction_mapping=open('data/default_transaction_mapping.json').read()\n",
    "default_transaction_mapping = json.loads(default_transaction_mapping)\n",
    "# Pretty print your configuration\n",
    "pprint.pprint(default_transaction_mapping)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have imported the default configuration you can build our request to set it inside LUSID.\n",
    "\n",
    "For further usage of the build transactions API call refer to the [LUSID API Docs: Set Configuration Transaction Types](https://docs.lusid.com/#operation/SetConfigurationTransactionTypes).\n",
    "\n",
    "*Run the cell below to set the transaction type configuration*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "transaction_configuration_api = api_factory.build(lusid.api.TransactionConfigurationApi)\n",
    "# Add default sides to non-default scope\n",
    "default_side_definitions = [\n",
    "    models.SidesDefinitionRequest(\n",
    "        side=\"Side1\", \n",
    "        side_request=models.SideDefinitionRequest(\n",
    "            security=\"Txn:LusidInstrumentId\",\n",
    "            currency=\"Txn:TradeCurrency\",\n",
    "            rate=\"Txn:TradeToPortfolioRate\",\n",
    "            units=\"Txn:Units\",\n",
    "            amount=\"Txn:TradeAmount\")),\n",
    "    models.SidesDefinitionRequest(\n",
    "        side=\"Side2\", \n",
    "        side_request=models.SideDefinitionRequest(\n",
    "            security=\"Txn:SettleCcy\",\n",
    "            currency=\"Txn:SettlementCurrency\",\n",
    "            rate=\"SettledToPortfolioRate\",\n",
    "            units=\"Txn:TotalConsideration\",\n",
    "            amount=\"Txn:TotalConsideration\"))\n",
    "]\n",
    "\n",
    "transaction_configuration_api.set_side_definitions(default_side_definitions, scope = scope)\n",
    "\n",
    "\n",
    "# Add default transaction types\n",
    "default_transaction_mapping=open('data/default_transaction_mapping.json').read()\n",
    "default_transaction_mapping = json.loads(default_transaction_mapping)\n",
    "\n",
    "def map_properties(properties):\n",
    "    return {property[\"key\"]: models.PerpetualProperty(property[\"key\"], models.PropertyValue(property[\"value\"])) for property in properties}\n",
    "def map_alias(alias):\n",
    "    return models.TransactionTypeAlias(alias[\"type\"], alias[\"description\"], alias[\"transactionClass\"], alias[\"transactionRoles\"])\n",
    "def map_movement(movement):\n",
    "    return models.TransactionTypeMovement(movement[\"movementTypes\"], movement[\"side\"], movement[\"direction\"], map_properties(movement[\"properties\"]))\n",
    "def map_transaction_type_request(transaction_type_request):\n",
    "    return models.TransactionTypeRequest(\n",
    "        [map_alias(alias) for alias in transaction_type_request[\"aliases\"]],\n",
    "        [map_movement(movement) for movement in transaction_type_request[\"movements\"]],\n",
    "        map_properties(transaction_type_request[\"properties\"]))\n",
    "\n",
    "for configuration in default_transaction_mapping:\n",
    "    transaction_type_requests = [map_transaction_type_request(transaction_type_request) for transaction_type_request in configuration[\"transactionTypeRequests\"]]\n",
    "    \n",
    "    # Call LUSID to set your configuration for our transaction types\n",
    "    transaction_configuration_api.set_transaction_type_source(\n",
    "        source=configuration[\"source\"],\n",
    "        transaction_type_request=transaction_type_requests,\n",
    "        scope=scope\n",
    "    )\n",
    "\n",
    "# Call LUSID to update the transaction type scope of your portfolio \n",
    "patch_document = [\n",
    "    {\n",
    "        \"value\": scope,\n",
    "        \"path\": \"/transactiontypescope\",\n",
    "        \"op\": \"add\"\n",
    "    }\n",
    "]\n",
    "patch_response = api_factory.build(lusid.api.TransactionPortfoliosApi).patch_portfolio_details(\n",
    "    scope=scope,\n",
    "    code=portfolio_code,\n",
    "    operation=patch_document)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Instruments](img/migration-transactiontypesconfig.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7) Add your Transactions Throughout the Day"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have both imported your transactions and set your transaction type configuration you can upsert your transactions into LUSID. For simplicity each transaction has the same trade and settlement date.\n",
    "\n",
    "Read more about transactions here [LUSID Knowledge Base: Transactions](https://support.lusid.com/what-is-a-transaction). \n",
    "\n",
    "For further usage of the upsert transactions API call refer to the [LUSID API Docs: Upsert Transactions](https://docs.lusid.com/#operation/UpsertTransactions).\n",
    "\n",
    "*Run the cell below to upsert our transactions into LUSID*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mTransactions Successfully Upserted into Portfolio\u001b[0m\n",
      "\u001b[1mScope: \u001b[0minvestment-book-of-records-3bf2-7ffd-98a6-44\n",
      "\u001b[1mCode: \u001b[0mGlobal-Strategies\n",
      "\u001b[1mTransactions Effective From: \u001b[0m2023-06-20 16:38:55.568651+00:00\n",
      "\u001b[1mTransactions Created On: \u001b[0m2023-06-21 16:38:56.769105+00:00\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Set your transaction and settlement dates\n",
    "transaction_date = datetime.now(pytz.UTC)-timedelta(days=1)\n",
    "settlement_date = datetime.now(pytz.UTC)+timedelta(days=1)\n",
    "\n",
    "# Iterate over the portfolios in our transactions file, in this case only one\n",
    "for portfolio in daily_transactions['portfolio_code'].unique():\n",
    "    # Update the transaction type scope of the portfolio to be the scope of the upserted transaction type configuration\n",
    "    patch_response = api_factory.build(lusid.api.TransactionPortfoliosApi).patch_portfolio_details(\n",
    "        scope=scope,\n",
    "        code=portfolio_code,\n",
    "        operation=patch_document)\n",
    "\n",
    "    # Initialise a list to hold your transaction requests\n",
    "    transactions = []\n",
    "    \n",
    "    # Iterate over the transactions in the portfolio\n",
    "    for index, transaction in daily_transactions.loc[daily_transactions['portfolio_code'] == portfolio].iterrows():\n",
    "        \n",
    "        # Append your request to the list\n",
    "        transactions.append(models.TransactionRequest(\n",
    "            transaction_id=transaction['transaction_id'],\n",
    "            type=transaction['transaction_type'],\n",
    "            instrument_identifiers={\n",
    "                'Instrument/default/Figi': transaction['figi']\n",
    "            },\n",
    "            transaction_date=transaction_date.isoformat(),\n",
    "            settlement_date=settlement_date.isoformat(),\n",
    "            units=transaction['transaction_units'],\n",
    "            transaction_price=models.TransactionPrice(\n",
    "                  price=transaction['transaction_price'],\n",
    "                  type='Price'),\n",
    "            total_consideration=models.CurrencyAndAmount(\n",
    "              amount=transaction['transaction_cost'],\n",
    "              currency=transaction['transaction_currency']),\n",
    "             source='Default',\n",
    "             transaction_currency=transaction['transaction_currency']))\n",
    "        \n",
    "    # Call LUSID to upsert your transactions for this portfolio\n",
    "    response = api_factory.build(lusid.api.TransactionPortfoliosApi).upsert_transactions(\n",
    "        scope=scope,\n",
    "        code=portfolio,\n",
    "        transaction_request=transactions)\n",
    "    \n",
    "    # Print the response from LUSID using pretty formatting \n",
    "    prettyprint.transactions_response(\n",
    "        response,\n",
    "        scope,\n",
    "        portfolio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Transactions](img/migration-addtransactions.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8) Retrieve your Holdings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see how these transactions have affected your holdings by trying to retrieve the holdings of your portfolio. By configuring your transaction types LUSID should have been able to decompose each transaction into its appropriate economic movements. \n",
    "\n",
    "For further usage of the get holdings API call refer to the [LUSID API Docs: Get Holdings](https://docs.lusid.com/#operation/GetHoldings).\n",
    "\n",
    "*Run the cell below to get our holdings from our portfolio*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error retrieving holdings. (404)\n",
      "Reason: Not Found\n",
      "HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 21 Jun 2023 16:38:57 GMT', 'Content-Type': 'application/problem+json', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Vary': 'Accept-Encoding', 'lusid-meta-success': 'False', 'lusid-meta-requestid': '0HMRIE84F6H4T:00000026', 'lusid-meta-correlationid': '0HMRIE84F6H4T:00000026', 'lusid-meta-duration': '709', 'x-envoy-upstream-service-time': '712', 'Strict-Transport-Security': 'max-age=15724800; includeSubDomains', 'Server': 'FINBOURNE', 'Content-Security-Policy': \"default-src 'self' https://*.lusid.com https://*.finbourne.com; script-src 'unsafe-inline' 'self' https://*.lusid.com https://*.finbourne.com https://editor.swagger.io; font-src 'self' fonts.googleapis.com; img-src data: 'self' https://*.lusid.com https://*.finbourne.com https://validator.swagger.io; style-src 'unsafe-inline' 'self' https://*.lusid.com https://*.finbourne.com; report-uri https://lusid.report-uri.com/r/d/csp/enforce\", 'X-Frame-Options': 'SAMEORIGIN', 'Permissions-Policy': 'accelerometer=(), ambient-light-sensor=(), autoplay=(self), battery=(), camera=(), cross-origin-isolated=(self), display-capture=(), document-domain=*, encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(self), geolocation=(self), gyroscope=(), keyboard-map=(), magnetometer=(), microphone=(), midi=(), navigation-override=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=()', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'X-Content-Type-Options': 'nosniff', 'Expect-CT': \"max-age=3600, enforce, report-uri='https://lusid.report-uri.com/r/d/ct/enforce'\", 'Access-Control-Max-Age': '600', 'Content-Encoding': 'gzip'})\n",
      "HTTP response body: {\"name\":\"TransactionTypeNotFound\",\"errorDetails\":[],\"code\":230,\"type\":\"https://docs.lusid.com/#section/Error-Codes/230\",\"title\":\"A transaction type is not configured\",\"status\":404,\"detail\":\"There is no configuration for the transaction type: 'BrokerSell' from the source: 'Default' at time: EffectiveAt: '2023-06-20T16:38:55.5686510+00:00' AsAt: 'AsAt=2023-06-21T16:38:56.7691050+00:00'\",\"instance\":\"https://fbn-ci.lusid.com/app/insights/logs/0HMRIE84F6H4T:00000026\",\"extensions\":{}}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Try and get your holdings, otherwise provide the error message\n",
    "try:\n",
    "    response = api_factory.build(lusid.api.TransactionPortfoliosApi).get_holdings(\n",
    "        scope=scope,\n",
    "        code=portfolio_code,\n",
    "        property_keys=['Instrument/default/Name'])\n",
    "    prettyprint.holdings_response(response, scope, portfolio_code)\n",
    "except lusid.rest.ApiException as e:\n",
    "    print ('Error retrieving holdings. {}'.format(e))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here you can see that it looks like you have a transaction type in your transactions file which is not in your configuration."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9) Update our Transaction Type Configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can find out what might be the problematic transaction type by listing all of your unique transaction types.\n",
    "\n",
    "*Run the cell below to find your unique transaction types*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Sell' 'Buy' 'BrokerSell']\n"
     ]
    }
   ],
   "source": [
    "# Get your unique list of transaction types\n",
    "transaction_types = daily_transactions['transaction_type'].unique()\n",
    "print (transaction_types)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here you can see that you have three transaction types in your list of transactions, you can check that they are all in your current configuration. You can do this by listing your current configuration in LUSID.\n",
    "\n",
    "For further usage of the list configuration transaction types API call refer to the [LUSID API Docs: List Transaction Type Configuration](https://docs.lusid.com/#operation/ListConfigurationTransactionTypes).\n",
    "\n",
    "*Run the cell below to check your transaction type configuration*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m\u001b[4mTransaction Configuration #40\u001b[0m\n",
      "\n",
      "\u001b[1m\u001b[91mTransaction Type Aliases\u001b[0m\n",
      "\u001b[1mTransaction Type: \u001b[0m\u001b[91mBuy\u001b[0m\n",
      "\u001b[1mAlias Description: \u001b[0mPurchase\n",
      "\u001b[1mTransaction Class: \u001b[0mBasic\n",
      "\u001b[1mTransaction Group: \u001b[0mdefault\n",
      "\u001b[1mTransaction Roles: \u001b[0mLongLonger\n",
      "\n",
      "\n",
      "\u001b[1m\u001b[91mTransaction Movements\u001b[0m\n",
      "\u001b[1mMovement Types: \u001b[0mStockMovement\n",
      "\u001b[1mSide: \u001b[0mSide1\n",
      "\u001b[1mDirection: \u001b[0m1\n",
      "\u001b[1mMovement Types: \u001b[0mCashCommitment\n",
      "\u001b[1mSide: \u001b[0mSide2\n",
      "\u001b[1mDirection: \u001b[0m-1\n",
      "\n",
      "\n",
      "\n",
      "\u001b[1m\u001b[4mTransaction Configuration #41\u001b[0m\n",
      "\n",
      "\u001b[1m\u001b[91mTransaction Type Aliases\u001b[0m\n",
      "\u001b[1mTransaction Type: \u001b[0m\u001b[91mSell\u001b[0m\n",
      "\u001b[1mAlias Description: \u001b[0mSale\n",
      "\u001b[1mTransaction Class: \u001b[0mBasic\n",
      "\u001b[1mTransaction Group: \u001b[0mdefault\n",
      "\u001b[1mTransaction Roles: \u001b[0mLongShorter\n",
      "\n",
      "\n",
      "\u001b[1m\u001b[91mTransaction Movements\u001b[0m\n",
      "\u001b[1mMovement Types: \u001b[0mStockMovement\n",
      "\u001b[1mSide: \u001b[0mSide1\n",
      "\u001b[1mDirection: \u001b[0m-1\n",
      "\u001b[1mMovement Types: \u001b[0mCashCommitment\n",
      "\u001b[1mSide: \u001b[0mSide2\n",
      "\u001b[1mDirection: \u001b[0m1\n",
      "\n",
      "\n",
      "\n",
      "\u001b[1m\u001b[4mTransaction Configuration #82\u001b[0m\n",
      "\n",
      "\u001b[1m\u001b[91mTransaction Type Aliases\u001b[0m\n",
      "\u001b[1mTransaction Type: \u001b[0m\u001b[91mBrokerSell\u001b[0m\n",
      "\u001b[1mAlias Description: \u001b[0mA sale of an equity via a broker\n",
      "\u001b[1mTransaction Class: \u001b[0mEquity\n",
      "\u001b[1mTransaction Group: \u001b[0mBrokerSystem\n",
      "\u001b[1mTransaction Roles: \u001b[0mShorter\n",
      "\n",
      "\n",
      "\u001b[1m\u001b[91mTransaction Movements\u001b[0m\n",
      "\u001b[1mMovement Types: \u001b[0mStockMovement\n",
      "\u001b[1mSide: \u001b[0mSide1\n",
      "\u001b[1mDirection: \u001b[0m1\n",
      "\u001b[1mMovement Types: \u001b[0mCashCommitment\n",
      "\u001b[1mSide: \u001b[0mSide2\n",
      "\u001b[1mDirection: \u001b[0m1\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Call LUSID to get your transaction type configuration\n",
    "response = transaction_configuration_api.list_transaction_types(scope=scope)\n",
    "# Pretty print the configuration\n",
    "prettyprint.transaction_type_response(response, filters=['Buy', 'Sell', 'BrokerSell'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here you can see that although you have the 'Sell' and 'Buy' transaction types, you are missing the 'BrokerSell' transaction type. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Transactions](img/migration-missingconfig.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You need to add this transaction type into your configuration so that LUSID can correctly deconstruct the transaction into its underlying economic movements. To do this you can take a look at the 'BrokerSell' transactions to see what they represent.\n",
    "\n",
    "*Run the cell below to see all your BrokerSell transactions*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>portfolio_code</th>\n",
       "      <th>transaction_id</th>\n",
       "      <th>instrument_name</th>\n",
       "      <th>transaction_description</th>\n",
       "      <th>transaction_type</th>\n",
       "      <th>transaction_units</th>\n",
       "      <th>transaction_price</th>\n",
       "      <th>transaction_currency</th>\n",
       "      <th>transaction_strategy</th>\n",
       "      <th>transaction_cost</th>\n",
       "      <th>figi</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Global-Strategies</td>\n",
       "      <td>tid_124319009950924</td>\n",
       "      <td>USTreasury_2.00_2021</td>\n",
       "      <td>Equity Sale</td>\n",
       "      <td>BrokerSell</td>\n",
       "      <td>-28000</td>\n",
       "      <td>98.10</td>\n",
       "      <td>USD</td>\n",
       "      <td>incomeRequirements</td>\n",
       "      <td>2746800.00</td>\n",
       "      <td>BBG00FN3B5K8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      portfolio_code       transaction_id       instrument_name  \\\n",
       "5  Global-Strategies  tid_124319009950924  USTreasury_2.00_2021   \n",
       "\n",
       "  transaction_description transaction_type  transaction_units  \\\n",
       "5             Equity Sale       BrokerSell             -28000   \n",
       "\n",
       "   transaction_price transaction_currency transaction_strategy  \\\n",
       "5              98.10                  USD   incomeRequirements   \n",
       "\n",
       "   transaction_cost          figi  \n",
       "5        2746800.00  BBG00FN3B5K8  "
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get all your transactions with the BrokerSell transaction type\n",
    "brokersell_transactions = daily_transactions.loc[daily_transactions['transaction_type']=='BrokerSell']\n",
    "brokersell_transactions.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here you can see that it looks just like a sell transaction, you can add it into your configuration so that it increases your cash balance and reduces the number of units of your security. \n",
    "\n",
    "*Run the cell below to add your BrokerSell transaction type to the configuration*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Call LUSID to create your transaction type configuration for the BrokerSell type\n",
    "brokersell_type=lusid.models.TransactionTypeRequest(\n",
    "    aliases=[\n",
    "        lusid.models.TransactionTypeAlias(\n",
    "            type=\"BrokerSell\",\n",
    "            description=\"A sale of an equity via a broker\",\n",
    "            transaction_class=\"Equity\",\n",
    "            transaction_roles=\"Shorter\",\n",
    "        )\n",
    "    ],\n",
    "    movements=[\n",
    "        lusid.models.TransactionTypeMovement(\n",
    "            movement_types=\"StockMovement\",\n",
    "            side=\"Side1\",\n",
    "            direction=-1,\n",
    "            properties=None,\n",
    "            mappings=None,\n",
    "        ),\n",
    "        lusid.models.TransactionTypeMovement(\n",
    "            movement_types=\"CashCommitment\",\n",
    "            side=\"Side2\",\n",
    "            direction=1,\n",
    "            properties=None,\n",
    "            mappings=None,\n",
    "        )\n",
    "    ],\n",
    "    properties=None\n",
    ")\n",
    "\n",
    "upsert_brokersell_config = transaction_configuration_api.set_transaction_type(\n",
    "    source=\"BrokerSystem\",\n",
    "    type=\"BrokerSell\",\n",
    "    transaction_type_request=brokersell_type,\n",
    "    scope=scope\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You have added the BrokerSell transaction type under the 'BrokerSystem' transaction group. This allows you to group all your broker transaction types together and keep them seperate from your internal systems. To ensure that the transaction uses this group of types to resolve the underlying economic movements you can change the source parameter to 'BrokerSystem' by upserting the broker transaction again. \n",
    "\n",
    "*Run the cell below to re-upsert the transaction with the BrokerSystem source*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mTransactions Successfully Upserted into Portfolio\u001b[0m\n",
      "\u001b[1mScope: \u001b[0minvestment-book-of-records-3bf2-7ffd-98a6-44\n",
      "\u001b[1mCode: \u001b[0mGlobal-Strategies\n",
      "\u001b[1mTransactions Effective From: \u001b[0m2023-06-20 16:38:55.568651+00:00\n",
      "\u001b[1mTransactions Created On: \u001b[0m2023-06-21 16:39:09.828301+00:00\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Initialise a list to hold your transaction requests\n",
    "transactions = []\n",
    "\n",
    "# Iterate over the transactions in the portfolio\n",
    "for index, transaction in brokersell_transactions.iterrows():\n",
    "\n",
    "    # Append your request to the list\n",
    "    transactions.append(models.TransactionRequest(\n",
    "        transaction_id=transaction['transaction_id'],\n",
    "        type=transaction['transaction_type'],\n",
    "        instrument_identifiers={\n",
    "            'Instrument/default/Figi': transaction['figi']\n",
    "        },\n",
    "        transaction_date=transaction_date.isoformat(),\n",
    "        settlement_date=settlement_date.isoformat(),\n",
    "        units=transaction['transaction_units'],\n",
    "        transaction_price=models.TransactionPrice(\n",
    "              price=transaction['transaction_price'],\n",
    "              type='Price'),\n",
    "        total_consideration=models.CurrencyAndAmount(\n",
    "          amount=transaction['transaction_cost'],\n",
    "          currency=transaction['transaction_currency']),\n",
    "         source='BrokerSystem',\n",
    "         transaction_currency=transaction['transaction_currency']))\n",
    "\n",
    "# Call LUSID to upsert your transactions for this portfolio\n",
    "response = api_factory.build(lusid.api.TransactionPortfoliosApi).upsert_transactions(\n",
    "    scope=scope,\n",
    "    code=portfolio,\n",
    "    transaction_request=transactions)\n",
    "\n",
    "# Print the response from LUSID using pretty formatting \n",
    "prettyprint.transactions_response(\n",
    "    response,\n",
    "    scope,\n",
    "    portfolio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Transactions](img/migration-addedconfig.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can try again to generate the holdings from your portfolio.\n",
    "\n",
    "*Run the cell below to try and retrieve your holdings again*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mHoldings for Portfolio\u001b[0m\n",
      "\u001b[1mScope: \u001b[0minvestment-book-of-records-3bf2-7ffd-98a6-44\n",
      "\u001b[1mCode: \u001b[0mGlobal-Strategies\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Try and get your holdings, otherwise provide the error message\n",
    "try:\n",
    "    response = api_factory.build(lusid.api.TransactionPortfoliosApi).get_holdings(\n",
    "        scope=scope,\n",
    "        code=portfolio_code,\n",
    "        property_keys=['Instrument/default/Name'])\n",
    "    prettyprint.holdings_response(\n",
    "        response, \n",
    "        scope, \n",
    "        portfolio_code)\n",
    "except lusid.exceptions.ApiException as e:\n",
    "    print ('Error retrieving holdings as {}'.format(e))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10) Adjust Holdings with your End of Day Position"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, you are able to generate your holdings from LUSID! Now that you have succesfully added your transactions into LUSID, you can update your portfolio with the end of day position to ensure that your source system and LUSID are in sync.\n",
    "\n",
    "You will import your end of day holdings from a CSV file.\n",
    "\n",
    "*Run the cell below to import your end of day holdings*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>portfolio_code</th>\n",
       "      <th>instrument_name</th>\n",
       "      <th>quantity</th>\n",
       "      <th>price</th>\n",
       "      <th>currency</th>\n",
       "      <th>figi</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Global-Strategies</td>\n",
       "      <td>GBP_Cash</td>\n",
       "      <td>8336000</td>\n",
       "      <td>1.00</td>\n",
       "      <td>GBP</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Global-Strategies</td>\n",
       "      <td>Glencore_LondonStockEx_GLEN</td>\n",
       "      <td>905141</td>\n",
       "      <td>2.76</td>\n",
       "      <td>GBP</td>\n",
       "      <td>BBG001MM1KV4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Global-Strategies</td>\n",
       "      <td>Kingfisher_LondonStockEx_KGF</td>\n",
       "      <td>1037038</td>\n",
       "      <td>2.28</td>\n",
       "      <td>GBP</td>\n",
       "      <td>BBG000BKH1W6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Global-Strategies</td>\n",
       "      <td>UKGiltTreasury_2.0_2025</td>\n",
       "      <td>405589</td>\n",
       "      <td>106.64</td>\n",
       "      <td>GBP</td>\n",
       "      <td>BBG0088JSC32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Global-Strategies</td>\n",
       "      <td>UKGiltTreasury_3.5_2045</td>\n",
       "      <td>266169</td>\n",
       "      <td>134.43</td>\n",
       "      <td>GBP</td>\n",
       "      <td>BBG006N6HZM7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      portfolio_code               instrument_name  quantity  price currency  \\\n",
       "0  Global-Strategies                      GBP_Cash   8336000   1.00      GBP   \n",
       "1  Global-Strategies   Glencore_LondonStockEx_GLEN    905141   2.76      GBP   \n",
       "2  Global-Strategies  Kingfisher_LondonStockEx_KGF   1037038   2.28      GBP   \n",
       "3  Global-Strategies       UKGiltTreasury_2.0_2025    405589 106.64      GBP   \n",
       "4  Global-Strategies       UKGiltTreasury_3.5_2045    266169 134.43      GBP   \n",
       "\n",
       "           figi  \n",
       "0           NaN  \n",
       "1  BBG001MM1KV4  \n",
       "2  BBG000BKH1W6  \n",
       "3  BBG0088JSC32  \n",
       "4  BBG006N6HZM7  "
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "end_of_day_holdings = pd.read_csv('data/demo-holdings2.csv')\n",
    "end_of_day_holdings.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have your holdings you can set your portfolio holdings just like you did at the start of the day to ensure that it matches the source system perfectly.\n",
    "\n",
    "*Run the cell below to adjust your holdings to these levels*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mHoldings Successfully Set for Portfolio\u001b[0m\n",
      "\u001b[1mScope: \u001b[0minvestment-book-of-records-3bf2-7ffd-98a6-44\n",
      "\u001b[1mCode: \u001b[0mGlobal-Strategies\n",
      "\u001b[1mHoldings Effective From: \u001b[0m2020-08-03 16:38:35.884666+00:00\n",
      "\u001b[1mHoldings Created On: \u001b[0m2023-06-21 16:39:12.816871+00:00\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Set the holdings to be effective from half a day ago\n",
    "holdings_effective_date = datetime.now(pytz.UTC) - timedelta(days=0.5)\n",
    "\n",
    "# Iterate over the portfolios in our holdings, in this case only one\n",
    "for portfolio in end_of_day_holdings['portfolio_code'].unique():\n",
    "    \n",
    "    # Initialise a list to hold your adjustments\n",
    "    holding_adjustments = []\n",
    "    \n",
    "    # Iterate over the end of day holdings for this portfolio\n",
    "    for index, holding in end_of_day_holdings.loc[end_of_day_holdings['portfolio_code'] == portfolio].iterrows():\n",
    "        \n",
    "        # Ensure that you use the correct identifier based on the instrument\n",
    "        if 'Cash' in holding['instrument_name']:\n",
    "            identifier_key = 'Instrument/default/Currency'\n",
    "            identifer = holding['instrument_name'].split('_')[0]\n",
    "        else:\n",
    "            identifier_key = 'Instrument/default/Figi'\n",
    "            identifer = holding['figi']\n",
    "            \n",
    "        # Build your adjustment and add it to your list\n",
    "        holding_adjustments.append(\n",
    "            models.AdjustHoldingRequest(\n",
    "                instrument_identifiers={\n",
    "                    identifier_key: identifer},\n",
    "                tax_lots=[\n",
    "                    models.TargetTaxLotRequest(\n",
    "                        units=holding['quantity'],\n",
    "                        cost=models.CurrencyAndAmount(\n",
    "                            amount=holding['quantity'] * holding['price'],\n",
    "                            currency=holding['currency']),\n",
    "                        portfolio_cost=holding['quantity'] * holding['price'],\n",
    "                        price=holding['price'])\n",
    "                ]\n",
    "            )\n",
    "        )\n",
    "        \n",
    "    # Call LUSID to set your end of day position\n",
    "    response = api_factory.build(lusid.api.TransactionPortfoliosApi).set_holdings(\n",
    "        scope=scope,\n",
    "        code=portfolio,\n",
    "        effective_at=holdings_effective_date.isoformat(),\n",
    "        adjust_holding_request=holding_adjustments)\n",
    "\n",
    "    holdings_as_at = response.version.as_at_date\n",
    "\n",
    "    # Pretty print the response from LUSID\n",
    "    prettyprint.set_holdings_response(\n",
    "        response, \n",
    "        scope, \n",
    "        portfolio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Transactions](img/migration-endofday.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11) Bi-temporal Reconciliation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can make use of LUSID's bi-temporal data store to compare your portfolio before and after adjusting for the end of day position. You can do this by performing a bi-temporal reconciliation. Read more about reconciliations here [LUSID Knowledge Base: Performing a Reconciliation](https://support.lusid.com/how-do-i-reconcile-portfolios-from-different).\n",
    "\n",
    "By looking for reconciliation breaks between your portfolio before it was adjusted and after it was adjusted you can see what discrepancies have arisen from any issues with the transaction feed during the day. \n",
    "\n",
    "For further usage of the reconcile holdings API call refer to the [LUSID API Docs: Reconcile Holdings](https://docs.lusid.com/#operation/ReconcileValuation).\n",
    "\n",
    "*Run the cell below to reconcile your portfolio with itself before and after the adjustment*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mReconciliation Breaks for Portfolio\u001b[0m\n",
      "\u001b[1mScope: \u001b[0minvestment-book-of-records-3bf2-7ffd-98a6-44\n",
      "\u001b[1mCode: \u001b[0mGlobal-Strategies\n",
      "\n",
      "\u001b[1mReconciliation Break\u001b[0m\n",
      "\u001b[1mInstrument LUID: \u001b[0mLUID_0000MIPE\n",
      "\u001b[1mLeft Units: \u001b[0m1411244.0\n",
      "\u001b[1mRight Units: \u001b[0m1355244.0\n",
      "\u001b[1mDifference In Units: \u001b[91m-56000.0\u001b[0m\n",
      "\u001b[1mLeft Cost: \u001b[0m138166387.6\n",
      "\u001b[1mRight Cost: \u001b[0m132678387.6\n",
      "\u001b[1mDifference In Cost: \u001b[91m-5488000.0\u001b[0m\n",
      "\u001b[1mCurrency: \u001b[0mUSD\n",
      "\u001b[1mInstrument/default/Name: \u001b[0m{'label_value': 'USTreasury_2.00_2021',\n",
      " 'label_value_set': None,\n",
      " 'metric_value': None}\n",
      "\u001b[1mInstrument/default/Figi: \u001b[0m{'label_value': 'BBG00FN3B5K8', 'label_value_set': None, 'metric_value': None}\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Build your left side of the reconciliation before you made the adjustment\n",
    "before_adjustment = models.PortfolioReconciliationRequest(\n",
    "    portfolio_id=models.ResourceId(\n",
    "        scope=scope,\n",
    "        code=portfolio_code),\n",
    "    effective_at=(holdings_effective_date - timedelta(hours=1)).isoformat(),\n",
    "    as_at=holdings_as_at)\n",
    "\n",
    "# Build your right side of the reconciliation after you made the adjustment\n",
    "after_adjustment = models.PortfolioReconciliationRequest(\n",
    "    portfolio_id=models.ResourceId(\n",
    "        scope=scope,\n",
    "        code=portfolio_code),\n",
    "    effective_at=(holdings_effective_date + timedelta(hours=1)).isoformat(),\n",
    "    as_at=holdings_as_at)\n",
    "\n",
    "# Create your reconciliation request\n",
    "reconcile_holdings_request = models.PortfoliosReconciliationRequest(\n",
    "    left=before_adjustment,\n",
    "    right=after_adjustment,\n",
    "    instrument_property_keys=['Instrument/default/Name', 'Instrument/default/Figi'])\n",
    "\n",
    "# Reconcile your two portfolios\n",
    "response = api_factory.build(lusid.api.ReconciliationsApi).reconcile_holdings(\n",
    "    portfolios_reconciliation_request=reconcile_holdings_request)\n",
    "\n",
    "# Pretty print the response\n",
    "prettyprint.reconciliation_response(\n",
    "    response,\n",
    "    scope,\n",
    "    portfolio_code)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Transactions](img/migration-reconciliation.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 12) Investigate Breaks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this reconciliation you can see that there is a single break. You can find the transactions that may been responsible for this break by getting all transactions made against the instrument in the break.\n",
    "\n",
    "For further usage of the get transactions API call refer to the [LUSID API Docs: Get Transactions](https://docs.lusid.com/#operation/GetTransactions).\n",
    "\n",
    "*Run the cell below to find all transactions made against this instrument*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[91m\u001b[1mTransactions Retrieved from Portfolio\u001b[0m\n",
      "\u001b[1mScope: \u001b[0m investment-book-of-records-3bf2-7ffd-98a6-44\n",
      "\u001b[1mCode: \u001b[0m Global-Strategies \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Transaction ID</th>\n",
       "      <th>Transaction Type</th>\n",
       "      <th>Instrument/default/Name</th>\n",
       "      <th>Units</th>\n",
       "      <th>Price</th>\n",
       "      <th>Currency</th>\n",
       "      <th>Transaction Date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>tid_234942982496001</td>\n",
       "      <td>Sell</td>\n",
       "      <td>USTreasury_2.00_2021</td>\n",
       "      <td>57000.00</td>\n",
       "      <td>97.80</td>\n",
       "      <td>USD</td>\n",
       "      <td>2023-06-20 16:38:55.568651+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>tid_124319009950924</td>\n",
       "      <td>BrokerSell</td>\n",
       "      <td>USTreasury_2.00_2021</td>\n",
       "      <td>-28000.00</td>\n",
       "      <td>98.10</td>\n",
       "      <td>USD</td>\n",
       "      <td>2023-06-20 16:38:55.568651+00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Transaction ID Transaction Type Instrument/default/Name     Units  \\\n",
       "0  tid_234942982496001             Sell    USTreasury_2.00_2021  57000.00   \n",
       "1  tid_124319009950924       BrokerSell    USTreasury_2.00_2021 -28000.00   \n",
       "\n",
       "   Price Currency                 Transaction Date  \n",
       "0  97.80      USD 2023-06-20 16:38:55.568651+00:00  \n",
       "1  98.10      USD 2023-06-20 16:38:55.568651+00:00  "
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Parse the Lusid Instrument Id from the reconciliation break\n",
    "luid = response.values[0].instrument_uid\n",
    "\n",
    "# Call LUSID to get your transactions made against this Lusid Instrument Id\n",
    "response = api_factory.build(lusid.api.TransactionPortfoliosApi).get_transactions(\n",
    "    scope=scope,\n",
    "    code=portfolio_code,\n",
    "    from_transaction_date=transaction_date,\n",
    "    to_transaction_date=transaction_date,\n",
    "    property_keys=['Instrument/default/Name'],\n",
    "    filter=\"instrumentuid eq '\" + luid + \"'\"\n",
    "    )\n",
    "\n",
    "# Pretty print the response\n",
    "prettyprint.get_transactions_response(\n",
    "    response,\n",
    "    scope,\n",
    "    portfolio_code,\n",
    "    ['Instrument/default/Name'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here you can see something interesting. Even though you have configured the transaction type for Sell and BrokerSell to have the same underlying economic movements, it looks like the units for a BrokerSell are passed in as a signed number whereas for the Sell the number of units is not signed. \n",
    "\n",
    "In fact the break is for 56000 units and our BrokerSell transaction is for 28000 units, exactly half the size of the break. Perhaps this is due to the sign on the units of the trade resulting in the number of this instrument being held moving in the wrong direction. \n",
    "\n",
    "You can fix this by updating the transaction configuration. In this case we need to adjust the direction of the 'Side1' movement from -1 to 1. This will take into account the BrokerSell transaction using a signed number for units.\n",
    "\n",
    "*Run the cell below to update our transaction configuration*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add our updated BrokerSell configuration to our default configuration\n",
    "brokersell_type=lusid.models.TransactionTypeRequest(\n",
    "    aliases=[\n",
    "        lusid.models.TransactionTypeAlias(\n",
    "            type=\"BrokerSell\",\n",
    "            description=\"A sale of an equity via a broker\",\n",
    "            transaction_class=\"Equity\",\n",
    "            transaction_roles=\"Shorter\",\n",
    "        )\n",
    "    ],\n",
    "    movements=[\n",
    "        lusid.models.TransactionTypeMovement(\n",
    "            movement_types=\"StockMovement\",\n",
    "            side=\"Side1\",\n",
    "            direction=1,\n",
    "            properties=None,\n",
    "            mappings=None,\n",
    "        ),\n",
    "        lusid.models.TransactionTypeMovement(\n",
    "            movement_types=\"CashCommitment\",\n",
    "            side=\"Side2\",\n",
    "            direction=1,\n",
    "            properties=None,\n",
    "            mappings=None,\n",
    "        )\n",
    "    ],\n",
    "    properties=None\n",
    ")\n",
    "\n",
    "upsert_brokersell_config = transaction_configuration_api.set_transaction_type(\n",
    "    source=\"BrokerSystem\",\n",
    "    type=\"BrokerSell\",\n",
    "    transaction_type_request=brokersell_type,\n",
    "    scope=scope\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So that you can easily check the last time that you updated your transaction type configuration, you can set a property on your portfolio with the date of the last change. Read more about properties here [LUSID Knowledge Base: Properties](https://support.lusid.com/what-is-a-property). \n",
    "\n",
    "For further usage of the create property definition API call refer to the [LUSID API Docs: Create Property Definition](https://docs.lusid.com/#operation/CreatePropertyDefinition).\n",
    "\n",
    "*Run the cell below to create a property to hold the date of the last change*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mPortfolio Property Key: \u001b[0mPortfolio/investment-book-of-records-3bf2-7ffd-98a6-44/lastconfigurationchange\n"
     ]
    }
   ],
   "source": [
    "# Create our request to define a new property\n",
    "request = models.CreatePropertyDefinitionRequest(\n",
    "    domain='Portfolio',\n",
    "    scope=scope,\n",
    "    code='lastconfigurationchange',\n",
    "    value_required=False,\n",
    "    display_name='lastconfigurationchange',\n",
    "    data_type_id=models.ResourceId(scope='system', code='date'))\n",
    "\n",
    "# Call LUSID to create our new property\n",
    "response = api_factory.build(lusid.api.PropertyDefinitionsApi).create_property_definition(\n",
    "    create_property_definition_request=request)\n",
    "\n",
    "# Grab the key off the response to use when referencing this property in other LUSID calls\n",
    "portfolio_property_key = response.key\n",
    "\n",
    "# Pretty print our key\n",
    "prettyprint.heading('Portfolio Property Key', portfolio_property_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can set the value of this property on your portfolio, as you just changed the configuration you can use the current datetime.\n",
    "\n",
    "For further usage of the upsert portfolio properties API call refer to the [LUSID API Docs: Upsert Portfolio Properties](https://docs.lusid.com/#operation/UpsertPortfolioProperties).\n",
    "\n",
    "*Run the cell below to add the date of the last change to your portfolio*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mProperties Sucessfully Updated for Portfolio\u001b[0m\n",
      "\u001b[1mProperty key: \u001b[0mPortfolio/investment-book-of-records-3bf2-7ffd-98a6-44/lastconfigurationchange\n",
      "\u001b[1mValue: \u001b[0m2023-06-21T16:39:33.506184+00:00\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Call LUSID to add the date of the last change to the portfolio\n",
    "response = api_factory.build(lusid.api.PortfoliosApi).upsert_portfolio_properties(\n",
    "    scope=scope,\n",
    "    code=portfolio_code,\n",
    "    request_body={\n",
    "        portfolio_property_key: models.ModelProperty(\n",
    "            key=portfolio_property_key,\n",
    "            value=models.PropertyValue(label_value=datetime.now(pytz.UTC)))\n",
    "    }\n",
    ")\n",
    "\n",
    "# Pretty print the response \n",
    "prettyprint.portfolio_properties_response(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Portfolio Property](img/migration-portfolioproperty.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13) Re-run Bi-temporal Reconciliation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now re-run the bi-temporal reconciliation to see if changing the transaction type configuration has removed the break. \n",
    "\n",
    "*Run the cell below to re-run the reconciliation to see if the break has been fixed*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mReconciliation Breaks for Portfolio\u001b[0m\n",
      "\u001b[1mScope: \u001b[0minvestment-book-of-records-3bf2-7ffd-98a6-44\n",
      "\u001b[1mCode: \u001b[0mGlobal-Strategies\n",
      "\n",
      "\u001b[1mNo Reconciliation Breaks\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# Build your left side of the reconciliation before you made the adjustment\n",
    "before_adjustment = models.PortfolioReconciliationRequest(\n",
    "    portfolio_id=models.ResourceId(\n",
    "        scope=scope,\n",
    "        code=portfolio_code),\n",
    "    effective_at=(holdings_effective_date - timedelta(hours=1)).isoformat())\n",
    "\n",
    "# Build your right side of the reconciliation after you made the adjustment\n",
    "after_adjustment = models.PortfolioReconciliationRequest(\n",
    "    portfolio_id=models.ResourceId(\n",
    "        scope=scope,\n",
    "        code=portfolio_code),\n",
    "    effective_at=(holdings_effective_date + timedelta(hours=1)).isoformat())\n",
    "\n",
    "# Create your reconciliation request\n",
    "reconcile_holdings_request = models.PortfoliosReconciliationRequest(\n",
    "    left=before_adjustment,\n",
    "    right=after_adjustment,\n",
    "    instrument_property_keys=['Instrument/default/Name', 'Instrument/default/Figi'])\n",
    "\n",
    "# Reconcile your two portfolios\n",
    "response = api_factory.build(lusid.api.ReconciliationsApi).reconcile_holdings(\n",
    "    portfolios_reconciliation_request=reconcile_holdings_request\n",
    ")\n",
    "\n",
    "# Pretty print the response\n",
    "prettyprint.reconciliation_response(\n",
    "    response,\n",
    "    scope,\n",
    "    portfolio_code)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great you no longer have any breaks and your transaction feed for the day is in-sync with the source system. You can now repeat this process as more transactions come in over the coming days."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}